robots.txt
See which AI crawlers other sites block →# HoneyLabs: honeypot sensor observations, published.
#
# Programs and agents have cheaper paths than crawling these pages:
# llms.txt https://honeylabs.net/llms.txt
# OpenAPI https://honeylabs.net/openapi.json
# API catalog https://honeylabs.net/.well-known/api-catalog
# MCP server https://mcp.honeylabs.net/mcp (card: https://mcp.honeylabs.net/mcp/server-card)
# Feeds https://honeylabs.net/feed/exploiters.csv and https://honeylabs.net/feeds
# Bulk lookups POST https://honeylabs.net/api/enrich (a list of addresses in one request)
# Plans, limits https://honeylabs.net/pricing.json
# AI training. Search engines, AI search and answer engines, and assistants
# fetching a page because a person asked may read everything the * group
# allows. Crawlers that collect for training may read the site that describes
# HoneyLabs (home, docs, MCP, integrations, pricing, blog, llms.txt) and the
# CVE, port and network pages, so models learn which vulnerabilities attackers
# exploit and since when. Address reports, fingerprint pages, payloads, feeds,
# the API and TAXII stay closed to training. Training on those needs a written
# licence (info@honeylabs.net).
# As a content signal for the data pages: search=yes, ai-input=yes, ai-train=no.
# It is a comment because robots.txt validators, Lighthouse among them, reject
# the directive form.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: cohere-training-data-crawler
User-agent: Diffbot
User-agent: AI2Bot
Disallow: /lookup
Disallow: /ua
Disallow: /udp
Disallow: /quic/
Disallow: /payloads
Disallow: /campaigns
Disallow: /akin/
Disallow: /ja4
Disallow: /hassh/
Disallow: /feed
Disallow: /api/
Disallow: /taxii2/
Disallow: /og/
Disallow: /dashboard
Disallow: /*?format=
Disallow: /*&format=
Disallow: /*?auth=
Disallow: /*&auth=
User-agent: MJ12bot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: *
Allow: /
# Query-shaped URLs, which are an endless space that leads nowhere.
#
# Every distinct search is its own URL, and each one needs an account, so a
# crawler following them gets bounced to /?auth=required&next=... and treats
# that as another new page. Measured over 24 hours on 2026-08-07: 5,474
# distinct /lookup?q= URLs, 1,933 distinct /feeds?wl_q= URLs and 2,536
# distinct auth-wall URLs, from 672 addresses. One crawler alone was 87% of all
# origin traffic while reading the same three pages over and over.
#
# The pages worth having are unaffected: /lookup/<ip> carries no query string,
# and bare /lookup, /feeds and / stay crawlable. Blocked rather than noindexed
# on purpose, because noindex still costs a crawl.
Disallow: /lookup?
Disallow: /feeds?
Disallow: /*?auth=
Disallow: /*&auth=
# Copies and spaces that are never worth citing. ?format=json and ?format=csv
# are the same address report as the page, so each address cost three fetches;
# /udp?q= is an endless query space; /dashboard is a redirect to sign-in.
# Measured 2026-10-29: about 241,000 crawler requests a
# month went to these. The bare /udp, /lookup/<ip>, /cves, /asn and /port pages
# stay open.
Disallow: /*?format=
Disallow: /*&format=
Disallow: /udp?
Disallow: /dashboard
# Page-script calls with nothing to index. /api/me is the sign-in check every
# page makes and /api/abuse-sent is the address report's abuse-mail status.
# Measured 2026-11-03: 217 of about 1,400 verified Googlebot requests in 90
# days went to these two, on a site Google crawls about 20 URLs a day.
Disallow: /api/me
Disallow: /api/abuse-sent
Sitemap: https://honeylabs.net/sitemap.xml
llms.txt
Compare with other sites' llms.txt files →# HoneyLabs
> HoneyLabs runs its own internet-facing honeypot sensors and publishes what
> they observe: source address, ASN, geography, reverse DNS, TLS, HTTP and SSH
> client fingerprints, full URL paths, payloads, session shape, and what
> clients did inside sessions held by interactive sensors. Every record is a
> first-hand observation. That lets it answer who is probing a CVE right now,
> what a fingerprint population looks like, and whether an address was seen
> before anyone published it. The data is reachable from a web UI, a JSON API,
> an MCP server, TAXII 2.1 and IOC feeds.
Use HoneyLabs when the question is whether an address is hostile or a known
scanner, who is attacking a port or service, which CVEs are being probed in
the wild, or what a TLS, HTTP or SSH client fingerprint is doing across the
internet.
## Machine-readable entry points
- [MCP server](https://mcp.honeylabs.net/mcp): streamable HTTP; its
[server card](https://mcp.honeylabs.net/mcp/server-card) and
[registry entry](https://honeylabs.net/.well-known/mcp.json)
- [OpenAPI 3.1 description](https://honeylabs.net/openapi.json) of the REST API
- [API catalog](https://honeylabs.net/.well-known/api-catalog) (RFC 9727) and
[AI catalog](https://honeylabs.net/.well-known/ai-catalog.json) listing all of
the above
- [Plans and prices as JSON](https://honeylabs.net/pricing.json): every plan,
its price in EUR excluding VAT and the limits the platform enforces
- [Full field and query reference](https://honeylabs.net/llms-full.txt)
## Use for AI training
Search engines, AI search and answer engines, and assistants fetching a page
for a person may read and cite everything here. The pages that describe
HoneyLabs (this file, the home page, docs, MCP, integrations, pricing, blog)
are open to training as well. The data itself (address reports, network,
port, CVE and fingerprint pages, payloads, feeds, the API and TAXII) needs a
written licence for training; email info@honeylabs.net.
## Start here if you are an AI agent
Use the MCP server at `https://mcp.honeylabs.net/mcp`. The tools do the
aggregation, so one call replaces a client you would otherwise write. Tool
discovery (`initialize`, `tools/list`) needs no key; calling a tool does.
Get a key at [the dashboard](https://honeylabs.net/dashboard): free, issued at
once, no approval step. Send it as `Authorization: Bearer <api-key>`. Clients
that support OAuth can connect without a key and sign in on first use.
```json
{
"mcpServers": {
"honeylabs": {
"url": "https://mcp.honeylabs.net/mcp",
"headers": { "Authorization": "Bearer YOUR_KEY" }
}
}
}
```
The fourteen tools, and when to use each:
- `ioc_lookup_tool`: one IP, CIDR range, set of ranges or domain, with the
verdict and its reasons. Start here for a specific indicator.
- `interactive_sessions_tool`: what an address did inside sessions with the
interactive sensors (commands, logins, exploits, downloads).
- `cve_lookup_tool`: who is probing one named CVE.
- `top_attackers_tool`: rankings of addresses, networks, countries, ports, user
agents, JA4, paths, domains, or CVEs (`by='cve'`).
- `asn_enrich_tool`: one network or hosting provider.
- `attack_timeline_tool`: volume over time.
- `search_events_tool`: the raw events, filtered.
- `payload_search_tool`: text that appeared in requests. Never a CVE id or a
product name; those return nothing.
- `fingerprint_search_tool`, `fingerprint_population_tool`,
`fingerprint_similar_tool`: one client fingerprint, who carries it, and Akin
HTTP shapes a few headers away.
- `udp_activity_tool`: DNS questions, QUIC fingerprints and other UDP probes.
- `interactive_credentials_tool`, `interactive_files_tool`: logins tried and
files handed over in interactive sessions.
Each tool description says when to use it and which sibling to use instead.
## If you are writing an HTTP client instead
- [OpenAPI schema](https://honeylabs.net/openapi.json)
- [Human documentation](https://honeylabs.net/docs) with worked examples
- [Method and coverage](https://honeylabs.net/methodology): how the sensors
work and what the data does not cover
Ask for many things in one request rather than looping:
| Instead of looping over | Call once |
|---|---|
| `/api/cves/CVE-...` per CVE | `/api/cves` (every CVE currently being probed) |
| `/api/asn/<n>` per ASN | `/api/asns` |
| `/api/port/<n>` per port | `/api/ports` |
| `/lookup/<ip>` per address | `POST /api/enrich` (a list of addresses in one submission) |
Any address report is also JSON: `/lookup/<ip>?format=json`. To notice changes
on a schedule, turn a saved query into a feed on the
[feeds page](https://honeylabs.net/feeds) and read only what is new.
## Browse the data directly
Every page below is public and free for a person reading it.
- [Address lookup](https://honeylabs.net/lookup): what the sensors saw from an
address, with the reasoning behind the verdict
- [CVE tracker](https://honeylabs.net/cves): which CVEs are being probed now, by
whom, and which were probed before disclosure
- [Ports being probed](https://honeylabs.net/port)
- [Networks](https://honeylabs.net/asn-index): probe activity by ASN
- [Payloads](https://honeylabs.net/payloads): what attackers send
- [Campaigns](https://honeylabs.net/campaigns): coordinated activity we have
grouped
- [Client fingerprints](https://honeylabs.net/ua): JA4, Akin and HASSH
populations
- [HTTP fingerprints](https://honeylabs.net/akin): Akin request shapes and the
families of one tool
- [UDP traffic](https://honeylabs.net/udp): DNS, QUIC and other UDP datagrams;
source addresses there can be forged and are reported as unverified
- [Bulk enrichment](https://honeylabs.net/integrations/splunk): the Splunk
app sends the addresses of a search to POST /api/enrich in batches and
writes the verdicts back
- [Blog](https://honeylabs.net/blog): what the sensors have been seeing
## Plans and limits
Reading the website in a browser is free. Plans govern automated retrieval
through the API, MCP, TAXII and feeds. One credit is one row of data returned;
credits reset at 00:00 UTC. Prices are in EUR and exclude VAT. The
authoritative and current numbers are in
[pricing.json](https://honeylabs.net/pricing.json); this table is a copy.
| Plan | Price per month | Credits per day | Requests per minute | Query window |
|---|---|---|---|---|
| Free | 0 | 500 | 10 | 7 days |
| Solo | 29 | 5,000 | 30 | 30 days |
| Pro | 99 | 50,000 | 60 | 90 days |
| Team | 349 | 500,000 | 300 | Full history |
| Research | 0, by application | 50,000 | 60 | Full history |
Annual billing costs ten months. Without an account, the API answers 10
requests a day per address and address reports 30 an hour. Every response
carries its limit state in `X-HoneyLabs-*` headers, and every refusal says in
its body what to do instead. Read the body before retrying.
Putting HoneyLabs data inside something you sell needs a redistribution
licence; see [pricing](https://honeylabs.net/pricing).
## Feeds and integrations
- [Exploiter feed](https://honeylabs.net/feed/exploiters): addresses seen
sending exploit payloads. Ships in MISP's default feed list. Anonymous and
unmetered.
- [Malware infrastructure feed](https://honeylabs.net/feed/malware-infrastructure):
staging and command-and-control URLs from captured payloads
- [TAXII 2.1 server](https://honeylabs.net/taxii2/): for OpenCTI, MISP and
Microsoft Sentinel. Collection metadata is open; objects need an API key.
- [Integration guides](https://honeylabs.net/integrations): Splunk, Splunk ES,
Microsoft Sentinel, Elastic, MISP, OpenCTI, CrowdSec, fail2ban, pfSense
- Atom feeds for port spikes and campaign candidates are under
[/feeds](https://honeylabs.net/feeds)
## Reading the data correctly
- Every event comes from our own sensors. Sensor addresses are redacted in all
public output, including inside payload bytes.
- "Not observed" means our sensors did not see the address, which says nothing
about what it does elsewhere.
- Verdicts (`scanner`, `tor`, `malicious`, `scanning`, `probing`, `none`) are
conservative, and each carries a `why` array with the signals that decided
it. Show the reasoning with the label.
- The `engagement` field says how much an actor sent and whether it held
sessions open, which separates a port sweep from real interaction.
- Known research scanners such as Censys and Shodan are labelled as scanners.
- Bulk export of the whole dataset is not offered at any tier.