robots.txt examples and the AI crawlers sites block

Root-level plain-text file of crawl rules that a site serves at /robots.txt for automated clients to read.

Sites with robots.txt10,57182% of the 12,924 websites we track

Block at least one AI crawler9079% of sites with robots.txt

Block every crawler176with a site-wide Disallow for User-agent: *

Declare a sitemap9,35288% of sites with robots.txt

What robots.txt is

Martijn Koster defined the protocol in 1994. RFC 9309 later made it an IETF Standards Track document, though the RFC says its rules are not access authorisation and Cloudflare notes that compliance is voluntary. We count any root file holding a real directive rather than an HTML error page, so a near-empty robots.txt still counts.

Sources: RFC 9309: Robots Exclusion Protocol, Cloudflare docs: managed robots.txt · Checked 16 Oct 2026

AI crawlers blocked most

Share of the 10,571 sites with a robots.txt that disallow each crawler for the whole site. 661 also publish Content Signals, 361 of them with ai-train=no.

1
GPTBotThe opt-out OpenAI documents for keeping a site's content out of its foundation model training.
650 (6%)Examples
2
CCBotSite-wide refusal of the crawler that builds Common Crawl's free public archive of web data.
636 (6%)Examples
3
BytespiderByteDance-attributed crawler, blocked site-wide, that ByteDance itself documents nowhere we could find.
627 (6%)Examples
4
ClaudeBotAnthropic's documented training opt-out, covering what the site publishes from here on, not what was already collected.

Adoption by month

Share of the sites we scanned each month that had robots.txt.

Nov 2026 of 6785%
Oct 2026 of 13,03182%

Sites blocking Applebot-Extended

The number is how many of the AI crawlers above each site disallows. Each links to the site's report, where its whole robots.txt is shown.