robots.txt
See which AI crawlers other sites block →# llmstxtvalidator.dev — robots.txt
# We build tools that help sites be AI-readable, so we opt IN to the major
# AI vendors — both their retrieval crawlers and their training crawlers —
# and opt OUT of bulk dataset scrapers that offer nothing back.
# ---- Major AI vendors — ALLOWED ----
# Includes training crawlers (GPTBot, Google-Extended, Applebot-Extended) as
# well as retrieval ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
User-agent: ClaudeBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Claude-User
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Claude-SearchBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: ChatGPT-User
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: OAI-SearchBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: GPTBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: PerplexityBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Perplexity-User
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Google-Extended
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Applebot-Extended
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Applebot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Bingbot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Googlebot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: DuckDuckBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Slurp
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: Baiduspider
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
User-agent: YandexBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
# ---- Bulk dataset scrapers — BLOCKED ----
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: Omgilibot
Disallow: /
# ---- Default for everything else ----
User-agent: *
Allow: /
Crawl-delay: 1
# Important files for AI ingestion
Allow: /llms.txt
Allow: /llms-full.txt
Allow: /ai.txt
Allow: /examples
# ---- Content Signals (Cloudflare) ----
# Says what the content may be USED for, which Allow/Disallow cannot express.
# search — indexing and linking, not AI summaries
# ai-input — real-time use in a generated answer (RAG)
# ai-train — training or fine-tuning a model
# See https://contentsignals.org/
#
# Repeated inside every allowed group above, not declared once here: a crawler
# obeys only its most specific matching group, so a signal living solely in
# "User-agent: *" is read by none of the named bots. Keep the lines identical.
Content-Signal: search=yes, ai-input=yes, ai-train=yes
# IETF AIPREF defines an equivalent `Content-Usage:` rule, but
# draft-ietf-aipref-attach has not been published yet and almost nothing reads
# it. We will add it here once it ships rather than broadcast a preference no
# crawler acts on:
# Content-Usage: train-ai=y, search=y
Sitemap: https://llmstxtvalidator.dev/sitemap.xml
Host: llmstxtvalidator.dev
llms.txt
Compare with other sites' llms.txt files →# LLMs.txt Validator
> A free, browser-based validator for the files that tell AI systems how to read your site: llms.txt, llms-full.txt, ai.txt (both competing standards), robots.txt and security.txt. Everything runs client-side — nothing you paste or fetch is stored or sent to a server.
Validation covers the llms.txt v2 specification, RFC 9309 robots.txt (including AI crawler coverage, Cloudflare Content Signals and the IETF AIPREF Content-Usage draft), and RFC 9116 security.txt.
## Tools
- [Validator](https://llmstxtvalidator.dev/): Paste a file or give a URL; format is auto-detected, or pick one from the dropdown
- [Examples and templates](https://llmstxtvalidator.dev/examples): Downloadable valid and invalid samples for every supported format
## Guides
- [Complete guide to llms.txt](https://llmstxtvalidator.dev/blog/complete-guide-to-llms-txt): What the file is, what goes in it, and why
- [How to implement llms.txt](https://llmstxtvalidator.dev/blog/how-to-implement-llms-txt): Step-by-step, with framework-specific notes
- [Best practices](https://llmstxtvalidator.dev/blog/best-practices-for-llms-txt): Patterns that make a file useful to an agent rather than just valid
- [Common mistakes](https://llmstxtvalidator.dev/blog/common-llms-txt-mistakes): The errors that show up most often, and the fix for each
- [How to validate](https://llmstxtvalidator.dev/blog/how-to-validate-llms-txt): URL or paste, format detection, and what errors, warnings and notes mean
- [Automate generation](https://llmstxtvalidator.dev/blog/automate-llms-txt-generation): Build-time and CMS approaches
- [llms.txt for ecommerce](https://llmstxtvalidator.dev/blog/llms-txt-for-ecommerce): Product discovery through AI shopping assistants
## Comparisons
- [llms.txt vs robots.txt](https://llmstxtvalidator.dev/blog/llms-txt-vs-robots-txt): Different jobs — one invites, the other restricts
- [llms.txt vs sitemap.xml](https://llmstxtvalidator.dev/blog/llms-txt-vs-sitemap-xml): Curation versus completeness
- [What is security.txt](https://llmstxtvalidator.dev/blog/what-is-security-txt): The RFC 9116 disclosure convention
## Measurement
- [SEO benefits of llms.txt](https://llmstxtvalidator.dev/blog/llms-txt-seo-benefits): What it does and does not affect
- [Measure llms.txt impact](https://llmstxtvalidator.dev/blog/measure-llms-txt-impact): Tracking whether AI systems actually fetch the file
- [Optimize your website for AI](https://llmstxtvalidator.dev/blog/optimize-website-for-ai): The wider picture beyond one file
## Reference
- [llmstxt.org](https://llmstxt.org): The specification itself, currently v2
- [Our robots.txt](https://llmstxtvalidator.dev/robots.txt): How this site handles AI crawlers
- [Our ai.txt](https://llmstxtvalidator.dev/ai.txt): Anti-hallucination context about this tool
## Optional
- [Blog index](https://llmstxtvalidator.dev/blog): Every guide in one place
- [index.guru](https://llmstxtvalidator.dev/blog/index-guru-ai-directory): A curated directory of AI tools and communities