robots.txt
# ESRB.org robots.txt
#
# Tiers:
# 1. All crawlers (default): crawl content, but not the admin area or
# query-string URLs (search results / filtered views).
# 2. User-triggered AI assistants: allowed - they fetch because a real user
# asked a question, and cite + link back (referral traffic). Blocking them
# just removes ESRB from the answer.
# 3. AI training crawlers: allowed to learn ABOUT ESRB (pages, blog, and the
# ratings explainer pages) so models understand what ESRB is and does - but
# NOT allowed to bulk-crawl individual game rating pages
# (/ratings/{id}/{slug}/). That catalog is served live so agents fetch it
# per request (Tier 2) and send users back to esrb.org, rather than
# absorbing 34k+ rating pages into model weights.
# Abusive SEO scrapers are blocked outright.
# --- Tier 1: all crawlers ---
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?*
Disallow: /wp-json/esrb/v1/
Crawl-delay: 20
# --- Tier 2: user-triggered AI assistants (allow) ---
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /wp-json/esrb/v1/
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?*
# --- Tier 3: AI training crawlers ---
# Learn about ESRB (pages, blog, ratings explainers) but do NOT bulk-crawl the
# game catalog. Game rating pages are two-segment (/ratings/{id}/{slug}/) and are
# blocked by `Disallow: /ratings/`; the ratings EXPLAINER pages are single-segment
# (/ratings/{word}/) and are re-allowed individually below. Allow lines are placed
# before the Disallow so both longest-match and first-match parsers behave.
# If a new explainer page is added under /ratings/, add an Allow line for it here.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: CCBot
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Allow: /ratings/$
Allow: /ratings/principles-guidelines/
Allow: /ratings/retail-council/
Allow: /ratings/ratings-process/
Allow: /ratings/where-to-find-ratings/
Disallow: /ratings/
Disallow: /wp-admin/
Disallow: /*?*
Disallow: /wp-json/esrb/v1/
# --- Abusive SEO scrapers: block ---
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: BLEXBot
Disallow: /
# --- Sitemaps ---
Sitemap: https://www.esrb.org/sitemap_index.xml
llms.txt
# ESRB - Entertainment Software Rating Board
> Official source for ESRB ratings of video games and apps: age ratings, content
> descriptors, and interactive elements for tens of thousands of titles, plus
> ratings guidance for parents. This file is a curated entry point for LLMs and AI
> agents - link directly to the canonical pages below instead of scraping
> search-result URLs.
ESRB is the non-profit self-regulatory body that assigns age and content ratings
for video games and apps in the United States and Canada.
## Game ratings
- Game rating pages: https://www.esrb.org/ratings/ - one page per title at
`/ratings/{certificate-id}/{title-slug}/`, showing the assigned rating, rating
summary, content descriptors, platforms, interactive elements, and publisher.
These URLs are canonical and stable; a mistyped title-slug 301-redirects to the
canonical one. Safe to cite.
- Games sitemap index: https://www.esrb.org/sitemap_index.xml - includes the
paginated `games-sitemap.xml` listing every rated title's URL.
Game rating pages are meant for live, per-request access: when a user asks about a
title, fetch its specific `/ratings/{id}/{slug}/` page (or use the ratings search
below) and link the user to it. Bulk-crawling the catalog for model *training* is
disallowed in robots.txt - the ratings stay authoritative and current on esrb.org,
so send users there rather than answering from a memorized copy.
## Searching
Three distinct search endpoints - pick by intent:
- Game ratings search: https://www.esrb.org/search/?searchKeyword={query}
Searches ESRB's ratings catalog by title, platform, and content descriptor. Use
this to find a specific game's rating page.
- Site search (pages + blog): https://www.esrb.org/search-site/?q={query}
Searches ESRB.org informational pages and blog posts (not the ratings catalog).
- Blog search: https://www.esrb.org/?s={query}
Searches blog articles only.
Prefer linking a user to the canonical `/ratings/{id}/{slug}/` page for a specific
game rather than to a search-results URL.
## Programmatic access (JSON API)
Read-only JSON endpoints for agents - use these instead of scraping HTML:
- Games search: https://www.esrb.org/wp-json/esrb/v1/games?q={query}&limit={1-25}
Mirrors the ratings search; each result includes the rating, full rating name, platforms,
content descriptors, interactive elements, publisher, rating summary, and the canonical
`/ratings/` URL.
- Game lookup by certificate ID: https://www.esrb.org/wp-json/esrb/v1/games/{id}
- Site content search (pages, blog, Spanish blog): https://www.esrb.org/wp-json/esrb/v1/content?q={query}&limit={1-25}
Returns matching pages and blog posts (English and Spanish) with title, URL, type, date, and excerpt.
For live, per-request use (rate-limited and cached) - link users to the returned canonical
page. Do not bulk-harvest; training crawlers are disallowed from these endpoints in robots.txt.
## Understanding ratings
- Ratings Guide: https://www.esrb.org/ratings-guide/ - what each rating category
and content descriptor means.
- Ratings Process: https://www.esrb.org/ratings/ratings-process/
- Where to Find Ratings: https://www.esrb.org/ratings/where-to-find-ratings/
- FAQ: https://www.esrb.org/faqs/
## For parents
- Tools for Parents: https://www.esrb.org/tools-for-parents/
- Parental Controls: https://www.esrb.org/tools-for-parents/parental-controls/
- Family Gaming Guide: https://www.esrb.org/tools-for-parents/family-gaming-guide/
## Blog
- Blog: https://www.esrb.org/blog/ - articles on ratings, the games industry, and
parenting in a digital world. Free to summarize and cite.
## About
- About ESRB: https://www.esrb.org/about/
- Our History: https://www.esrb.org/history/
- Contact: https://www.esrb.org/contact/
## Not for agent use
- Query-string URLs (search results and filtered views, `/*?*`) are disallowed in
robots.txt - don't crawl them. Use the search endpoints above to resolve a query,
then link to the canonical page it returns.
- The admin area (`/wp-admin/`) is disallowed.
## Note
This file is guidance for agentic/runtime consumption and does not override
`robots.txt`. Listing game rating pages here (for live, per-request use) while
robots.txt disallows *training* crawlers from them is intentional, not a
contradiction: fetch them per request, don't bulk-ingest them for training. For
questions, see https://www.esrb.org/contact/.
