robots.txt
# robots.txt — www.oclc.org
#
# stack-id: one
#
# Owner: wcmsadministrators
# Source: /projects/CMSS/repos/installscripts-publisherapache-gen3 —
# edit there and deploy; do not hand-edit in place
# Last review: 2026-09-15
# Review cycle: every 6 months (AI crawler user-agent strings change often)
#
# ---------------------------------------------------------------------------
# AI CRAWLER POLICY — marketing web is OPEN
# ---------------------------------------------------------------------------
# AI crawlers, answer engines, and agent fetchers are ALLOWED here on the same
# terms as any other crawler. This is a deliberate decision, not an omission:
# OCLC's interest in its marketing content is discoverability, including being
# found and cited inside AI assistants.
#
# No per-crawler groups are defined below, and that is intentional. A named
# User-agent group REPLACES the * group rather than merging with it, so a
# named group that only restates the * rules adds a second copy to keep in
# sync and changes nothing behaviourally. Add a named group ONLY when a
# specific crawler needs rules that DIFFER from the default.
#
# Covered by the default group, explicitly on the record as allowed:
# OpenAI GPTBot, OAI-SearchBot, ChatGPT-User
# Anthropic ClaudeBot, Claude-User, Claude-SearchBot
# Perplexity PerplexityBot, Perplexity-User
# Google Google-Extended (Gemini training/grounding permission token)
# Apple Applebot, Applebot-Extended
# Meta meta-externalagent, meta-externalfetcher
# Others CCBot (Common Crawl), Amazonbot, Bytespider
#
# SCOPE WARNING: this posture applies to the marketing web only. The asset
# requiring protection is WorldCat bibliographic data, which lives on
# search.worldcat.org under a separate, stricter policy. Do not copy this
# file there, and do not copy that file here.
#
# ---------------------------------------------------------------------------
# WHY THE DISALLOW RULES BELOW EXIST — read before changing any of them
# ---------------------------------------------------------------------------
# These are NOT content suppression. They target AEM SELECTOR permutations —
# faceted views of a page, where the selected facets are encoded as extra
# dot-separated segments before .html:
#
# /en/member-stories.html <- canonical, CRAWLABLE
# /en/member-stories.academic.emea.html <- facet permutation, blocked
#
# The `.*.` in each pattern matches those selectors. Because `.` is a literal
# in robots.txt and cannot match `/`, these patterns deliberately do NOT block
# the canonical page or any child page under a real path segment
# (e.g. /en/member-stories/auckland.html is crawlable, and is in sitemap.xml).
#
# Verified 2026-09-15: zero of the 8,243 URLs across all 25 child sitemaps
# match any rule in this file. robots.txt, sitemap.xml, and the noindex logic
# are consistent. Keep them that way.
#
# This is defence in depth, and all three layers already agree:
# 1. MetatagsBean emits <meta name="robots" content="noindex"> on any URL
# whose last path segment contains more than one period (i.e. faceted).
# 2. These rules stop the fetch outright, so facet permutations never
# consume crawl budget.
# 3. SitemapGeneratorServlet lists canonical URLs only.
#
# Removing a rule here does not "open up content" — the content is already
# reachable at its canonical URL. It opens the facet combinatorial explosion
# to crawlers. Don't.
#
# EXCEPTION to the sitemap check above: /bibformats/{en,fr}/search-results.html
# are listed in sitemap.xml and are matched by the search-results rule below.
# That is intended. They are Coveo results pages, not content, and should be
# dropped from the sitemap (CMSS-10414).
#
# ---------------------------------------------------------------------------
# COVEO SEARCH RESULTS PAGES (CMSS-10414)
# ---------------------------------------------------------------------------
# search-results.html calls executeFirstSearch() on load, so every render by
# a JavaScript-executing crawler runs a Coveo query that counts against the
# Coveo license. Coveo reported mostly YisouSpider, sogou, Applebot, NaverBot and Googlebot.
# The locale pages (/en/ etc.) are already noindex; that keeps them out of
# indexes but does not stop the fetch. This rule stops the fetch. The
# /bibformats/ results pages have no noindex tag at all.
#
# This is not an AI-crawler restriction. The rule sits in the * group and
# applies to every crawler equally; the policy above is unchanged.
# Crawlers that ignore robots.txt (e.g. spoofed Bytespider) need an Apache
# rule instead; robots.txt cannot stop them.
User-agent: *
# Faceted-view selector permutations — see explanation above.
Disallow: */contacts/all-contacts.*.html$
Disallow: */contacts/all-contacts/us-sales.*.html$
Disallow: */member-stories.*.html$
Disallow: */services/a-z.*.html$
Disallow: */developer/gallery.*.en.html$
Disallow: */developer/news.*.en.html$
Disallow: */developer/develop/web-services.*.en.html$
# Test-parameter URLs.
Disallow: /*?test=*
Disallow: /*&test=*
# Coveo search results pages — see explanation above. `*` matches across
# slashes, so this covers every locale (/en, /de, /es, /fr, /nl, /zh-Hans),
# /bibformats/*, the /content/marketing/publish/en_us/ path that 301s to
# /en/, and any query string. URL fragments (#q=...) never reach the server.
Disallow: /*/search-results.html
Sitemap: https://www.oclc.org/sitemap.xml
