robots.txt
See which AI crawlers other sites block →#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used: http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html
# Slow down aggressive SEO crawlers
User-agent: SemrushBot
Crawl-delay: 30
User-agent: AhrefsBot
Crawl-delay: 30
User-agent: MJ12bot
Crawl-delay: 30
# AI training crawlers — higher delay (or Disallow: / if preferred)
User-agent: GPTBot
Crawl-delay: 30
User-agent: Google-Extended
Crawl-delay: 30
User-agent: CCBot
Crawl-delay: 60
User-agent: anthropic-ai
Crawl-delay: 30
User-agent: ClaudeBot
Crawl-delay: 30
# Meta's social preview crawler ignores Crawl-delay, but document intent
User-agent: facebookexternalhit
Crawl-delay: 10
User-agent: *
Crawl-delay: 30
# CSS, JS, Images
Allow: /misc/*.css$
Allow: /misc/*.css?
Allow: /misc/*.js$
Allow: /misc/*.js?
Allow: /misc/*.gif
Allow: /misc/*.jpg
Allow: /misc/*.jpeg
Allow: /misc/*.png
Allow: /modules/*.css$
Allow: /modules/*.css?
Allow: /modules/*.js$
Allow: /modules/*.js?
Allow: /modules/*.gif
Allow: /modules/*.jpg
Allow: /modules/*.jpeg
Allow: /modules/*.png
Allow: /profiles/*.css$
Allow: /profiles/*.css?
Allow: /profiles/*.js$
Allow: /profiles/*.js?
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /themes/*.css$
Allow: /themes/*.css?
Allow: /themes/*.js$
Allow: /themes/*.js?
Allow: /themes/*.gif
Allow: /themes/*.jpg
Allow: /themes/*.jpeg
Allow: /themes/*.png
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
# Files
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/
# EFF Stuff
# Don't crawl pages endlessly.
Disallow: /*?*page=
Allow: /updates?page=
Disallow: /documents?language=
Allow: /documents?page=
# Disallow language prefixes, these should be passed via the `language` param.
Disallow: /am/*
Disallow: /ar/*
Disallow: /de/*
Disallow: /el/*
Disallow: /en/*
Disallow: /es/*
Disallow: /fa/*
Disallow: /fr/*
Disallow: /hi/*
Disallow: /it/*
Disallow: /ja/*
Disallow: /ko/*
Disallow: /my/*
Disallow: /nb/*
Disallow: /nl/*
Disallow: /pl/*
Disallow: /pt/*
Disallow: /pt-br/*
Disallow: /ro/*
Disallow: /ru/*
Disallow: /sh/*
Disallow: /sv/*
Disallow: /th/*
Disallow: /tr/*
Disallow: /uk/*
Disallow: /vi/*
Disallow: /wp/*
Disallow: /zh/*
Disallow: /zh-hans/*
Disallow: /zh-hant/*
llms.txt
Compare with other sites' llms.txt files →# Electronic Frontier Foundation (EFF)
> The Electronic Frontier Foundation is the leading nonprofit organization defending civil liberties in the digital world. Founded in 1990 and based in San Francisco, EFF champions user privacy, free expression, and innovation through impact litigation, policy analysis, grassroots activism, and technology development.
EFF's work is organized into issue areas (free speech, privacy, creativity and innovation, transparency, international, security, and artificial intelligence). Most current analysis is published on the Deeplinks blog; press releases, whitepapers, legal case pages, and events are collected under Updates. EFF also builds free privacy and security tools.
EFF is a donor-funded, member-supported US 501(c)(3) nonprofit. Donations and memberships: https://supporters.eff.org/donate/join-4--llm
Original material on eff.org is licensed under Creative Commons Attribution 4.0 (CC BY) unless otherwise noted; please attribute EFF and link to the source page. Material not created by EFF may require permission from its copyright holder.
EFF provides general information, not legal advice. People seeking legal help should use the legal assistance page rather than relying on summaries of EFF content.
## About EFF
- [About EFF](https://www.eff.org/about): Mission and overview of the organization
- [History](https://www.eff.org/about/history): EFF's history since 1990
- [Victories](https://www.eff.org/victories): Notable legal and policy wins
- [Staff](https://www.eff.org/about/staff): Lawyers, technologists, activists, and other staff
- [Annual Reports and Financials](https://www.eff.org/about/annual-reports-and-financials): Annual reports, audited financials, and tax filings
- [Contact](https://www.eff.org/about/contact): General contact information
- [Press Contact](https://www.eff.org/press/contact): Media inquiries
## Issues
- [All Issues](https://www.eff.org/work): Index of EFF issue areas and campaigns
- [Free Speech](https://www.eff.org/issues/free-speech): Online expression, content moderation, censorship, and Section 230
- [Privacy](https://www.eff.org/issues/privacy): Government and corporate surveillance, data protection, and encryption
- [Creativity and Innovation](https://www.eff.org/issues/innovation): Copyright, DMCA, patents, fair use, and right to repair
- [Transparency](https://www.eff.org/issues/transparency): FOIA work and government accountability
- [International](https://www.eff.org/issues/international): Digital rights work outside the United States
- [Security](https://www.eff.org/issues/security): Encryption, security research, and secure technology
- [Artificial Intelligence](https://www.eff.org/issues/ai): AI policy, copyright, surveillance, and civil liberties
## Updates and Analysis
- [Deeplinks Blog](https://www.eff.org/deeplinks): EFF's primary blog of analysis and commentary on current digital rights issues
- [All Updates](https://www.eff.org/updates): Blog posts, press releases, whitepapers, and events in one listing
- [Press Releases](https://www.eff.org/updates?type=press_release): Official EFF announcements
- [Whitepapers](https://www.eff.org/updates?type=whitepaper): Long-form research and policy papers
- [Legal Cases](https://www.eff.org/pages/legal-cases): Cases EFF has litigated or filed briefs in, with case documents
- [EFFector Newsletter](https://www.eff.org/effector): EFF's long-running email newsletter and archive
- [Updates RSS Feed](https://www.eff.org/rss/updates.xml): Machine-readable feed of the latest blog posts and press releases
## Tools
- [Tools Overview](https://www.eff.org/pages/tools): All EFF-built tools and projects
- [Surveillance Self-Defense](https://ssd.eff.org/): Guides for protecting yourself and others from surveillance
- [Privacy Badger](https://privacybadger.org/): Browser extension that blocks invisible trackers
- [Certbot](https://certbot.eff.org/): Free tool for obtaining Let's Encrypt HTTPS certificates
- [Cover Your Tracks](https://coveryourtracks.eff.org/): Test how well your browser protects against tracking and fingerprinting
- [Atlas of Surveillance](https://www.eff.org/pages/atlas-surveillance): Database of surveillance technology used by U.S. law enforcement
- [Street Level Surveillance](https://sls.eff.org/): Explainers on police surveillance technologies
## Get Help or Get Involved
- [Legal Assistance](https://www.eff.org/pages/legal-assistance): How to request legal help from EFF
- [Security Contact](https://www.eff.org/security): Reporting security issues to EFF
- [Action Center](https://act.eff.org/): Current campaigns and calls to action
- [Donate](https://supporters.eff.org/donate/join-4--llm): Become an EFF member
- [Shop](https://shop.eff.org/): EFF merchandise; purchases support EFF's work
- [Membership FAQ](https://www.eff.org/pages/membership-faq): Questions about EFF membership
## Optional
- [Podcasts](https://www.eff.org/pages/keep-digital-rights-eff-podcasts): How to Fix the Internet and EFFector podcasts
- [Events](https://www.eff.org/events/list): Upcoming and past EFF events
- [Opportunities](https://www.eff.org/about/opportunities): Jobs, internships, and volunteering
- [Where to Find EFF](https://www.eff.org/pages/where-find-eff): EFF's official social media accounts
- [Copyright Policy](https://www.eff.org/copyright): Licensing terms for EFF content
- [Privacy Policy](https://www.eff.org/policy): How EFF handles visitor and member data
- [Trademark and Brand Usage](https://www.eff.org/pages/trademark-and-brand-usage-policy): Rules for using EFF's name and logo
