robots.txt
See which AI crawlers other sites block →# youasktube.
#
# Crawler policy. Groups are ordered narrowest-first: a crawler obeys exactly ONE
# group (the most specific user-agent token that matches it), so any crawler not
# named below falls into "*" and is welcome.
#
# Two different things are expressed here; do not confuse them. A Disallow on a
# FETCHING crawler (GPTBot, ClaudeBot, CCBot ...) is a traffic control - an
# obedient bot stops requesting pages, so it costs less. A Disallow on a
# USAGE-CONTROL token (Google-Extended, Applebot-Extended) is a licensing signal
# only: neither token ever fetches a page, so neither line changes request
# volume, bandwidth or cost by one byte. Both kinds are requests, not
# enforcement - the enforcement layer is the Cloudflare WAF user-agent rules.
#
# Content-Signal states the licensing preference for this content in the form
# Cloudflare's Content Signals Policy defines (contentsignals.org):
# search=yes index it and link back, that is what the site is for
# ai-input=yes a live assistant may read a page to answer a user's question
# ai-train=no do not use it to train or fine-tune a model
# It is a stated preference, not an enforcement mechanism. The directive is a
# member of the "*" group, so it sits after that User-agent line, not above it.
#
# The fetching group below is training-corpus crawlers: they take every page and
# send nothing back. The AI clients that DO send readers are deliberately NOT
# listed, so they inherit "*" and stay allowed - ChatGPT-User, OAI-SearchBot,
# Claude-User, Claude-SearchBot, PerplexityBot. Measured 2026-08 over 7 days,
# every AI referral the portfolio received came from one of them (chatgpt.com).
# Search engines are likewise unlisted and fully allowed: Googlebot, bingbot,
# Applebot, YandexBot, DuckDuckBot.
#
# A user-agent token CANNOT CONTAIN A DIGIT. RFC 9309 2.2.1 defines
# identifier = 1*(%x2D / %x41-5A / %x5F / %x61-7A)
# - hyphen, A-Z, underscore, a-z, and no digit range at all. Google's reference
# parser implements exactly that: ExtractUserAgent() in robots.cc advances while
# (ascii_isalpha(*end) || *end == '-' || *end == '_') and stops at the first
# digit. A line reading "User-agent: AI2Bot" is therefore parsed as the token
# "AI", never matches AI2Bot, and is an inert directive that reads like
# protection. AI2Bot and MJ12bot are absent from this file for that reason and
# are blocked by user agent at the WAF instead, where substring matching is real
# (ai2bot in TRAINING_UAS, mj12bot in SEO_UAS -
# strategy/audit-2026-08-14/apply-bot-rules.py). Do not add them back here, and
# check any new token for digits before adding it.
#
# Applebot-Extended is Apple's TRAINING token, not its crawler. Apple: "Applebot-
# Extended does not crawl webpages ... only used to determine how to use the data
# crawled by the Applebot user agent." Disallowing it opts out of Apple AI
# training and changes traffic by nothing at all: Applebot itself keeps full
# access through "*" and keeps costing exactly what it costs today. Over the same
# 7 days Applebot was the largest allowed consumer of the portfolio - 2,363,464
# requests, 20.9% of all traffic, 18.2 GB - with no measurable referrals, and
# nothing in this file reduces that. There is no robots.txt lever for it: Apple
# states plainly "Applebot does not follow crawl-delay", and giving Applebot its
# own group to carry one would take it OUT of the "*" group and silently drop the
# rules below. Cutting Applebot volume means a path-level Disallow or the WAF.
#
# Google-Extended is the same kind of token - it never fetches, so it saves no
# traffic either - but it is the one entry that costs us something real: Google
# uses it for Gemini TRAINING *and* for grounding (feeding indexed pages to the
# model at prompt time), with no way to allow one and refuse the other.
# Disallowing it is what ai-train=no means for Google, at the price of Gemini
# grounding. It does not affect Google Search inclusion or ranking - that is
# Googlebot, untouched. Drop this one line if grounding is later judged worth the
# training use.
#
# meta-externalagent is a real fetching crawler and is already blocked by user
# agent at the WAF on every zone here (2026-08-09, after Meta's hybrid UA rode
# the honest-bot exemption). Listing it keeps the published policy and the
# enforced one saying the same thing.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: anthropic-ai
User-agent: Diffbot
User-agent: Timpibot
User-agent: omgili
User-agent: ImagesiftBot
Disallow: /
# PetalBot (Huawei Petal Search). Cloudflare classifies it as an AI crawler, its
# search share in this site's markets is negligible, and the ai-train=no policy
# applies, so it is refused here (D23, 26-10-2026).
User-agent: PetalBot
Disallow: /
# Usage-control tokens. Neither fetches anything - disallowing them is a
# statement about how already-crawled content may be used, not a traffic control.
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /api/
Sitemap: https://youasktube.com/sitemap.xml
llms.txt
Compare with other sites' llms.txt files →# YouAskTube
> Ask questions about any YouTube video. Type "ask" into a YouTube link (youtube.com becomes youasktube.com) or paste a link, then ask. Answers come from the video's captions and link to the moment they were said. Tap one and the player jumps there.
## How it works
- Works on public YouTube videos that have captions. No captions means no answers.
- No sign-up needed to start. A free account gets more each day.
- Answers come in the language you ask in.
- Videos play in YouTube's own embedded player.
## Plans (per day)
- Without an account: a small daily allowance
- Free account: 2 videos, 4 questions, videos up to about 1 hour
- Pro, US$5 a month: 5 videos, 10 questions, videos up to about 3.5 hours
- Max, US$10 a month: 15 videos, 30 questions, videos up to about 6 hours
- Free answers show one time mark. Pro and Max show all of them.
## Pages
- Home: https://youasktube.com/
- FAQ: https://youasktube.com/faq/
- Plans: https://youasktube.com/account
- Terms: https://youasktube.com/terms/
- Privacy: https://youasktube.com/privacy/
## About
Made by BitVibe Labs (https://bitvibelabs.com). Not affiliated with YouTube or Google.