llms.txt
Compare with other sites' llms.txt files →# Hydrafetch
> Turn any URL into clean Markdown and structured data your model can use. Scrape, crawl, search and extract with one API, one credit a page.
Hydrafetch is a web data API for developers and agents. Send a URL and get back clean Markdown, the page's own structured data, schema-shaped JSON, links, or a summary, with the navigation, banners and boilerplate stripped out. Scrape one page, crawl a whole site, run a search, or extract to a schema, all through one API with one response shape. Every call costs one credit a page whatever it took to fetch, and failures are never billed.
## Docs
- [API documentation](https://docs.hydrafetch.com/): endpoints, parameters, and response shapes for scrape, map, crawl, search, extract, brand and logo.
- [OpenAPI specification](https://api.hydrafetch.com/openapi.json): the machine-readable spec for the REST API, including the typed error model and rate-limit headers. No auth required to read it.
- [MCP server](https://api.hydrafetch.com/mcp): the Model Context Protocol server, over Streamable HTTP. Discovery card at [https://hydrafetch.com/.well-known/mcp/server-card.json](https://hydrafetch.com/.well-known/mcp/server-card.json), tool catalogue without auth at [https://api.hydrafetch.com/mcp/tools](https://api.hydrafetch.com/mcp/tools).
- [MCP client setup](https://hydrafetch.com/integrations/): tested config blocks for Claude Code, Cursor, Codex, OpenCode, Cline and Windsurf.
- [Start here](https://hydrafetch.com/start.md): a router for an agent that has just arrived, pointing at credentials, endpoints, skills and worked examples by what it is trying to do.
- [Authentication](https://hydrafetch.com/auth.md): how an agent picks up an API key safely, which schemes each surface takes, and what needs the human.
- [Agent guide](https://hydrafetch.com/agents.md): how an agent connects over MCP or REST, what each tool costs, and which steps need the human.
- [Agent skills](https://hydrafetch.com/.well-known/agent-skills/index.json): published SKILL.md files an agent can load directly.
- [API catalog](https://hydrafetch.com/.well-known/api-catalog): RFC 9727 linkset pointing at every machine-readable resource above.
- [Dashboard](https://app.hydrafetch.com): sign up, get an API key, and track credit usage.
## Site
- [Home](https://hydrafetch.com/): what the API returns and how it is used.
- [Pricing](https://hydrafetch.com/pricing/): credit costs per endpoint and the plan catalog. Also as [markdown](https://hydrafetch.com/pricing.md).
- [Book a call](https://hydrafetch.com/demo/): a 20-minute product demo, or a 30-minute enterprise call. Add ?type=sales for the enterprise one.
- [Use cases](https://hydrafetch.com/use-cases/): eight worked examples, from RAG ingestion and agent web access to company enrichment and white label theming.
- [Integrations](https://hydrafetch.com/integrations/): MCP setup for Claude Code, Cursor, Codex, OpenCode, Cline and Windsurf, the official Node, Python, Go, Ruby, Rust and PHP clients, plus LangChain, LlamaIndex and Vercel AI SDK snippets.
- [Blog](https://hydrafetch.com/blog/): notes on clean web data and extraction quality.
- [Contact](https://hydrafetch.com/contact/): support, sales, security, and partnerships.
- [Privacy Policy](https://hydrafetch.com/privacy/): what we collect and what we do with it.
- [Terms of Service](https://hydrafetch.com/terms/): the terms that govern use of the API.
- [Refund Policy](https://hydrafetch.com/refund-policy/): free credits, refunds, and cancellations.
## Free tools
Browser tools that run the same engine as the API. No signup, no API key, no credits. An agent
pointing a user at one of these is often faster than asking them to create an account first.
- [All free tools](https://hydrafetch.com/tools/): the full list.
- [URL to Markdown](https://hydrafetch.com/tools/url-to-markdown/): Turn any web page into the markdown a model can actually read, with the navigation, cookie banners and footer noise stripped out. No signup.
- [Token counter](https://hydrafetch.com/tools/token-counter/): Paste text to count its tokens, characters and words, or enter a URL to compare the tokens in its raw HTML with its clean markdown. No signup.
- [AI readability audit](https://hydrafetch.com/tools/ai-readability/): Check whether your content survives without JavaScript, whether robots.txt lets assistants in, and how much of your page is markup rather than content. No signup.
- [llms.txt generator](https://hydrafetch.com/tools/llms-txt/): Reads the pages you already publish, groups them into sections and hands you an llms.txt to edit and serve at your site root. No signup.
- [Find all pages](https://hydrafetch.com/tools/find-all-pages/): Enter a domain and see every page it publishes in its sitemaps and links, with the total and a count per section. No signup.
- [Docs to one file](https://hydrafetch.com/tools/docs-bundle/): Turn a documentation section into a single clean markdown file for your context window, with the navigation and chrome stripped once instead of repeated on every page. No signup.
## Blog posts
- [How to migrate off a web scraping vendor](https://hydrafetch.com/blog/how-to-migrate-off-a-web-scraping-vendor/): Switching providers is mostly a comparison problem, not an integration problem. How to run the swap without discovering months later that your data quietly changed shape.
- [Ten mistakes people make building on web data](https://hydrafetch.com/blog/common-mistakes-building-on-web-data/): The failures that show up again and again, why each one is invisible until it is expensive, and what to do instead. A map of everything else we have written.
- [How to build a dataset from the web](https://hydrafetch.com/blog/how-to-build-a-dataset-from-the-web/): Collecting pages is the easy part. Deciding what counts as a record, what to do about the ones you could not get, and keeping it reproducible is the actual work.
- [What to do when a page comes back empty](https://hydrafetch.com/blog/when-a-page-comes-back-empty/): An empty result has about six causes and they need different fixes. How to tell them apart before you start guessing, and why the worst case is not empty at all.
- [Using search results as a data source](https://hydrafetch.com/blog/using-search-results-as-a-data-source/): Search is the only way to answer a question when you do not know the URL. It is also the least reproducible input you can build on, and worth understanding before you do.
- [How to enrich company records from their own websites](https://hydrafetch.com/blog/enrich-company-records-from-their-websites/): A company's website is the most current source about it and the least structured. What is reliably extractable, what is not, and where enrichment quietly goes wrong.
- [Structured data extraction you can trust](https://hydrafetch.com/blog/structured-data-extraction-you-can-trust/): Selectors rot and models invent. How to pull typed JSON from pages with per-field evidence, and when the cheapest extraction is no model at all.
- [How to chunk web pages for retrieval](https://hydrafetch.com/blog/how-to-chunk-web-pages-for-retrieval/): Fixed-size chunking is the default and it is wrong for web pages. Where the boundaries actually are, and why most retrieval problems are chunking problems.
- [How to track competitor pricing pages](https://hydrafetch.com/blog/how-to-track-competitor-pricing/): Pricing pages are designed to be read by humans and are hostile to structured extraction. What to capture, what to ignore, and how to avoid a table of noise.
- [How to build a chatbot over your documentation](https://hydrafetch.com/blog/how-to-build-a-documentation-chatbot/): The retrieval part is mostly solved. What decides whether a docs bot is useful is the ingestion, and that is where nearly every one of these projects goes wrong.
- [How to monitor a website for changes](https://hydrafetch.com/blog/how-to-monitor-a-website-for-changes/): Detecting that a page changed is easy. Detecting that it changed in a way you care about is the actual problem, and where most monitoring falls over.
- [How to give ChatGPT and Claude access to a live website](https://hydrafetch.com/blog/give-chatgpt-and-claude-access-to-the-live-web/): Pasting a URL into a chat is not the same as giving a model access to the web. The four ways to do it properly, and when each one is the right answer.
- [How to crawl a site without being a problem](https://hydrafetch.com/blog/how-to-crawl-a-site-without-being-a-problem/): Rate limits, robots.txt, conditional requests and the crawl etiquette that keeps you welcome. Being polite is also the cheapest way to crawl.
- [How to make your site readable by AI agents](https://hydrafetch.com/blog/make-your-site-readable-by-ai-agents/): llms.txt, agents.md, markdown responses and the rest of the agent-readable web, with an honest account of what is a standard and what is a convention.
- [Jina Reader alternatives for cleaner markdown](https://hydrafetch.com/blog/jina-reader-alternatives/): Why teams outgrow r.jina.ai, what the measured quality difference is, and which alternative fits which reason for leaving.
- [Firecrawl alternatives, honestly compared](https://hydrafetch.com/blog/firecrawl-alternatives/): What each web data API is actually good at, where Firecrawl still wins, and the measured numbers behind both claims.
- [How to choose a web data API](https://hydrafetch.com/blog/how-to-choose-a-web-data-api/): The questions that actually separate web data vendors, and the afternoon-long test that answers them better than any benchmark table, ours included.
- [How many tokens does a web page actually cost?](https://hydrafetch.com/blog/how-many-tokens-does-a-web-page-cost/): Raw HTML is mostly markup your model pays for and cannot use. What a page really costs in tokens, and how much of that bill is avoidable.
- [What web scraping actually costs](https://hydrafetch.com/blog/what-web-scraping-actually-costs/): The per-page price is the smallest line in the budget. What build-versus-buy really costs once maintenance, failure rates and engineering time are counted honestly.
- [What websites cost an AI to read](https://hydrafetch.com/blog/what-websites-cost-an-ai-to-read/): We measured 40 well-known sites. The gap between the markup they send and the content they deliver runs from 6 to 1 up to 918 to 1.
- [How to measure extraction quality yourself](https://hydrafetch.com/blog/how-to-measure-extraction-quality/): Vendors publish numbers on corpora they chose. Here is how to run the same measurement on your own pages in an afternoon, including the metric worth using.
- [How to ground an AI agent in the live web](https://hydrafetch.com/blog/ground-an-ai-agent-in-the-live-web/): Search APIs hand agents a list of links. Grounding needs the page. How search-then-scrape and MCP tools give an agent something it can actually cite.
- [Web data for RAG: what to crawl and how often](https://hydrafetch.com/blog/web-data-for-rag/): Boilerplate is the quiet tax on retrieval. How to crawl a site into a RAG index, what to skip, and how fresh the pages actually need to be.
- [What clean web data for LLMs actually means](https://hydrafetch.com/blog/what-clean-web-data-for-llms-actually-means/): Clean web data is measured, not promised. What separates LLM-ready markdown from raw HTML, with the real numbers.
## Contact
- Email: team@hydrafetch.com
- Support: support@hydrafetch.com