robots.txt
See which AI crawlers other sites block →# As a condition of accessing this website, you agree to abide by the following
# content signals:
# (a) If a Content-Signal = yes, you may collect content for the corresponding
# use.
# (b) If a Content-Signal = no, you may not collect content for the
# corresponding use.
# (c) If the website operator does not include a Content-Signal for a
# corresponding use, the website operator neither grants nor restricts
# permission via Content-Signal with respect to the corresponding use.
# The content signals and their meanings are:
# search: building a search index and providing search results (e.g., returning
# hyperlinks and short excerpts from your website's contents). Search does not
# include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
# augmented generation, grounding, or other real-time taking of content for
# generative AI search answers).
# ai-train: training or fine-tuning AI models.
# use: how AI systems may consume the content (immediate, reference, or full).
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
# Training crawlers
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: omgili
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: MistralAI-Training
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: GoogleOther
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: AwarioSmartBot
User-agent: AwarioRssBot
Disallow: /
User-agent: Google-CloudVertexBot
Disallow: /
User-agent: QualifiedBot
Disallow: /
User-agent: Cotoyogi
Disallow: /
User-agent: ICC-Crawler
Disallow: /
User-agent: atlassian-bot
Disallow: /
User-agent: FishBot
Disallow: /
User-agent: BorderxBot
Disallow: /
User-agent: NavuBot
Disallow: /
User-agent: SemrushBot-SWA
Disallow: /
User-agent: WARDBot
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: KimiBot
Disallow: /
User-agent: CitibotSiteCrawler
Disallow: /
# END Cloudflare Managed Content
# Crawlee Cloud - robots.txt
# https://crawlee.cloud
User-agent: *
Allow: /
Disallow: /api/
Disallow: /_next/
Disallow: /.vitepress/
# Sitemap
Sitemap: https://crawlee.cloud/sitemap.xml
# AI Crawlers - Welcome!
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: CCBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: cohere-ai
Allow: /
# LLMs.txt location
# See: https://crawlee.cloud/llms.txt
llms.txt
Compare with other sites' llms.txt files →# Crawlee Cloud - llms.txt
> Self-hosted, open-source platform for running Crawlee and Apify Actors on your own infrastructure
## About
Crawlee Cloud is a developer-friendly platform for deploying and managing web scrapers. It provides complete infrastructure for running Crawlee-based scrapers (Playwright, Puppeteer, Cheerio) with an Apify-compatible API, allowing existing Apify SDK code to work without modifications.
## Full Documentation (Context for LLMs)
For a complete overview of the platform, including CLI usage, API reference, and deployment guide, please read the full context file:
- **[llms-full.txt](https://crawlee.cloud/llms-full.txt)**
## Quick Start Context
```bash
# Install CLI
npm install -g @crawlee-cloud/cli
# Login
crawlee-cloud login --url https://your-server.com
# Push a scraper
crawlee-cloud push my-scraper
# Run a scraper
crawlee-cloud run my-scraper --input '{"url": "https://example.com"}'
```
## Use Cases
- **Web scraping at scale**: Run thousands of concurrent scrapers on your own servers
- **Data extraction pipelines**: Collect product data, prices, leads, and content
- **E-commerce monitoring**: Track competitor prices and inventory
- **Lead generation**: Extract business information from directories
- **Content aggregation**: Gather news, reviews, and social media data
- **Research automation**: Academic and market research data collection
## Key Features
### Infrastructure
- Self-hosted on Docker, Kubernetes, or any VPS
- PostgreSQL for metadata, Redis for queues
- S3-compatible storage (MinIO for local dev)
- Horizontal scaling with multiple runners
### Developer Experience
- Apify SDK compatible - no code changes required
- CLI for actor deployment (`npx @crawlee-cloud/cli`)
- REST API for programmatic control
- Modern React dashboard for monitoring
### Crawlee Support
- Playwright scrapers (browser automation)
- Puppeteer scrapers (headless Chrome)
- Cheerio scrapers (fast HTML parsing)
- Request queues and datasets
- Proxy rotation and error handling
## Technology Stack
| Component | Technology |
|-----------|------------|
| API Server | Fastify 5, TypeScript 5, Zod 3 |
| Dashboard | Next.js 16, React 19, Tailwind CSS v4 |
| CLI | Commander.js 12, Node.js 20+ |
| Database | PostgreSQL 16, Redis 7 (ioredis) |
| Storage | S3 (AWS SDK v3), MinIO |
| Runner | Docker, Dockerode 4 |
## Documentation Pages
### Getting Started
- Overview: https://crawlee.cloud/docs/
- Deployment Guide: https://crawlee.cloud/docs/deployment
### Components
- CLI Reference: https://crawlee.cloud/docs/cli
- Dashboard Guide: https://crawlee.cloud/docs/dashboard
- Runner Architecture: https://crawlee.cloud/docs/runner
- REST API Reference: https://crawlee.cloud/docs/api
### Integration
- Apify SDK Compatibility: https://crawlee.cloud/docs/apify-sdk-environment
## Keywords
web scraping, crawlee, apify, self-hosted, open source, web crawler, data extraction, playwright scraper, puppeteer scraper, cheerio scraper, actor platform, scraping infrastructure, headless browser, data pipeline, web automation
## Links
- Website: https://crawlee.cloud
- GitHub Repository: https://github.com/crawlee-cloud/crawlee-cloud
- npm Package: https://www.npmjs.com/package/@crawlee-cloud/cli
- Product Hunt: https://www.producthunt.com/products/crawlee-cloud
## License
MIT License - Free and open source
## Contact
- GitHub Issues: https://github.com/crawlee-cloud/crawlee-cloud/issues
- GitHub Discussions: https://github.com/crawlee-cloud/crawlee-cloud/discussions