crawlforge.dev

mx

smtp.google.com

ns

dns1.registrar-servers.com

dns2.registrar-servers.com

caa

txt

v=spf1 include:_spf.google.com ~all

smithery-verification=7fd9374799af5983ff1258ef150029ac646758245f5d4c47dcc675b5e7c47c31

dmarc

v=DMARC1; p=none; rua=mailto:simon@crawlforge.dev; fo=1

# CrawlForge Robots.txt
# https://www.crawlforge.dev
# Last reviewed: 2026-08-23

User-agent: *
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

# Content usage signals (https://contentsignals.org) — CrawlForge welcomes AI
# training, AI search indexing, and AI inference/grounding on its public content.
Content-Signal: ai-train=yes, search=yes, ai-input=yes

# --- AI Search & Retrieval Bots (drive citations in AI search results) ---

User-agent: OAI-SearchBot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: ChatGPT-User
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: Claude-SearchBot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: Claude-User
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: PerplexityBot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: Perplexity-User
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: DuckAssistBot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: DuckDuckBot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: MistralAI-User
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: Applebot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: Google-CloudVertexBot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

# --- AI Training Bots ---

User-agent: GPTBot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: ClaudeBot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: Amazonbot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: Meta-ExternalAgent
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: CCBot
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: cohere-ai
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

User-agent: Bytespider
Allow: /
Allow: /api/og
Allow: /api/logo
Disallow: /api/
Disallow: /dashboard/
Disallow: /signin
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

# --- Opt-out control tokens (not crawlers; they make no requests) ---
# A bare "Allow: /" is the opt-in signal for AI training/grounding.

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

Sitemap: https://www.crawlforge.dev/sitemap.xml

# IndexNow key for fast search engine indexation
# https://www.crawlforge.dev/b9b567a2380d9e4203de3e503333a1aa.txt

# AI-friendly content files
# See https://llmstxt.org for specification
# llms.txt: https://www.crawlforge.dev/llms.txt
# llms-ctx.txt: https://www.crawlforge.dev/llms-ctx.txt
# llms-full.txt: https://www.crawlforge.dev/llms-full.txt
# ai.txt: https://www.crawlforge.dev/ai.txt

# CrawlForge MCP
> Enterprise web scraping MCP server with 31 specialized tools for AI applications

## Overview
CrawlForge MCP is a Model Context Protocol server that provides 31 web scraping and
data extraction tools purpose-built for AI workflows. It integrates natively with
Claude Desktop, Claude Code, Cursor IDE, LangChain, LlamaIndex, and any MCP-compatible
AI framework. CrawlForge handles anti-bot detection, browser automation, geo-targeting,
and deep research so AI agents can focus on analysis instead of data collection.

## Privacy: redact_pii
Nine tools take a redact_pii parameter that removes personal data from the text they return:
scrape, extract_content, extract_text, batch_scrape, crawl_deep, stealth_mode (operation
"scrape"), scrape_with_actions, process_document and search_web; on the REST API
browser_session takes it too, on its snapshot, act and read operations. It is off by default;
redact_pii: true is shorthand for { mode: "fast" }. Four regex classes are detected — EMAIL,
PHONE, FINANCIAL (card numbers validated with Luhn, IBANs with mod-97) and SECRET (API keys,
bearer tokens, labelled passwords) — and a match is replaced as a tag (<EMAIL>), a mask
([REDACTED]) or nothing, with SECRET keeping its label and losing only its value in every
style. Redaction runs before the result is stored, so an oversized result read back later
with read_result is already redacted. The response carries redaction: { entities, count, mode }
at result.redaction on the MCP server and inside data on the REST API — the same access path
on purpose — whenever redaction was requested, count: 0 included. The default fast mode adds
0 credits; mode "model" covers PERSON and LOCATION for 3 credits once per call on the MCP
server and is rejected with a 400, unbilled, on the REST API, which runs no LLM. Counters
derived from the text (content_length, word_count, character_count) describe it before
redaction, and address-like fields (url, link, href, canonical_url) are never redacted.

## Localized Versions
CrawlForge is available in English (default), Spanish, Simplified Chinese, and Malay. The site
mirrors the English structure under locale path prefixes:
- [English (default)](https://www.crawlforge.dev/)
- [Español](https://www.crawlforge.dev/es)
- [简体中文 (zh-Hans)](https://www.crawlforge.dev/zh)
- [Bahasa Melayu (ms)](https://www.crawlforge.dev/ms)
Marketing, pricing, legal, documentation, and blog pages are translated; only the newest
blog posts stay English-only until their translations land. Every page declares hreflang
alternates; the XML sitemap at https://www.crawlforge.dev/sitemap.xml lists all localized
URLs.

## Tools
- fetch_url: Fetch and return webpage content (1 credit)
- extract_text: Extract clean text from URLs (1 credit)
- extract_links: Discover all links on a page (1 credit)
- extract_metadata: Extract page metadata including Open Graph and structured data (1 credit)
- scrape_template: Pre-built scrapers for 18 popular sites and APIs (Shopify, Amazon, GitHub, YouTube, six ATS job boards, NHTSA, NPI, etc.) (1 credit)
- get_batch_results: Retrieve results from a queued batch_scrape job (1 credit)
- read_result: Read a stored result by handle — slice, search, lines or json_path a result that came back truncated with a result_handle; stored results are kept 1 hour (1 credit)
- scrape: Multi-format scrape of a single URL — markdown, HTML, text, links, metadata, branding, or screenshot in one call; add a highlights or question format to get only the sentences, table rows and code blocks matching a query, verbatim with offsets into the markdown, no model in the path; set escalate: true and a page that comes back as a bot wall is re-fetched once in the stealth browser inside the same call, saving a second round trip and reported as an unbilled block if that render is refused too (2 credits, +1 with a highlights or question format, +5 when the escalation stage actually runs)
- scrape_structured: CSS selector-based scraping for precise data extraction (2 credits)
- extract_content: Smart content extraction that identifies main page content (2 credits)
- map_site: Site structure mapping and link analysis (2 credits)
- process_document: PDF and document processing with table extraction (2 credits)
- localization: Country locale lookups (language, Accept-Language, timezone, currency) and page language audits; returns values and applies none, uses no proxy and cannot lift a geo-block (2 credits)
- extract_embedded_state: Return the JSON state a page already ships in its own HTML — Next.js, RSC, Nuxt, SvelteKit, Apollo and Redux payloads, YouTube, Inertia and Shopify data, plus inline JSON blocks; no LLM, so nothing can be fabricated (2 credits)
- reddit_search: Search Reddit posts/comments or read a full thread via the Arctic Shift community archive; a Reddit-wide keyword search discovers posts with a web search and reads them from the archive (5 credits)
- browser_session: One browser page kept alive across API calls — open a session, snapshot the page for stable element refs (@e1), act on those refs, read the live DOM as it now stands, close; a REST API key holds one session at a time (priced per operation: open 3, read 2, snapshot/act/screenshot/close/list 1 credit each)
- track_changes: Content change tracking with baselines; operation "monitor" creates a hosted monitor that checks the page on a cron and posts signed webhooks (3 credits per call; a monitor is free to create and bills 3 credits per compared target per check)
- analyze_content: Sentiment, language, and readability analysis (3 credits)
- extract_structured: LLM-powered schema-driven extraction with CSS selector fallback (3 credits)
- extract_with_llm: LLM-powered extraction with local Ollama default (no API key needed), OpenAI/Anthropic optional (3 credits)
- summarize_content: AI-powered content summarization (4 credits)
- crawl_deep: Deep website crawling across multiple pages (4 credits)
- stealth_mode: Anti-detection browsing with fingerprint randomization (5 credits)
- scrape_with_actions: Browser automation scraping with clicks, scrolls, and form fills (5 credits)
- batch_scrape: Bulk URL processing in a single request (5 credits)
- search_web: Google-backed web search; pass queries instead of query to run 1-10 searches in one call, returned as results_by_query in the order sent, one entry per query, a failed query carrying error in place of its results (5 credits per query that returned results; a 10-query batch is projected at 50, a ceiling, and each failed query comes off it, so a batch in which none succeeded costs nothing and still returns 200)
- serp_rank: Check a domain's organic Google SERP position for a keyword — powered by DataForSEO (5 credits)
- generate_llms_txt: Generate standard-compliant llms.txt and llms-full.txt files (5 credits)
- agent: Autonomous multi-step web task — plans and runs a sequence of tool calls to fulfill a goal (8 credits)
- deep_research: Multi-stage AI research with source verification (10 credits)
- list_ollama_models: List available local Ollama models (1 credit)

## Tool Documentation
- [fetch_url](https://www.crawlforge.dev/docs/api-reference/tools/fetch-url)
- [extract_text](https://www.crawlforge.dev/docs/api-reference/tools/extract-text)
- [extract_links](https://www.crawlforge.dev/docs/api-reference/tools/extract-links)
- [extract_metadata](https://www.crawlforge.dev/docs/api-reference/tools/extract-metadata)
- [scrape_structured](https://www.crawlforge.dev/docs/api-reference/tools/scrape-structured)
- [extract_content](https://www.crawlforge.dev/docs/api-reference/tools/extract-content)
- [extract_embedded_state](https://www.crawlforge.dev/docs/api-reference/tools/extract-embedded-state)
- [map_site](https://www.crawlforge.dev/docs/api-reference/tools/map-site)
- [process_document](https://www.crawlforge.dev/docs/api-reference/tools/process-document)
- [localization](https://www.crawlforge.dev/docs/api-reference/tools/localization)
- [browser_session](https://www.crawlforge.dev/docs/api-reference/tools/browser-session)
- [track_changes](https://www.crawlforge.dev/docs/api-reference/tools/track-changes)
- [analyze_content](https://www.crawlforge.dev/docs/api-reference/tools/analyze-content)
- [extract_structured](https://www.crawlforge.dev/docs/api-reference/tools/extract-structured)
- [summarize_content](https://www.crawlforge.dev/docs/api-reference/tools/summarize-content)
- [crawl_deep](https://www.crawlforge.dev/docs/api-reference/tools/crawl-deep)
- [stealth_mode](https://www.crawlforge.dev/docs/api-reference/tools/stealth-mode)
- [scrape_with_actions](https://www.crawlforge.dev/docs/api-reference/tools/scrape-with-actions)
- [batch_scrape](https://www.crawlforge.dev/docs/api-reference/tools/batch-scrape)
- [search_web](https://www.crawlforge.dev/docs/api-reference/tools/search-web)
- [serp_rank](https://www.crawlforge.dev/docs/api-reference/tools/serp-rank)
- [reddit_search](https://www.crawlforge.dev/docs/api-reference/tools/reddit-search)
- [generate_llms_txt](https://www.crawlforge.dev/docs/api-reference/tools/generate-llms-txt)
- [deep_research](https://www.crawlforge.dev/docs/api-reference/tools/deep-research)
- [scrape_template](https://www.crawlforge.dev/docs/api-reference/tools/scrape-template)
- [extract_with_llm](https://www.crawlforge.dev/docs/api-reference/tools/extract-with-llm)
- [list_ollama_models](https://www.crawlforge.dev/docs/api-reference/tools/list-ollama-models)
- [scrape](https://www.crawlforge.dev/docs/api-reference/tools/scrape)
- [agent](https://www.crawlforge.dev/docs/api-reference/tools/agent)
- [get_batch_results](https://www.crawlforge.dev/docs/api-reference/tools/get-batch-results)
- [read_result](https://www.crawlforge.dev/docs/api-reference/tools/read-result)
- [All tools index](https://www.crawlforge.dev/docs/api-reference)
- [Hosted monitors API (cron schedule, signed webhooks, no per-monitor fee)](https://www.crawlforge.dev/docs/api-reference/monitors)
- [OpenAPI specification (3.1)](https://www.crawlforge.dev/openapi.json)
- [TypeScript SDK (npm install crawlforge-sdk)](https://github.com/mysleekdesigns/crawlforge-sdk-ts)
- [Python SDK (pip install crawlforge)](https://github.com/mysleekdesigns/crawlforge-sdk-python)
- [MCP configurator](https://www.crawlforge.dev/docs/integration/configure)
- [Agent onboarding (SKILL.md)](https://www.crawlforge.dev/agent-onboarding/SKILL.md)
- [MCP discovery file](https://www.crawlforge.dev/.well-known/mcp.json)

## Blog Posts
- [CrawlForge MCP v6.7–v6.12: Scrape Cloudflare-Protected Sites](https://www.crawlforge.dev/blog/crawlforge-v6-12-scrape-cloudflare-protected-sites-mcp-server)
- [CrawlForge MCP v6.6.0: Browser Sessions That Stay Open](https://www.crawlforge.dev/blog/crawlforge-v6-6-0-browser-sessions)
- [Market Research API for RevOps: One Question, Every Vendor](https://www.crawlforge.dev/blog/market-research-api-for-revops)
- [Market Research API for Consultants: Zero to Sourced](https://www.crawlforge.dev/blog/market-research-api-for-consultants)
- [Lead Enrichment API Without a Sales Call: First Lead Today](https://www.crawlforge.dev/blog/lead-enrichment-api-without-a-sales-call)
- [Lead Enrichment API Pay As You Go: Spend Only on Good Leads](https://www.crawlforge.dev/blog/lead-enrichment-api-pay-as-you-go)
- [Market Research API for Agencies: One Setup, Every Client](https://www.crawlforge.dev/blog/market-research-api-for-agencies)
- [Job Postings API for Consultants: Read the Board](https://www.crawlforge.dev/blog/job-postings-api-for-consultants)
- [Market Research API for Due Diligence: Test the Claims](https://www.crawlforge.dev/blog/market-research-api-for-due-diligence)
- [Competitor Monitoring API for RevOps: No Scraper to Run](https://www.crawlforge.dev/blog/competitor-monitoring-api-for-revops)
- [Competitor Monitoring API Pay As You Go: What a Check Costs](https://www.crawlforge.dev/blog/competitor-monitoring-api-pay-as-you-go)
- [Competitor Monitoring API for Consultants: One Per Client](https://www.crawlforge.dev/blog/competitor-monitoring-api-for-consultants)
- [Shipping an MCP Server Inside Your Product: Five Decisions](https://www.crawlforge.dev/blog/mcp-server-inside-your-product)
- [MCP Server for Due Diligence: What an Agent Can Verify](https://www.crawlforge.dev/blog/mcp-server-for-due-diligence)
- [Reddit Scraper for Claude Code: What Claude Actually Calls](https://www.crawlforge.dev/blog/reddit-scraper-for-claude-code)
- [Web Search MCP Server With Local LLM: Which Model Actually Runs](https://www.crawlforge.dev/blog/web-search-mcp-server-with-local-llm)
- [Web Search MCP Server: Self-Hosted With SearXNG](https://www.crawlforge.dev/blog/web-search-mcp-server-self-hosted)
- [CrawlForge MCP v5.2: Every Change Across 28 Tools, Tested Against Live Sites](https://www.crawlforge.dev/blog/crawlforge-mcp-v5-2-release-notes)
- [CrawlForge MCP v5.2.0: Shopify Product Data Without Parsing HTML](https://www.crawlforge.dev/blog/crawlforge-v5-2-0-shopify-product-template)
- [Web Scraping API Credits That Don't Expire](https://www.crawlforge.dev/blog/web-scraping-api-credits-that-dont-expire)
- [AI Crawler List 2026: Every Bot + Should You Block Them?](https://www.crawlforge.dev/blog/ai-crawlers-list-2026)
- [Answer 3 Questions, Get 1,000 Free CrawlForge Credits](https://www.crawlforge.dev/blog/survey-1000-free-credits)
- [CrawlForge MCP v5.1.0: Search Reddit Without the API](https://www.crawlforge.dev/blog/crawlforge-v5-1-0-reddit-search-tool)
- [Reddit API Alternatives That Still Work in 2026](https://www.crawlforge.dev/blog/reddit-api-alternatives-2026)
- [How to Scrape Reddit Without the API (2026)](https://www.crawlforge.dev/blog/scrape-reddit-without-api)
- [Reddit Data for AI Agents: The MCP Route](https://www.crawlforge.dev/blog/reddit-data-for-ai-agents)
- [The Web Scraping Keyword Gap: A 2026 SERP Study](https://www.crawlforge.dev/blog/web-scraping-keyword-gap-2026)
- [How to Run a Keyword Gap Analysis with an MCP Server](https://www.crawlforge.dev/blog/keyword-gap-analysis-mcp-server)
- [Agent Scraper: What It Is and How to Build One](https://www.crawlforge.dev/blog/agent-scraper)
- [CrawlForge v5.0.4: 34 Fixes From Live-Testing 27 MCP Tools](https://www.crawlforge.dev/blog/crawlforge-v5-0-4-live-testing-27-tools)
- [Is Web Scraping Legal in 2026? The EDPB's New AI Rules](https://www.crawlforge.dev/blog/is-web-scraping-legal-ai-2026)
- [CrawlForge v5.0.0: Security, Correctness, MCP Spec](https://www.crawlforge.dev/blog/crawlforge-v5-0-0-security-hardening-mcp-spec)
- [SSRF in MCP Servers: Why Scrapers Leak Cloud Secrets](https://www.crawlforge.dev/blog/mcp-server-ssrf-cloud-metadata-security)
- [How to Actually Use CrawlForge MCP in Claude Code](https://www.crawlforge.dev/blog/use-crawlforge-mcp-claude-code-terminal)
- [CrawlForge v4.8.0: Claude Skills That Auto-Activate](https://www.crawlforge.dev/blog/crawlforge-v4-8-0-claude-agent-skills)
- [CrawlForge vs Firecrawl vs Tavily vs Exa: Best Web Data API for AI Agents (2026)](https://www.crawlforge.dev/blog/crawlforge-vs-firecrawl-tavily-exa-web-data-api)
- [How to Build a Web-Scraping MCP Server in TypeScript (2026)](https://www.crawlforge.dev/blog/build-web-scraping-mcp-server)
- [How to Give ChatGPT Web Scraping with MCP Connectors (2026)](https://www.crawlforge.dev/blog/connect-chatgpt-web-scraping-mcp)
- [Web Scraping with Claude: The Complete Guide (2026)](https://www.crawlforge.dev/blog/web-scraping-with-claude-complete-guide)
- [Best MCP Servers for Web Scraping in 2026 (Top 8 Ranked)](https://www.crawlforge.dev/blog/best-mcp-servers-web-scraping-2026)
- [Best Web Scraping Tools for AI Agents in 2026](https://www.crawlforge.dev/blog/best-web-scraping-tools-ai-agents-2026)
- [How to Use Web Scraping Templates the Right Way](https://www.crawlforge.dev/blog/how-to-use-web-scraping-templates)
- [How to Scrape Websites with Claude Code (2026 Guide)](https://www.crawlforge.dev/blog/scrape-websites-with-claude-code)
- [How to Scrape Websites in Cursor IDE with CrawlForge MCP](https://www.crawlforge.dev/blog/scrape-websites-in-cursor-ide)
- [How to Scrape Websites in Zed AI with CrawlForge MCP](https://www.crawlforge.dev/blog/scrape-websites-in-zed-ai)
- [How to Scrape Websites with GitHub Copilot Agents in VS Code](https://www.crawlforge.dev/blog/scrape-websites-github-copilot-vs-code)
- [LlamaIndex Web Scraping Guide with CrawlForge MCP](https://www.crawlforge.dev/blog/llamaindex-web-scraping-guide)
- [CrawlForge v4.2.2 - New CLI + 3 Tools for Local AI Scraping](https://www.crawlforge.dev/blog/crawlforge-v4-2-2-cli-and-3-new-tools)
- [Web Scraping From the CLI - The CrawlForge CLI Guide](https://www.crawlforge.dev/blog/web-scraping-cli-complete-guide)
- [Extract Web Data With Local LLMs (Ollama + CrawlForge)](https://www.crawlforge.dev/blog/extract-data-with-local-llms-ollama)
- [Scrape Amazon, LinkedIn & 8 More Sites With One Tool](https://www.crawlforge.dev/blog/scrape-amazon-linkedin-github-templates)
- [CrawlForge MCP Is Now Live](https://www.crawlforge.dev/blog/crawlforge-mcp-is-now-live)
- [CrawlForge MCP Quickstart Guide](https://www.crawlforge.dev/blog/crawlforge-mcp-quickstart)
- [Install CrawlForge MCP in Claude Code - Beginner's Guide](https://www.crawlforge.dev/blog/install-crawlforge-mcp-claude-code-beginners-guide)
- [31 Web Scraping Tools in One MCP Server](https://www.crawlforge.dev/blog/18-web-scraping-tools-one-mcp-server)
- [5 Ways to Use CrawlForge with LangChain](https://www.crawlforge.dev/blog/5-ways-use-crawlforge-langchain)
- [CrawlForge vs Apify vs ScrapingBee Comparison](https://www.crawlforge.dev/blog/crawlforge-vs-apify-vs-scrapingbee-comparison)
- [How to Add Web Scraping to Claude Desktop](https://www.crawlforge.dev/blog/how-to-add-web-scraping-to-claude-desktop)
- [Building an AI Research Assistant with Claude MCP](https://www.crawlforge.dev/blog/building-ai-research-assistant-claude-mcp)
- [MCP vs REST - Why Native MCP Beats REST for Scraping](https://www.crawlforge.dev/blog/mcp-vs-rest-native-mcp-scraping-server)
- [Introducing Deep Research - AI-Powered Analysis](https://www.crawlforge.dev/blog/introducing-deep-research-ai-powered-analysis)
- [Web Scraping for AI Training Data - 2026 Guide](https://www.crawlforge.dev/blog/web-scraping-ai-training-data-2025-guide)
- [Welcome to CrawlForge](https://www.crawlforge.dev/blog/welcome-to-crawlforge)
- [CrawlForge vs Firecrawl - MCP Web Scraper Comparison](https://www.crawlforge.dev/blog/crawlforge-vs-firecrawl-mcp-web-scraper-comparison)
- [Build a Competitive Intelligence Agent with Claude and CrawlForge](https://www.crawlforge.dev/blog/build-competitive-intelligence-agent-claude-crawlforge)
- [Stealth Mode Scraping - Bypass Anti-Bot Detection](https://www.crawlforge.dev/blog/stealth-mode-scraping-bypass-anti-bot-detection)
- [Deep Research Automation - 10 Hours to 10 Minutes](https://www.crawlforge.dev/blog/deep-research-automation-10-hours-to-10-minutes)
- [Complete Guide to MCP Web Scraping](https://www.crawlforge.dev/blog/complete-guide-mcp-web-scraping)
- [CrawlForge with n8n - Workflow Automation Guide](https://www.crawlforge.dev/blog/crawlforge-n8n-integration-guide)
- [CrawlForge with Vercel AI SDK](https://www.crawlforge.dev/blog/crawlforge-vercel-ai-sdk-guide)
- [CrawlForge with Windsurf IDE](https://www.crawlforge.dev/blog/crawlforge-windsurf-integration-guide)
- [CrawlForge with Cline (VS Code)](https://www.crawlforge.dev/blog/crawlforge-cline-integration-guide)
- [CrawlForge with OpenAI Agents SDK](https://www.crawlforge.dev/blog/crawlforge-openai-agents-integration)
- [CrawlForge with Anthropic Claude API](https://www.crawlforge.dev/blog/crawlforge-anthropic-claude-api-guide)
- [Build an AI-Powered Price Monitoring System](https://www.crawlforge.dev/blog/ai-price-monitoring-system-guide)
- [Automate SEO Audits with CrawlForge MCP](https://www.crawlforge.dev/blog/automating-seo-audits-crawlforge)
- [Real-Time Competitive Intelligence with AI Agents](h

# CrawlForge security contact. The disclosure policy is at https://www.crawlforge.dev/security#disclosure
Contact: mailto:simon@crawlforge.dev
Expires: 2027-09-07T00:00:00.000Z
Preferred-Languages: en, es, zh, ms
Canonical: https://www.crawlforge.dev/.well-known/security.txt
Policy: https://www.crawlforge.dev/security#disclosure