sitedex.dev is built with React, hosted on Cloudflare. It measures traffic with Cloudflare Web Analytics.
Built with React and React Router, hosted on Cloudflare. Domain registered in 2026.
| Technology | Version | First seen |
|---|---|---|
| JavaScript Frameworks & Libraries2 | ||
| – | 8 Nov 2026 | |
| – | 8 Nov 2026 | |
| Frontend Build Tools & CSS1 | ||
| – | 8 Nov 2026 | |
| CDN & Hosting1 | ||
| – | 8 Nov 2026 | |
| Analytics & Tag Management1 | ||
| – | 8 Nov 2026 | |
| Security, WAF & Bot Protection1 | ||
| – | 8 Nov 2026 | |
Registration from the registry's RDAP record, the certificate the site serves, and its rank in public traffic lists.
The crew hasn’t read this site’s colors yet. We read a palette when we capture a site’s screenshot, busiest sites first. Meanwhile, browse palettes from other sites.
Counted from every stack change we've recorded: of the sites we watched running one of this site's tools, how many went on to add another. A pattern across sites, not a forecast for this one.
route1.mx.cloudflare.netroute3.mx.cloudflare.netroute2.mx.cloudflare.nettim.ns.cloudflare.comliz.ns.cloudflare.comgoogle-site-verification=LOxgm0RJ0zExgKynFMGftueqvDPZ7FAlcbX-OMYHeR4v=spf1 include:_spf.mx.cloudflare.net ~allv=DMARC1; p=reject;# robots.txt for sitedex.dev
#
# Sitedex indexes public websites for AI agents — we depend on AI crawlers
# being allowed to access content, so we explicitly allow the major ones
# here and declare a Cloudflare-compatible Content-Signal policy. Same
# pattern we recommend to our users via the fix list at
# app.sitedex.dev/sites/{domain}#fix-ai_crawler_allow.
#
# This file is deliberately permissive — there's nothing on sitedex.dev
# that shouldn't be indexed.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
# Explicit allow stanzas for major AI crawlers. Lets them tell, from a
# quick robots.txt fetch, that they're welcome here without having to
# fall through the wildcard.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: CCBot
Allow: /
# Meta's current crawler tokens (developers.facebook.com/docs/sharing/webmasters/web-crawlers);
# FacebookBot is retired. One group, several user-agents (RFC 9309 §2.2.1).
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: meta-webindexer
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Sitemap: https://sitedex.dev/sitemap.xml
# Sitedex
> Sitedex is two products on one engine. For AI agents: a live web index — every public website crawled, scored, and served as raw markdown over MCP, a REST API, and a CLI. For site owners: an audit that knows what AI agents need to read a site clearly and finds exactly what's missing — vague claims, contradictions, content gaps, missing technical signals — then ships the fix list and the files AI agents look for. Reads are free. Audits cost 1 credit. Anyone can index a new site for free.
## Surfaces
- **MCP (hosted)** — `https://mcp.sitedex.dev/mcp`. Connect from any MCP client over Streamable HTTP. No setup.
- **MCP (per-site)** — `https://mcp.sitedex.dev/s/{slug}/mcp`. One endpoint per site, with the site already bound. Slug is the URL form (`fireflies-ai`, not `fireflies.ai`).
- **MCP (local)** — `npx -y @sitedex/cli mcp`. Adds audit, credit, and account tools once you sign in.
- **REST API** — `https://api.sitedex.dev/v1/*`. Authed with a Bearer key from [app.sitedex.dev/settings/api-keys](https://app.sitedex.dev/settings/api-keys).
- **CLI** — `npm install -g @sitedex/cli`. Sign in once, then run audits, search the index, and pull reports from the terminal.
## Tools (hosted MCP — no sign-in)
- **`sitedex_search`** — search every indexed site, scope to one domain with `site`.
- **`sitedex_list_sites`** — every indexed domain with category, page count, score.
- **`sitedex_get_site`** — site profile and headline score.
- **`sitedex_list_pages`** — paths Sitedex has on a domain.
- **`sitedex_get_page`** — full markdown of one page.
- **`sitedex_get_visibility_score`** — full audit: which buyer questions the site answers, which it misses, and what to fix.
- **`sitedex_index_url`** — submit a new domain. First-time indexing is free.
## Tools (local MCP — adds audit + account)
Run `npx -y @sitedex/cli mcp` and sign in to also get:
- **`sitedex_audits_create`** — run a fresh audit. 1 credit. Anyone can index a new site free, no account needed.
- **`sitedex_audits_get` / `sitedex_audits_list`** — your audit history.
- **`sitedex_credits_balance` / `sitedex_whoami`** — your wallet and account.
- **`sitedex_login_start` / `sitedex_login_finish` / `sitedex_logout`** — sign in from inside the agent.
## Tools (per-site MCP)
Same surface, smaller scope. Each tool acts on the bound site only — no `site` argument:
- **`search_site_content`**, **`get_site_info`**, **`list_site_pages`**, **`get_site_page`**
## REST API
Base URL: `https://api.sitedex.dev`. Auth: `Authorization: Bearer sdx_…`.
Errors:
```json
{ "error": { "type": "...", "code": "...", "message": "..." } }
```
### Public
| Method | Path | Description |
|---|---|---|
| `POST` | `/v1/audits` | Run an audit. |
| `GET` | `/v1/sites/:domain` | Site profile and headline score. |
| `GET` | `/v1/sites/:domain/report` | Full audit report. |
| `POST` | `/search` | Search the index. |
| `GET` | `/sites/:domain/pages/:path` | One page as markdown. |
### Signed in
| Method | Path | Description |
|---|---|---|
| `GET` | `/v1/auth/whoami` | Account info. |
| `GET` | `/v1/audits` / `/v1/audits/:id` | Audit history and a single audit. |
| `GET` | `/v1/sites` / `/v1/sites/:domain/audits` | Sites you've claimed and their audit history. |
| `GET` | `/v1/credits/balance` / `/v1/credits/ledger` | Wallet and transactions. |
| `POST` | `/v1/credits/purchase` | Buy a credit. |
| `*` | `/v1/auth/tokens` | Manage API keys. |
Pre-`/v1` paths return a `Sunset` header pointing at the new route. They stop responding on 2026-08-04.
## Pricing
- **Reads** — free. Search, browse, fetch markdown, view the score.
- **Index a new site** — free, no account needed. Anyone can submit a URL Sitedex doesn't have yet.
- **Audits** — 1 credit each. Every new account gets 1 free credit. Credits never expire.
- **Credit packs** — Single $5 (1 credit · $5.00/credit), Starter $50 (15 credits · $3.33/credit), Pro $100 (40 credits · $2.50/credit). Bigger packs cost less per credit; same product everyone gets.
- **No subscription, no seats.**
## Site owners
[Index your site](https://app.sitedex.dev/audits/new). The first audit is on us.
[Claim your domain](https://app.sitedex.dev/team/domains) with a DNS TXT record to manage it, see all audit history, and unlock deeper crawls.
The site page at `https://app.sitedex.dev/sites/{domain}` is the same page buyers see. Score, gaps, citations, and a fix list. No sign-in required to view.
### Files Sitedex builds for you
Generated from your indexed pages, downloadable from your site's fix list. Same data the audit produces, packaged for install.
- **`llms.txt`** — the file ChatGPT, Claude, and Perplexity read first. Free starter for any URL at [sitedex.dev/llms-txt](https://sitedex.dev/llms-txt); enhanced per-domain version once indexed.
- **AI-crawler `robots.txt` block** — per-bot rules for the crawlers that answer buyers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, meta-webindexer, Applebot) and, separately, the training crawlers (GPTBot, ClaudeBot, Google-Extended, meta-externalagent). Append to your existing `robots.txt`.
- **MCP server-card** — `/.well-known/mcp/server-card.json`. Points agents that probe your domain at Sitedex's hosted endpoint with your site bound. No server to run.
- **WebMCP manifest** — `/.well-known/webmcp.json` + a one-line script tag. Discovery for the W3C draft `navigator.modelContext` runtime registrations.
- **Organization JSON-LD** — `<script type="application/ld+json">` block drop-in for `<head>`. Drives AI Knowledge Graph and how agents introduce your company.
- **Canonical / meta description / `<html lang>` snippets** — three one-line fixes most sites miss. Copy-paste ready.
## Free tools (browser, no sign-in)
Two standalone diagnostic pages. Paste a URL, get an answer, nothing stored. Both run on SitedexBot, our signed crawler.
- **fetchcheck** — [sitedex.dev/tools/fetchcheck](https://sitedex.dev/tools/fetchcheck). Sends one signed SitedexBot request to any site and shows exactly what came back: allowed, blocked, challenged, redirected, or a page with no readable content. Answers "can a trusted crawler get in, and is your content actually there?" before you run a full audit.
- **keycheck** — [sitedex.dev/tools/keycheck](https://sitedex.dev/tools/keycheck). Lints your own Web Bot Auth signing-key directory, key by key, for the mistakes that stop verification — wrong key type or curve, an expiring key, a leaked private key.
SitedexBot's published identity and public signing keys: [sitedex.dev/bot](https://sitedex.dev/bot).
## How it works
We crawl. We extract markdown. We embed it. We serve it.
The audit is where Sitedex's intelligence shows up. For each site we generate a bank of buyer questions specific to its category — the actual questions a real customer asks during evaluation, not generic "what does X do" prompts. We then ask a real LLM to answer each one from the site's content, and grade the answer on a sharp diagnosis taxonomy:
- **answered** — content addresses the question specifically. Cite-worthy.
- **vague claim** — topic mentioned, specific detail (number, threshold, policy edge case) missing.
- **collapsed content** — page exists, body hidden behind JavaScript that didn't render.
- **missing page** — no dedicated page covers the topic.
- **contradictory pages** — two or more pages disagree; agents pick one at random.
- **genuinely absent** — topic isn't anywhere on the site.
Every miss gets a fix hint a writer or engineer can ship. The protocol layer separately checks the technical signals that decide whether AI agents can read the site at all — `llms.txt`, sitemap, robots-for-AI-crawlers, MCP card, WebMCP, JSON-LD.
Score: `0.7 × content + 0.3 × protocol`, capped so protocol can't more than double a weak content score.
Levels: 80+ Agent-Native, 60+ Agent-Ready, 40+ Agent-Accessible, 20+ Basic Presence, below 20 Invisible.
## Discovery
- [Install guide](https://sitedex.dev/docs)
- [OpenAPI 3.1 spec](https://sitedex.dev/openapi.json)
- [MCP server card](https://sitedex.dev/.well-known/mcp.json)
- [Agent discovery](https://sitedex.dev/.well-known/agentroot.json)
- [CLI on npm](https://www.npmjs.com/package/@sitedex/cli)
- [Long-form docs for AI agents](https://sitedex.dev/llms-full.txt)
## Contact
`hello@sitedex.dev`