robots.txt
See which AI crawlers other sites block →# Kingpin Marketplace - Robots.txt
# Last Updated: 2026-09-16
#
# This robots.txt protects user privacy, prevents content scraping,
# and ensures search engines index only public, appropriate content.
#
# Why every group that blocks /api/ also carries "Allow: /api/og/":
# Kingpin's OpenGraph and Twitter card images are generated on demand at
# /api/og/listing/{id}, /api/og/profile/{username}, /api/og/drop/{id} and
# /api/og/site. A blanket "Disallow: /api/" therefore told Googlebot and every
# link-preview bot (Twitter/X, Facebook, LinkedIn, Slack, Discord, Telegram,
# WhatsApp) that it may not fetch the share image for a listing or a shopfront,
# so shared Kingpin links rendered a card with no picture. The longer, more
# specific Allow wins over the shorter Disallow for Google and Bing.
#
# These routes are public and read-only by design, and are already exempted
# from the edge bot block by isApiBotExemptRoute() in config/security.ts —
# robots.txt was the one place still contradicting that.
#
# The AI-crawler blocks further down are deliberate. /llms.txt is published for
# assistants fetching a page on a person's behalf, not for training crawlers.
# ============================================================================
# DEFAULT POLICY - Allow Most Good Bots, Block Known Bad Actors
# ============================================================================
User-agent: *
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/profile/*/edit
Disallow: /*/settings/
Disallow: /*/manage/
Disallow: /*/manage/billing/
Disallow: /*/manage/orders/
Disallow: /*/manage/addresses/
Disallow: /*/auth/
Disallow: /*/admin/
Disallow: /*/marketplace/dashboard/
Disallow: /*/marketplace/listings/new
Disallow: /*/marketplace/listings/*/update
Disallow: /*/marketplace/listings/*/manage
Disallow: /*/notifications/
Disallow: /*/member$
Disallow: /*/member/
Disallow: /static/private/
Crawl-delay: 2
# Allow public marketplace pages
Allow: /*/marketplace/listings$
Allow: /*/marketplace/listings?*
Allow: /*/marketplace/listings/*/$
Allow: /*/u/
Allow: /$
# Locale home pages. Matches every supported locale — never hard-code
# individual locale segments here, they go stale as locales are added.
Allow: /*/$
# ============================================================================
# GOOGLE - Standard Access with Rate Limiting
# ============================================================================
User-agent: Googlebot
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/profile/*/edit
Disallow: /*/settings/
Disallow: /*/manage/
Disallow: /*/manage/billing/
Disallow: /*/manage/orders/
Disallow: /*/manage/addresses/
Disallow: /*/auth/
Disallow: /*/admin/
Disallow: /*/dashboard/
Disallow: /*/marketplace/listings/new
Disallow: /*/marketplace/listings/*/update
Disallow: /*/marketplace/listings/*/manage
Disallow: /*/notifications/
Disallow: /*/member$
Disallow: /*/member/
Crawl-delay: 1
# Allow public content
Allow: /*/marketplace/listings$
Allow: /*/marketplace/listings?*
Allow: /*/marketplace/listings/*/$
Allow: /*/u/
# ============================================================================
# GOOGLE IMAGE SEARCH - Controlled Image Indexing
# ============================================================================
User-agent: Googlebot-Image
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/profile/*/edit
Disallow: /*/auth/
Disallow: /*/admin/
Crawl-delay: 2
# Allow listing images only
Allow: /uploads/listings/
Allow: /public/branding/
# ============================================================================
# BING - Similar to Google
# ============================================================================
User-agent: Bingbot
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/profile/*/edit
Disallow: /*/settings/
Disallow: /*/manage/
Disallow: /*/manage/billing/
Disallow: /*/manage/orders/
Disallow: /*/manage/addresses/
Disallow: /*/auth/
Disallow: /*/admin/
Disallow: /*/dashboard/
Disallow: /*/marketplace/listings/new
Disallow: /*/marketplace/listings/*/update
Disallow: /*/marketplace/listings/*/manage
Disallow: /*/notifications/
Disallow: /*/member$
Disallow: /*/member/
Crawl-delay: 2
# ============================================================================
# AI CRAWLERS - BLOCK TRAINING, ALLOW SEARCH
# ============================================================================
#
# The distinction that matters commercially: a TRAINING crawler takes Kingpin's
# listings to build a model and sends nothing back. A SEARCH or user-directed
# fetcher is how a buyer who asked an assistant "where can I buy a used
# Blackriver deck" ends up on a Kingpin page, cited. Kingpin is a marketplace
# that needs buyers, so the first stays blocked and the second is welcome.
#
# Retired identifiers (anthropic-ai, Claude-Web) are kept below deliberately:
# they cost nothing, and removing them would silently drop coverage if any
# client still sends them. The live ones are listed alongside.
# OpenAI GPTBot
User-agent: GPTBot
Disallow: /
# Google's AI Crawler
User-agent: Google-Extended
Disallow: /
# Common Crawl (used for AI training)
User-agent: CCBot
Disallow: /
# Anthropic — training crawler (current identifier)
User-agent: ClaudeBot
Disallow: /
# Anthropic — retired identifier, kept for coverage
User-agent: anthropic-ai
Disallow: /
# Meta AI
User-agent: FacebookBot
Disallow: /
# Bytedance/TikTok AI
User-agent: Bytespider
Disallow: /
# Amazon AI
User-agent: Amazonbot
Disallow: /
# Cohere AI
User-agent: cohere-ai
Disallow: /
# Perplexity AI
# Anthropic — retired identifier, kept for coverage
User-agent: Claude-Web
Disallow: /
# Apple — AI training opt-out (Applebot itself is search; see below)
User-agent: Applebot-Extended
Disallow: /
# ----------------------------------------------------------------------------
# AI SEARCH & USER-DIRECTED FETCHERS - ALLOWED
# ----------------------------------------------------------------------------
# These build answer-engine indexes or fetch a page because a person just asked
# for it. Both cite and link, which is how a buyer reaches a listing. They get
# the same access as any search engine: public pages yes, private areas no.
# The OG card images are allowed so a cited link renders with its picture.
# OpenAI - search index (NOT GPTBot, which trains and stays blocked)
User-agent: OAI-SearchBot
Allow: /api/og/
Disallow: /api/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/settings/
Disallow: /*/manage/
# OpenAI - fetches a page because a user asked ChatGPT for it
User-agent: ChatGPT-User
Allow: /api/og/
Disallow: /api/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/settings/
Disallow: /*/manage/
# Anthropic - search index (NOT ClaudeBot, which trains and stays blocked)
User-agent: Claude-SearchBot
Allow: /api/og/
Disallow: /api/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/settings/
Disallow: /*/manage/
# Anthropic - fetches a page because a user asked Claude for it
User-agent: Claude-User
Allow: /api/og/
Disallow: /api/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/settings/
Disallow: /*/manage/
# Perplexity - answer-engine index; its citations are a traffic source
User-agent: PerplexityBot
Allow: /api/og/
Disallow: /api/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/settings/
Disallow: /*/manage/
# Perplexity - fetches a page because a user asked for it
User-agent: Perplexity-User
Allow: /api/og/
Disallow: /api/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/settings/
Disallow: /*/manage/
# Apple - Siri and Spotlight search (Applebot-Extended trains and is blocked)
User-agent: Applebot
Allow: /api/og/
Disallow: /api/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/settings/
Disallow: /*/manage/
# Diffbot
User-agent: Diffbot
Disallow: /
# Omgili
User-agent: omgili
Disallow: /
# ============================================================================
# SCRAPERS & CONTENT THIEVES - BLOCK ALL
# ============================================================================
# AhrefsBot (SEO crawler, often aggressive)
User-agent: AhrefsBot
Disallow: /
# SemrushBot (SEO crawler)
User-agent: SemrushBot
Disallow: /
# MJ12bot (Majestic SEO)
User-agent: MJ12bot
Disallow: /
# DotBot (Moz crawler)
User-agent: DotBot
Disallow: /
# BLEXBot
User-agent: BLEXBot
Disallow: /
# PetalBot
User-agent: PetalBot
Disallow: /
# Yandex (Russian search engine - often used for scraping)
User-agent: YandexBot
Disallow: /
# Mail.ru crawler
User-agent: Mail.RU_Bot
Disallow: /
# Scrapy
User-agent: Scrapy
Disallow: /
# Python requests
User-agent: python-requests
Disallow: /
# Wget
User-agent: Wget
Disallow: /
# curl
User-agent: curl
Disallow: /
# ============================================================================
# ARCHIVE BOTS - Block Private Content, Allow Public
# ============================================================================
# Internet Archive Wayback Machine
User-agent: ia_archiver
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/profile/*/edit
Disallow: /*/settings/
Disallow: /*/manage/
Disallow: /*/auth/
Disallow: /*/admin/
Allow: /*/marketplace/listings$
Allow: /$
# ============================================================================
# SOCIAL MEDIA LINK PREVIEW BOTS - Allow for Sharing
# ============================================================================
# Twitter/X
User-agent: Twitterbot
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/auth/
Disallow: /*/admin/
Allow: /*/marketplace/listings/*/$
Allow: /*/u/
Crawl-delay: 1
# Facebook
User-agent: facebookexternalhit
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/auth/
Disallow: /*/admin/
Allow: /*/marketplace/listings/*/$
Allow: /*/u/
Crawl-delay: 1
# LinkedIn
User-agent: LinkedInBot
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/auth/
Disallow: /*/admin/
Allow: /*/marketplace/listings/*/$
Allow: /*/u/
Crawl-delay: 1
# Slack
User-agent: Slackbot
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/auth/
Disallow: /*/admin/
Allow: /*/marketplace/listings/*/$
Crawl-delay: 1
# Discord
User-agent: Discordbot
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/auth/
Disallow: /*/admin/
Allow: /*/marketplace/listings/*/$
Crawl-delay: 1
# Telegram
User-agent: TelegramBot
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/auth/
Disallow: /*/admin/
Allow: /*/marketplace/listings/*/$
Crawl-delay: 1
# WhatsApp
User-agent: WhatsApp
Disallow: /api/
Allow: /api/og/
Disallow: /*/messages/
Disallow: /*/conversations/
Disallow: /*/auth/
Disallow: /*/admin/
Allow: /*/marketplace/listings/*/$
Crawl-delay: 1
# ============================================================================
# SITEMAPS
# ============================================================================
Sitemap: https://www.kingpinmarket.com/sitemap.xml
llms.txt
Compare with other sites' llms.txt files →# Kingpin Market
> Kingpin is an independent online marketplace for fingerboards and fingerboard gear, where riders, collectors and brands buy and sell decks, completes, wheels, trucks, tuning parts, grip, obstacles and accessories. Sellers can run their own shopfront on the platform.
Kingpin is a specialist marketplace for the fingerboarding scene, not a general
store. The catalogue is small and changes as sellers list and sell gear, so the
sitemap below — not this file — is the live answer to what is currently for
sale. Listings cover both new and secondhand items, graded from new through
heavily used.
Sellers list gear either as individual members or as a shopfront with its own
public storefront page, themes and scheduled releases. Checkout and seller
payouts run on Stripe and Stripe Connect. The site is published in 7 languages
(en, ko, de, es, fr, pt, it); every canonical URL is prefixed with its locale,
for example `/en/marketplace/listings`.
Public URL shapes worth knowing: `/en/marketplace/listings/{id}` is a single
item for sale, `/en/u/{username}` is a public member profile,
`/en/shop/{username}` is a seller's public shopfront, and
`/en/marketplace/brands/{slug}` is a brand's page: that brand's gear listed
for resale on Kingpin, what has sold, and links to the brand's own website and
Instagram. Kingpin is not the brand and is not affiliated with it; to buy new
stock, send people to the brand's own site, which the brand page links to.
## Marketplace
- [Browse listings](https://www.kingpinmarket.com/en/marketplace/listings): Every item currently for sale, filterable by category, condition and price.
- [Brand directory](https://www.kingpinmarket.com/en/marketplace/brands): Every fingerboard brand in Kingpin's catalog, A to Z, with the brands that have gear for sale right now first. Each brand links to its own brand page.
- [Rare and collectible](https://www.kingpinmarket.com/en/marketplace/rare): Listings flagged as rare or hard to find.
- [Drops](https://www.kingpinmarket.com/en/drop): Timed community drops and Grail Draws.
- [Competitions](https://www.kingpinmarket.com/en/competitions): Community fingerboard competitions hosted on Kingpin.
## Buying
- [Getting started](https://www.kingpinmarket.com/en/guides/getting-started): How to create an account, browse and buy.
- [Buyer protection](https://www.kingpinmarket.com/en/guides/buyer-protection): What is covered when an order goes wrong, and how to open a dispute.
- [Fingerboard components explained](https://www.kingpinmarket.com/en/guides/buyer/fingerboard-components): Reference for decks, trucks, wheels, tuning and grip, and how parts fit together.
- [Currency and pricing](https://www.kingpinmarket.com/en/guides/currency): How prices are shown and charged across countries.
## Selling and shopfronts
- [Sell fingerboards on Kingpin](https://www.kingpinmarket.com/en/sell): What it costs to sell, how long listing takes, how sellers are paid, and an honest account of how small the marketplace currently is.
- [For shop owners](https://www.kingpinmarket.com/en/for-shop-owners): Overview of running a seller shopfront on Kingpin.
- [Seller guide](https://www.kingpinmarket.com/en/guides/seller-success): How to list gear, price it and get it sold.
- [Fees](https://www.kingpinmarket.com/en/guides/fees): Authoritative explanation of what a seller is charged. Use this page rather than quoting a rate from memory.
- [Payouts](https://www.kingpinmarket.com/en/guides/payouts): How sellers are paid out, and the supported payout countries.
- [Shopfront guides](https://www.kingpinmarket.com/en/guides/shop): Index of the seller shopfront documentation, including themes, custom pages, releases and custom domains.
- [Shop releases](https://www.kingpinmarket.com/en/guides/shop/releases): Scheduled seller drops: preorders, restocks and per-buyer limits.
## About and policies
- [About Kingpin](https://www.kingpinmarket.com/en/about): What the marketplace is and who runs it.
- [All guides](https://www.kingpinmarket.com/en/guides): Full index of buyer, seller and shopfront documentation.
- [Marketplace policies](https://www.kingpinmarket.com/en/guides/policies): Selling rules, prohibited items and the notice period for fee changes.
- [Kingpin in Korea](https://www.kingpinmarket.com/en/korea): Korea-specific marketplace information, including local payouts.
- [Terms of service](https://www.kingpinmarket.com/en/legal/terms)
- [Privacy policy](https://www.kingpinmarket.com/en/legal/privacy)
- [Cookie policy](https://www.kingpinmarket.com/en/legal/cookies)
## Optional
- [Kingpin Magazine](https://magazine.kingpinmarket.com/en): Separate editorial site covering the fingerboard scene. It owns its own sitemap and is not part of the marketplace.
- [Sitemap index](https://www.kingpinmarket.com/sitemap.xml): Authoritative, live list of indexable Kingpin URLs, including every current listing, brand page and public shopfront.
- [Brand pages sitemap](https://www.kingpinmarket.com/sitemap-brands.xml): Every active catalog brand's page, one URL per brand, stocked brands first.
---
Site: https://www.kingpinmarket.com/
Contact: support@kingpinmarket.com
Note for automated clients: Kingpin blocks AI training crawlers in
`/robots.txt` and at the edge. This file is published for assistants fetching a
page on behalf of a person who asked about Kingpin; it is not a grant of
permission to crawl the site for training data.