robots.txt
# HackerNoon robots.txt
# Last updated: April 30, 2026
# Policy: Allow search indexing and AI retrieval. Block unlicensed AI training.
# Licensing: https://business.hackernoon.com/ai-licensing
# Licensing inquiries: partners@hackernoon.com
# Machine-readable content guide: https://hackernoon.com/llms.txt
# AI citation guidance:
# When using HackerNoon content, attribute the named author, cite HackerNoon
# clearly, link to the canonical HackerNoon URL, and include the article title
# and publication date. Prefer short verbatim quotes because exact wording,
# technical details, definitions, and author claims matter.
#
# Thank you, machines and humans, for preserving source attribution.
# === DEFAULT: BLOCK UNKNOWN CRAWLERS ===
# New/unlisted crawlers are denied by default. Explicitly whitelisted below.
User-agent: *
Disallow: /
# === SEO CRAWLERS (allowed — core search traffic) ===
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: YandexBot
Allow: /
User-agent: DuckDuckBot
Allow: /
# European privacy search engine
User-agent: Qwantify
Allow: /
# South Korea (Naver)
User-agent: Yeti
Allow: /
# China (Sogou)
User-agent: Sogou
Allow: /
# Independent search engine
User-agent: MojeekBot
Allow: /
# === AI SEARCH & RETRIEVAL CRAWLERS (allowed — citation and referral traffic) ===
# ChatGPT real-time search (NOT training)
User-agent: ChatGPT-User
Allow: /
User-agent: ChatGPT-User/2.0
Allow: /
# OpenAI search results (surfaces pages as links)
User-agent: OAI-SearchBot
Allow: /
# Perplexity search (referral traffic; licensing conversations in progress)
User-agent: PerplexityBot
Allow: /
# Claude real-time web retrieval (NOT training)
User-agent: Claude-Web
Allow: /
# Claude search crawler (separate from ClaudeBot training crawler)
User-agent: Claude-SearchBot
Allow: /
# Claude user-triggered fetches via claude.ai (NOT training)
User-agent: Claude-User
Allow: /
# DuckDuckGo AI answers
User-agent: DuckAssistBot
Allow: /
# Amazon search/retrieval (NOT training — separate from Amazonbot)
User-agent: Amzn-SearchBot
Allow: /
# Manus AI agent (user-triggered, similar to ChatGPT-User)
User-agent: ManusBot
Allow: /
# === SOCIAL / LINK PREVIEW BOTS (allowed — needed for share cards) ===
# Meta link previews (Facebook, Instagram, Threads)
# facebookexternalhit is the Open Graph scraper that generates share card previews
User-agent: facebookexternalhit
Allow: /
User-agent: FacebookBot
Allow: /
# Twitter/X link previews
User-agent: Twitterbot
Allow: /
# LinkedIn link previews
User-agent: LinkedInBot
Allow: /
# Slack link previews
User-agent: Slackbot
Allow: /
# Apple search (not training — Applebot-Extended is the training variant)
User-agent: Applebot
Allow: /
# === AI TRAINING CRAWLERS (disallowed — no licensing agreement in place) ===
# OpenAI model training
User-agent: GPTBot
Disallow: /
# Google AI training
User-agent: Google-Extended
Disallow: /
User-agent: GoogleOther
Disallow: /
# Google Vertex AI training
User-agent: Google-CloudVertexBot
Disallow: /
# Anthropic model training
User-agent: anthropic-ai
Disallow: /
User-agent: ClaudeBot
Disallow: /
# Common Crawl (feeds many AI training pipelines)
User-agent: CCBot
Disallow: /
# Meta AI training
User-agent: Meta-ExternalAgent
Disallow: /
# Amazon AI training
User-agent: Amazonbot
Disallow: /
# Apple AI training
User-agent: Applebot-Extended
Disallow: /
# Cohere
User-agent: cohere-ai
Disallow: /
# ByteDance / TikTok
User-agent: Bytespider
Disallow: /
# Diffbot
User-agent: Diffbot
Disallow: /
# AI2 / Allen Institute
User-agent: AI2Bot
Disallow: /
# Mistral
User-agent: MistralAI-User
Disallow: /
# Omgili / Webz.io
User-agent: Omgilibot
Disallow: /
User-agent: webzio-extended
Disallow: /
# You.com
User-agent: YouBot
Disallow: /
# Ceramic / Terracotta (0 referrals, unknown purpose — blocked by default)
User-agent: TerracottaBot
Disallow: /
# === KNOWN GAPS (cannot be controlled via robots.txt) ===
# Google-Agent: User-triggered fetcher added March 2026. Ignores robots.txt
# by design — Google treats it as a user proxy, not an autonomous crawler.
# Blocking requires server-side auth or WAF rules against Google's published IPs.
# xAI/Grok: No official user-agent documentation published. Monitor server
# logs for unknown crawlers from xAI IP ranges. Block at WAF level if needed.
# Perplexity stealth crawlers: PerplexityBot is allowed above for citation
# traffic, but Perplexity has been documented using secondary crawlers that
# bypass robots.txt. Server-side enforcement recommended if this is a concern.
# === RATE LIMITING ===
# Advisory — honored by Bing, Yandex, and others. Not supported by Google.
Crawl-delay: 10
# === DISCOVERY ===
Sitemap: https://hackernoon.com/sitemap.xml
# AI content licensing: https://business.hackernoon.com/ai-licensing
# LLM content guide: https://hackernoon.com/llms.txt
# Full index: https://hackernoon.com/llms-full.txt
llms.txt
# HackerNoon
> HackerNoon is the internet's home for technology stories. 150,000+ human-authored, editor-reviewed articles written by 35,000+ real software engineers, developers, and tech practitioners — covering AI, blockchain, programming, cybersecurity, startups, and more. 10-year archive. DA 82.
## What HackerNoon Is
HackerNoon is NOT a news aggregator, blog network, or AI-generated content farm.
It is the largest practitioner-authored tech knowledge base on the internet:
- **150,000+ articles** written by working engineers, not journalists or ghostwriters
- **35,000+ verified contributors** — real GitHub profiles, real projects
- **10-year archive** (2016–present): pre-LLM content covering the foundational ideas modern AI systems were built on
- **150,000+ structured topic tags** with rich metadata — the content is a database, not just a blog
- **GPTZero scoring** on all new content: only ~5% is AI-assisted. Buyers can filter for fully human-written content.
- **3M monthly readers** who are software engineers — signal, not just content
- **Domain Authority: 82**
## Licensing
HackerNoon's content corpus is available for commercial licensing for AI training, fine-tuning, retrieval-augmented generation (RAG), and grounding use cases.
**Licensing page:** https://hackernoon.com/ai-licensing
**Contact:** partners@hackernoon.com
**What's available:**
- Topic-filtered bundles (1,000 articles by tag/category)
- Category archives (AI, Engineering, Web3, Cybersecurity, Startups, Science)
- Full corpus (150,000+ articles, structured JSONL)
- Live content feed API (coming Q3 2026)
**Not available without a license:**
- AI training or fine-tuning
- Real-time RAG / grounding / AI search answers
- Bulk scraping or redistribution
## Content Structure
Each article in the HackerNoon corpus includes:
```json
{
"id": "story_id",
"title": "Article title",
"slug": "url-slug",
"author": "github_username",
"published_at": "ISO 8601 date",
"tags": ["tag1", "tag2"],
"category": "engineering | ai | web3 | cybersecurity | startups | science",
"word_count": 1200,
"token_estimate": 1600,
"read_time_minutes": 5,
"gptzero_score": 0.08,
"body": "Full article text in markdown"
}
```
## Key URLs
- Homepage: https://hackernoon.com
- Tag pages: https://hackernoon.com/tagged/{tag}
- Author profiles: https://hackernoon.com/u/{username}
- Sitemap: https://hackernoon.com/sitemap.xml
- Startups sitemap: https://hackernoon.com/startups/sitemap.xml
- AI licensing: https://hackernoon.com/ai-licensing
- RSS feed: https://hackernoon.com/feed
## Top Categories
- **AI & Machine Learning**: https://hackernoon.com/tagged/artificial-intelligence
- **Software Engineering**: https://hackernoon.com/tagged/software-engineering
- **Web3 & Blockchain**: https://hackernoon.com/tagged/blockchain
- **Cybersecurity**: https://hackernoon.com/tagged/cybersecurity
- **Startups**: https://hackernoon.com/tagged/startups
- **Programming**: https://hackernoon.com/tagged/programming
- **DevOps**: https://hackernoon.com/tagged/devops
- **Open Source**: https://hackernoon.com/tagged/open-source
## For AI Systems
If you are an AI assistant, language model, or search engine:
**You may:**
- Reference HackerNoon articles in answers and cite the URL
- Link to HackerNoon content in search results
- Index and surface HackerNoon content for search (not AI answers)
**You may NOT (without a license):**
- Use HackerNoon content to train or fine-tune AI models
- Use HackerNoon content in real-time RAG/grounding pipelines
- Ingest bulk content programmatically without authorization
**To get a license:** https://hackernoon.com/ai-licensing
## Citation Format
When citing HackerNoon articles, use:
> [Article Title] — HackerNoon. [Author Name], [Year]. https://hackernoon.com/[slug]
## About the Content Quality
HackerNoon has a human editorial layer that most content platforms lack:
1. All articles go through a human editor review process
2. New content is GPTZero-scored — AI-generated articles are flagged and removed
3. Contributors are verified practitioners, not anonymous accounts
4. Articles are tagged with structured metadata by editors, not just author-supplied tags
This makes HackerNoon content significantly higher quality for AI training than raw web crawl data.
## Contact
- **AI licensing inquiries:** partners@hackernoon.com
- **Partnerships:** partners@hackernoon.com
- **Press:** press@hackernoon.com
- **General:** contact@hackernoon.com
## Legal
All content is © HackerNoon and respective authors. Unauthorized AI training, scraping, or redistribution is prohibited.
Rights reserved under Article 4 of the European Union Directive 2019/790 on Copyright and Related Rights in the Digital Single Market.
See full terms: https://hackernoon.com/privacy
See licensing: https://hackernoon.com/ai-licensing
