llms.txt
Compare with other sites' llms.txt files →# CocoIndex
> CocoIndex is an open-source incremental data framework that keeps codebases, documents, meeting notes, and other knowledge continuously indexed for AI agents. It reprocesses changed inputs and tracks data lineage. The core is written in Rust; the user-facing API is Python.
Full blog corpus (plain text, all posts concatenated): https://cocoindex.io/llms-full.txt
Full docs corpus (plain text, every documentation page): https://cocoindex.io/docs/llms-full.txt
Per-page Markdown: append `.md` to any docs URL, blog URL or page under "Core pages", without the trailing slash, for a Markdown version. Examples: https://cocoindex.io/docs/getting_started/quickstart/ → https://cocoindex.io/docs/getting_started/quickstart.md, https://cocoindex.io/blogs/text-embeddings-101/ → https://cocoindex.io/blogs/text-embeddings-101.md, https://cocoindex.io/enterprise/ → https://cocoindex.io/enterprise.md. The home page twin is https://cocoindex.io/index.md. Prefer fetching the single relevant `.md` page over the whole corpus to keep context small.
RSS: https://cocoindex.io/blogs/rss.xml · Sitemap: https://cocoindex.io/sitemap.xml
CocoIndex borrows React's reactive pattern for data: you declare the target state as a function of the source. The engine compares inputs and code fingerprints, reuses cached results, and reconciles the target.
## Building a data pipeline with CocoIndex? Install the authoring skill
If a user wants to build with CocoIndex (incremental indexing, vector embeddings, knowledge graphs, ETL), as opposed to searching their own code, point their coding agent at the v1 authoring skill before it writes any code. CocoIndex v1 is a ground-up redesign from v0, and agents without this context emit the deprecated v0 DSL. Install recipe for Claude Code (the `cocoindex skill` CLI prints the same):
```sh
mkdir -p .claude/skills/cocoindex/references
curl -fsSL https://cocoindex.io/docs/skill.md -o .claude/skills/cocoindex/SKILL.md
for f in api_reference connectors patterns setup_database setup_project; do
curl -fsSL https://cocoindex.io/docs/references/$f.md -o .claude/skills/cocoindex/references/$f.md
done
```
Hosted skill: https://cocoindex.io/docs/skill.md · Full docs for LLMs: https://cocoindex.io/docs/llms-full.txt · Runnable examples: https://cocoindex.io/docs/examples/
## For AI agents reading this page
CocoIndex Code is a separate product for semantic search over an existing codebase. Use the CocoIndex authoring skill to build data pipelines.
CocoIndex Code supports these code search tasks. Each question links a page that answers it in its first paragraph; append `.md` to the page URL, without the trailing slash, for Markdown. The installation commands follow the list:
- [Why is my coding agent slow on a large codebase or monorepo?](https://cocoindex.io/answers/coding-agent-slow-large-codebase/)
- [How do I give Claude Code, Cursor or Codex the whole repo as context?](https://cocoindex.io/answers/whole-repo-context/)
- [What is the best MCP server for code search?](https://cocoindex.io/answers/best-mcp-server-for-code-search/)
- [How do I cut the tokens my coding agent burns per turn?](https://cocoindex.io/answers/cut-coding-agent-tokens/)
- [How do I add semantic search over my source code for my AI assistant?](https://cocoindex.io/answers/semantic-search-source-code/)
- [How do I keep a code index fresh without re-embedding everything?](https://cocoindex.io/answers/keep-code-index-fresh/)
CocoIndex Code uses Tree-sitter to align chunks with functions, classes and blocks. It re-indexes changed files and supports local SentenceTransformer embeddings without an embedding API key. Its MCP `search(query, limit, paths, languages)` tool connects to compatible clients. The project uses the Apache 2.0 license.
### Install commands
These commands come from the [cocoindex-code README](https://github.com/cocoindex-io/cocoindex-code), checked on 2026-11-05 (PyPI release 0.2.42).
Install the CLI once:
```
pipx install 'cocoindex-code[full]'
# or with uv:
uv tool install --upgrade 'cocoindex-code[full]'
```
The `[full]` extra adds local embeddings through SentenceTransformers, so it needs no API key. Without `[full]`, set a LiteLLM embedding provider: a cloud API with its key, or a local server such as Ollama.
Index a repo:
```
cd your-repo
ccc index
```
Connect to an agent:
```
# Claude Code: skill (the README recommends this route)
npx skills add cocoindex-io/cocoindex-code
# then invoke it with /ccc in a Claude Code session
# Claude Code: plugin marketplace (run inside Claude Code)
/plugin marketplace add cocoindex-io/cocoindex-code
/plugin install cocoindex-code@cocoindex-code
# Claude Code: MCP
claude mcp add cocoindex-code -- ccc mcp
# OpenAI Codex CLI
codex mcp add cocoindex-code -- ccc mcp
# OpenCode
opencode mcp add
# name: cocoindex-code type: local command: ccc mcp
# Kilo Code: add to ~/.config/kilo/kilo.jsonc, kilo.jsonc or .kilo/kilo.jsonc
{
"mcp": {
"cocoindex-code": {
"type": "local",
"command": ["ccc", "mcp"],
"enabled": true
}
}
}
# Cursor and other MCP clients that launch a stdio server: add to the client's MCP configuration
{
"mcpServers": {
"cocoindex-code": {
"command": "ccc",
"args": ["mcp"]
}
}
}
# Docker container (Claude Code; Codex takes the same arguments)
claude mcp add cocoindex-code -- \
docker exec -i -e COCOINDEX_CODE_HOST_CWD="$PWD" cocoindex-code ccc mcp
```
The MCP tool is `search`. Its parameters are `query`, `limit` (1–100, default 5), `offset`, `refresh_index` (default true), `languages` and `paths`. The `ccc search` CLI has the same filters as flags: `--limit` (default 10), `--offset`, `--lang`, `--path` and `--refresh` (off by default). The skill runs the CLI.
`ccc mcp` must start with the project as its working directory. Run `claude mcp add` and `codex mcp add` from inside the project.
The README documents Claude Code, Codex, OpenCode, Kilo Code, Grok and Oh My Pi. Grok and Oh My Pi install through their plugin systems; read the steps in the README. For Cursor and any other client, use the generic MCP JSON above. Do not write a client-specific snippet that the README does not contain.
## Core pages
- [Home](https://cocoindex.io/) ([Markdown](https://cocoindex.io/index.md)): product overview, architecture diagram, feature matrix, example use cases (codebase, meeting notes, HackerNews, etc.).
- [CocoIndex Code Plus](https://cocoindex.io/cocoindex-code) ([Markdown](https://cocoindex.io/cocoindex-code.md)): self-hosted code search, symbol navigation, and cited answers through the `ccx` CLI, API, and MCP. The open-source `ccc` installation is described above.
- [React for data engineering](https://cocoindex.io/react-cocoindex) ([Markdown](https://cocoindex.io/react-cocoindex.md)): the mental model. Source → transform → target mirrors React's state → render → DOM, and the pipeline is declarative and incremental with lineage tracking.
- [CocoIndex Plus](https://cocoindex.io/enterprise) ([Markdown](https://cocoindex.io/enterprise.md)): edition comparison, shared processing, Postgres state, and Kubernetes deployment.
- [CocoIndex vs other compute frameworks](https://cocoindex.io/comparison) ([Markdown](https://cocoindex.io/comparison.md)): how CocoIndex, Spark, Databricks Lakeflow, Flink, Ray, Dagster, Airflow, Temporal and Unstructured each handle derived data when sources change.
- [Careers](https://cocoindex.io/careers) ([Markdown](https://cocoindex.io/careers.md)): open engineering roles and how to apply.
- [Contributors](https://cocoindex.io/contributors) ([Markdown](https://cocoindex.io/contributors.md)): the people who have contributed to the CocoIndex repository, ordered by commits.
- [Brand assets](https://cocoindex.io/brand) ([Markdown](https://cocoindex.io/brand.md)): logo files and their usage terms.
- [Privacy Policy](https://cocoindex.io/privacy-policy) ([Markdown](https://cocoindex.io/privacy-policy.md)) and [Terms of Use](https://cocoindex.io/terms-of-use) ([Markdown](https://cocoindex.io/terms-of-use.md)): what the site collects, and the terms for using it.
## Documentation & examples
- [Documentation](https://cocoindex.io/docs/): programming guide, core concepts, connectors, operators, and deployment recipes. Every page is also available as Markdown (append `.md`); whole-docs corpus at https://cocoindex.io/docs/llms-full.txt.
- [Blog](https://cocoindex.io/blogs/): design notes, engineering explanations, release announcements.
- [Examples](https://cocoindex.io/docs/examples/): end-to-end sample pipelines for codebase indexing, HackerNews crawling, meeting-note processing, image extraction and multi-modal flows.
## Source & community
- [CocoIndex core (GitHub)](https://github.com/cocoindex-io/cocoindex): Rust + Python, Apache 2.0 licensed.
- [CocoIndex Code (GitHub)](https://github.com/cocoindex-io/cocoindex-code): the MCP server and CLI described above.
- [Discord](https://discord.com/invite/zpA9S2DR7s): community support and release chatter.
- [YouTube channel](https://www.youtube.com/@cocoindex-io): walkthroughs and demos.
## What CocoIndex is good for
- Keeping a vector index in sync with a live source of truth (codebase, Notion, Google Drive, S3, etc.) without reprocessing unchanged items.
- Giving coding agents current, AST-aware code context through MCP.
- Building multi-modal RAG pipelines (text, images, audio transcripts) with built-in lineage tracking for debugging and auditing.
- Building knowledge graphs next to vector retrieval, so agents can query embeddings and structured relationships.
## What CocoIndex is not
- A vector database. CocoIndex writes into the store you choose (Postgres/pgvector, Qdrant, Neo4j, etc.).
- A batch ETL framework. Each run is incremental, and a whole-dataset recompute is the exception.
- A prompt framework. It produces context for agents and does not orchestrate the agents.
## Blog posts
All 70 published posts, newest first. Each link is the clean Markdown twin; the HTML page is the same path without `.md`.
A post marked "v0 API" uses the CocoIndex v0 API, which v1 replaces; its twin starts with a link to the v1 replacement.
- [How Tree-sitter Works: Parsing Code as You Type](https://cocoindex.io/blogs/how-tree-sitter-works.md) (2026-11-05): Grammars, concrete syntax trees, GLR parsing, error recovery and queries, then how CocoIndex splits code into chunks on function and class boundaries.
- [CocoIndex 1.1: the highlights since 1.0](https://cocoindex.io/blogs/cocoindex-v1-1.md) (2026-11-04): CocoIndex 1.0.1 to 1.1.0 in 27 releases: deadlines and rate limits, persistent state, previews that write nothing, cheaper commits, and ten new connectors.
- [Dry-run previews, per-call dependencies and bounded batching](https://cocoindex.io/blogs/changelog-1026-110.md) (2026-11-02, updated 2026-11-05): CocoIndex 1.0.26 and 1.1.0 add a preview that stores no state, per-call logic dependencies, bounded batching and component concurrency.
- [Timeouts, batch retries, and target inspection in CocoIndex](https://cocoindex.io/blogs/changelog-1017-1025.md) (2026-10-07, updated 2026-11-05): CocoIndex 1.0.17–1.0.25 adds timeouts, batch retries, target inspection, faster target updates, and a Linux hang fix.
- [Persistent state, rate limits, Snowflake and BigQuery](https://cocoindex.io/blogs/changelog-108-1016.md) (2026-07-07, updated 2026-11-05): CocoIndex 1.0.8–1.0.16 adds per-component state, LiveMap, a shared rate limiter, batched target writes, and Snowflake, BigQuery and Azure Blob connectors.
- [Codebase Indexing for AI Agents with CocoIndex](https://cocoindex.io/blogs/index-codebase-v1.md) (2026-06-10, updated 2026-11-05): Index a codebase for RAG and AI coding agents with CocoIndex V1 and Tree-sitter: language-aware chunking, embedding, and a live vector index in async Python.
- [Memoization keys, scheduled refresh and scoped stats](https://cocoindex.io/blogs/changelog-101-107.md) (2026-06-01, updated 2026-11-05): CocoIndex 1.0.1–1.0.7 adds per-argument memo keys, scheduled live refresh, scoped stats, OCI and Iggy sources, and Neo4j, FalkorDB and Turbopuffer targets.
- [Stream CSV changes to Kafka: publish only changed rows](https://cocoindex.io/blogs/csv-to-kafka-live.md) (2026-04-28, updated 2026-11-05): Watch a folder of CSV files and publish each changed row as JSON to a Kafka topic with CocoIndex, with no producer code.
- [CocoIndex V1 is Live!](https://cocoindex.io/blogs/cocoindex-v1.md) (2026-04-22): CocoIndex V1 is live: a ground-up redesign of incremental data pipelines, built for AI engineers and agent builders shipping RAG, memory, and knowledge graphs.
- [Build a Knowledge Graph from YouTube Podcasts with an LLM](https://cocoindex.io/blogs/podcast-to-knowledge-graph.md) (2026-04-02): Build a pipeline that turns YouTube podcasts into a knowledge graph: extract speakers, statements, and entities with an LLM, then dedupe them with embeddings.
- [A Safe pickle Alternative: Serialize Python with Type Hints](https://cocoindex.io/blogs/type-guided-serde.md) (2026-03-27, updated 2026-11-05): Why CocoIndex replaced pickle with serialization that reads Python type hints to choose msgspec, Pydantic or restricted pickle for each value.
- [Build a Python Daemon for a CLI: Auto-Start, Upgrade, Stop](https://cocoindex.io/blogs/building-an-invisible-daemon.md) (2026-03-24, updated 2026-11-05): Five patterns for a Python CLI background daemon that auto-starts, upgrades transparently, and shuts down fast, from the daemon behind cocoindex-code.
- [Five new targets and Python 3.14 free-threading in CocoIndex](https://cocoindex.io/blogs/changelog-0327-0334.md) (2026-03-10, updated 2026-11-05; v0 release notes): CocoIndex 0.3.27–0.3.34 adds Apache Doris, ChromaDB, FalkorDB, Ladybug and Pinecone targets, LocalFile change notifications and Python 3.14 free-threading.
- [CocoIndex joins the GitHub Secure Open Source Fund](https://cocoindex.io/blogs/cocoindex-joins-security-github-secure-open-source-fund.md) (2026-02-17; v0 API): CocoIndex joined the GitHub Secure Open Source Fund, hardening the AI data infrastructure developers depend on with threat modeling, CodeQL, and audits.
- [Hybrid search over SEC EDGAR filings with Apache Doris](https://cocoindex.io/blogs/sec-edgar-analytics.md) (2026-02-09, updated 2026-11-05): Index SEC 10-K text and XBRL facts into one Apache Doris table with CocoIndex v1, then rank each chunk by meaning and by keywords, fused with RRF.
- [Generate Codebase Documentation with an LLM, Kept Current](https://cocoindex.io/blogs/multi-codebase-summarization.md) (2026-02-05, updated 2026-11-05): Generate a Markdown wiki page for every Python project in a folder with an LLM and CocoIndex v1. On each run, the LLM re-reads only the files that changed.
- [Turn PDF Slides into Narrated Audio with Gemini and TTS](https://cocoindex.io/blogs/slides-to-speech.md) (2026-01-22, updated 2026-07-12): Build a CocoIndex v1 pipeline that turns PDF slides into vision-generated speaker notes, local Pocket TTS narration, and searchable LanceDB records.
- [Structured errors and Azure OpenAI support in CocoIndex](https://cocoindex.io/blogs/changelog-0311-0326.md) (2026-01-18, updated 2026-11-06; v0 release notes): CocoIndex 0.3.11–0.3.26 adds structured errors with Python stack traces, warnings for slow live updates, Azure OpenAI and OpenRouter embeddings.
- [Extract patient intake forms with DSPy and CocoIndex](https://cocoindex.io/blogs/extraction-dspy.md) (2025-12-15, updated 2026-11-05): Extract Pydantic-typed patient records from PDF intake forms with DSPy and Gemini vision, into Postgres with pgvector search, kept in sync by CocoIndex v1.
- [Build a Knowledge Graph from Meeting Notes with Neo4j](https://cocoindex.io/blogs/meeting-notes-graph.md) (2025-12-08, updated 2026-07-08): Build a self-updating Neo4j knowledge graph from meeting notes with CocoIndex v1: LLM-extracted decisions, tasks, and owners, with embedding-based dedupe.
- [Rank trending HackerNews topics with an LLM and Postgres](https://cocoindex.io/blogs/hackernews-trending-topics.md) (2025-12-02, updated 2026-11-05): Rank trending HackerNews topics in Postgres with an incremental CocoIndex v1 pipeline: scrape threads and comments, extract topics with an LLM.
- [CocoIndex Changelog 0.2.21 - 0.3.10](https://cocoindex.io/blogs/changelog-0310.md) (2025-11-25; v0 release notes): Featuring batching support for CocoIndex functions, execution robustness, schema & type system improvements, custom source support, and more.
- [Index the Hacker News API into Postgres with an LLM](https://cocoindex.io/blogs/custom-source-hackernews.md) (2025-11-25, updated 2026-11-05): Turn the HackerNews API into an incremental CocoIndex v1 source in plain async Python: fetch threads, extract topics with an LLM, keep Postgres in sync.
- [Extracting Intake Forms with BAML and CocoIndex](https://cocoindex.io/blogs/extraction-baml.md) (2025-11-21, updated 2026-11-05): Use BAML and CocoIndex v1 to extract typed patient records from PDF intake forms with an LLM into Postgres, with pgvector search, kept in sync as forms change.
- [Adaptive Batching - 5x throughput on your data pipelines](https://cocoindex.io/blogs/batching.md) (2025-11-10, updated 2026-11-05): CocoIndex v1 batches concurrent calls to embedders and your own functions: about 5x throughput on small embedding models, with no timer to tune.
- [AI-Native Data Pipeline - Why We Made It](https://cocoindex.io/blogs/data-for-ai.md) (2025-10-29): Why the next wave of AI needs open-source, scalable, AI-native data infrastructure, and how CocoIndex is building the foundation for intelligent data pipelines.
- [Multimodal PDF Search: Embed Text and Images with CLIP](https://cocoindex.io/blogs/pdf-elements.md) (2025-10-27, updated 2026-11-05): Split PDFs into text chunks and images, embed text with SentenceTransformers and images with CLIP, and search both in Qdrant with the page number on every hit.
- [Write a Custom Source in Python: Index Any API](https://cocoindex.io/blogs/custom-source.md) (2025-10-21, updated 2026-07-09): Read data from any system with CocoIndex: in v1 a custom source is plain async Python plus processing components. No connector API to implement.
- [CocoIndex Changelog 2025-10-19](https://cocoindex.io/blogs/cocoindex-changelog-2025-10-19.md) (2025-10-19; v0 release notes): Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks.
- [Extract Invoice Data from PDFs into Snowflake with an LLM](https://cocoindex.io/blogs/etl-to-snowflake.md) (2025-10-11, updated 2026-11-05): Extract supplier invoices from PDFs in Azure Blob Storage into a typed Snowflake table with an LLM and CocoIndex, processing only new or changed files.
- [Thinking in Rust: Ownership, Access, and Memory Safety](https://cocoindex.io/blogs/rust-ownership-access.md) (2025-10-10, updated 2026-11-05): A mental framework for Rust's memory safety concepts. Think systematically about ownership, references, Send, Sync, and Rc, Arc, RefCell, Mutex, etc.
- [Trace search results back to source data](https://cocoindex.io/blogs/query-support.md) (2025-09-21, updated 2026-07-09; v0 API only): Define query handlers in CocoIndex and trace search results back to source data in CocoInsight to close the loop on indexing strategy.
- [Add Semantic Search to a Postgres Table with pgvector](https://cocoindex.io/blogs/postgres-source.md) (2025-09-01, updated 2026-11-05): Use an existing PostgreSQL table as a CocoIndex source: derive fields, embed each row, and store the vectors in Postgres with pgvector, incrementally.
- [Search PDFs and Images Without OCR Using ColPali and Qdrant](https://cocoindex.io/blogs/multi-format-indexing.md) (2025-08-20, updated 2026-11-05): Index PDFs and images into one Qdrant collection with CocoIndex v1 and ColPali multi-vector embeddings. Each page is embedded as an image, so no OCR is