llms.txt
Compare with other sites' llms.txt files →# Collinear AI
> Collinear AI builds the realistic RL environments, verifiers, and training data that frontier labs use for their hardest model improvement problems. Agents are exercised against simulated users, stateful tool clones, and multi-turn tasks before they reach production, and the verified trajectories and reward signals that come out feed post-training and evaluation.
Collinear is researcher-founded and researcher-led. The company is headquartered in Sunnyvale, California, and is opening a San Francisco office.
The premise: in production, agents navigate changing objectives, users, tools, and persistent state. Static evals reduce that to isolated prompts and final answers, and miss the failures that matter. Users interrupt and change direction. Work spans many turns, tools, and recoveries. A convincing answer can hide a failed task.
## Simulation Lab
A Simulation Lab is a self-contained world where an agent operates, holding the users, tools, data, and tasks it will face in production. Each lab contains:
1. **Agent endpoint**: any model, any harness, open or closed source, connected through your own endpoint. Collinear is endpoint-agnostic.
2. **Tasks**: multi-turn tasks with realistic constraints, including ambiguous specs, incomplete information, and competing stakeholder priorities.
3. **Scenario data**: org structures, policies, and historical context grounded in real enterprise data.
4. **Tools and APIs**: stateful sandboxes and clones of enterprise tools, with state persisting across turns.
5. **Simulated users**: non-scripted behavior including interruptions, objections, and preference shifts.
6. **Verifiers**: deterministic programmatic verifiers and rubric-scored outcomes, validated against human evaluators.
7. **Outputs**: training data and reward signals for post-training a model or improving an agent harness.
## Off-the-shelf inventory
Tasks, evaluations, and runnable environments are available off the shelf, organised four ways:
- **Capability**: calibrated where frontier models fail, across software engineering, computer use, and long-horizon reasoning.
- **Domain**: expert-calibrated where correctness is hard, including cybersecurity, finance, and customer service.
- **Benchmark**: built to diagnose failure rather than rank models. Includes CWE-bench, FrontierCUA, MCP-Atlas, and OSWorld-Verified.
- **Environment and tool surface**: real work rather than toy prompts, from GUIs and APIs to MCP, CLI, GitHub, Jira, ServiceNow, and Salesforce.
Task categories cover cyber, software engineering, computer use, knowledge work, and behavior.
## Calico
Calico by Collinear is the ready-made catalog: 18,000 off-the-shelf tasks across five capabilities, each delivered with a task specification, a runnable environment, and a reward verifier, plus a reference solution or customer-specific QA depending on the capability. Buyers start with five representative samples, evaluate them against their own models, then choose capabilities and volume.
- **MCP** (enterprise tool use): 5,000 tasks across connected enterprise systems. <https://www.collinear.ai/calico/mcp>
- **CUA** (computer use): 5,000 tasks in working copies of enterprise software. <https://www.collinear.ai/calico/cua>
- **Cyber**: 4,000 tasks, 3,000 defensive (audit and patch) and 1,000 offensive (black-box exploitation). <https://www.collinear.ai/calico/cyber>
- **DeepSWE**: 3,000 tasks on production repositories and full toolchains, with deterministic verification. <https://www.collinear.ai/calico/deepswe>
- **GDPval**: 1,000 tasks of economically valuable work across six enterprise domains. <https://www.collinear.ai/calico/gdpval>
Overview: <https://www.collinear.ai/calico>. Request the catalog and samples: <https://www.collinear.ai/request-catalog>
## How verification works
Every task starts from an explicit definition of success, then uses the strongest verification the task supports: programmatic checks for objective outcomes, state and trajectory checks for process constraints, and calibrated expert or model-judge rubrics where judgment is unavoidable. Verifiers are tested against successful, failed, and adversarial trajectories to catch false rewards and reward hacking before delivery.
The same task system can diagnose a gap, generate trajectories and reward for training, and measure improvement on a held-out set. Training and evaluation boundaries are kept explicit so higher reward is not mistaken for capability gain.
## Research areas
World models, recursive self-improvement, user simulation, long-horizon agents, and post-training.
## Benchmarks
- **CWE-bench**: coding agents against real vulnerabilities in real codebases, with strict gates and both programmatic and judge verification on every task. 120 held-out evaluation tasks across 73 CWEs and all 10 OWASP categories, covering C/C++, Go, Java, TS/JS, Python, and Rust; a separate training corpus of 4,000+ tasks spans 215 CWEs. <https://cwe-bench.com/> Paper: <https://arxiv.org/abs/2610.17020>
- **FrontierCUA**: a computer-use benchmark: AI agents executing enterprise workflows across running clones of real business applications, graded by weighted rubric. Agents use screenshots, clicks, typing, and scrolling, and cannot query databases, call app APIs, or inspect hidden browser state. 15 held-out workflows, with a public leaderboard ranked by pass@1; a separate training corpus of 3,000+ runnable tasks for enterprise computer use. <https://frontiercua.com/> (leaderboard: <https://frontiercua.com/leaderboard/>, methodology: <https://frontiercua.com/methodology/>, machine-readable summary: <https://frontiercua.com/llms.txt>)
- **MatFlowBench**: a materials science benchmark: agents carrying multi-step materials research workflows to a verified result, from DFT calculations in Quantum ESPRESSO, Wannier90 and Phonopy to planning measurements, analyzing archived instrument data and selecting materials. 45 private tasks, 25 computational and 20 experimental, with a public leaderboard of 10 models. <https://matflowbench.com/> (machine-readable summary: <https://matflowbench.com/llms.txt>)
- **YC-Bench**: a year-long simulated startup that tests whether an agent can hold a strategy and execute consistently across hundreds of turns. Open source, spanning one simulated startup-year. <https://arxiv.org/abs/2604.01212>
## Publications
- [YC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Execution](https://arxiv.org/abs/2604.01212)
- [The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Models](https://arxiv.org/abs/2510.06101) (NeurIPS)
- [Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents](https://arxiv.org/abs/2510.04491)
- [Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models](https://arxiv.org/abs/2503.01781)
- [VERITAS: A Unified Approach to Reliability Evaluation](https://arxiv.org/abs/2411.03300)
- [Self-rationalization improves LLM as a fine-grained judge](https://arxiv.org/abs/2410.05495)
## Field notes
Technical writing from the team, published at <https://blog.collinear.ai/> and indexed on the research page.
## Company
- **Founded by**: Nazneen Rajani (Co-founder and CEO), whose work spans model evaluation, safety, alignment, and post-training, and who led post-training research at Hugging Face and Salesforce Research; and Soumyadeep Bakshi (Co-founder), whose work spans data strategy, verification, and evaluation infrastructure, and who led Deloitte's AI practice advising frontier labs on data for model development.
- **Investors**: Engineering Capital, 112 Capital, B Capital, Storm Ventures, Hitachi Ventures.
- **Advisors**: James Zou (Stanford), Ritesh Agrawal.
- **Team**: researchers and engineers from Hugging Face, Salesforce, Google, Amazon, Stanford, MIT, Cornell, Yale, and Columbia.
- **Locations**: Sunnyvale, California (headquarters), San Francisco, and Bengaluru, India.
- **Certifications**: AICPA SOC, ISO 42001 v2023.
- **Security and compliance**: <https://trustcentre.collinear.ai/>
## Pages
- [Home](https://www.collinear.ai/)
- [Research](https://www.collinear.ai/research): benchmarks, publications, and field notes.
- [About us](https://www.collinear.ai/about-us): founders, investors, advisors, and press.
- [Media & Press](https://www.collinear.ai/media-kit): expert commentary topics, coverage, portraits, logos, and biographies for journalists.
- [Careers](https://www.collinear.ai/careers): open roles across research, engineering, and product in the San Francisco Bay Area and Bengaluru.
- [Book a demo](https://www.collinear.ai/book-a-demo): talk to a researcher about post-training needs.
- [Calico](https://www.collinear.ai/calico): the ready-made catalog of training tasks, environments, and rewards, with a page per capability.
- [Request the Calico catalog](https://www.collinear.ai/request-catalog): ask for the full catalog and five sample tasks.
- [Documentation](https://docs.collinear.ai/introduction)
- [Privacy Policy](https://www.collinear.ai/privacy-policy)
- [EULA](https://www.collinear.ai/eula)
- [Accessibility](https://www.collinear.ai/accessibility)
- [Subprocessors](https://trustcentre.collinear.ai/) (listed in the Trust Centre)
## Contact
info@collinear.ai