llms.txt
Compare with other sites' llms.txt files →# Vals AI
> Independent AI benchmark results, model comparisons, research, and evaluation tools.
Vals AI publishes benchmark methodology and results for AI systems across legal, finance, coding, medical, mathematics, and other domains. Use the published benchmark and model pages as the source of record.
## Primary pages
- [Home](https://www.vals.ai/home)
- [Benchmarks](https://www.vals.ai/benchmarks): every current benchmark leaderboard
- [Model comparisons](https://www.vals.ai/comparisons)
- [Compare any two models](https://www.vals.ai/comparison)
- [Methodology](https://www.vals.ai/methodology)
- [About](https://www.vals.ai/about)
## Benchmarks
### Index
- [Vals Index](https://www.vals.ai/benchmarks/vals_index): A single measure of AI's potential economic impact — agentic model performance across finance, coding, legal, and tax tasks, weighted by each sector's share of U.S. GDP.
- [Vals RSI Index](https://www.vals.ai/benchmarks/rsi_index): Can a model do the research that builds the next model? Autonomous LLM R&D scored against published, human, and frontier-model references.
- [Vals Web Search Index](https://www.vals.ai/benchmarks/web_search): Comparing native provider search against independent web-search tools on legal-research and finance-analysis tasks
- [Vals Finance Index](https://www.vals.ai/benchmarks/finance_index): Agentic model performance on finance work, from analyst research and financial modeling to corporate tax research, weighted by each sub-sector's share of U.S. GDP.
- [Vals Legal Index](https://www.vals.ai/benchmarks/legal_index): Model performance on legal research and legal drafting across practice areas, weighted by each area's share of U.S. law-office revenue.
- [Vals Science Index](https://www.vals.ai/benchmarks/science_index): A single measure of AI's ability to do science — agentic model performance on biology data investigation, mechanism discovery, and research workflows, weighted equally.
- [Vals Router Index](https://www.vals.ai/benchmarks/vals_router_index): How model routers compare with single models on long agentic work in finance, coding, law and tax
- [Vals Inference Provider Index](https://www.vals.ai/benchmarks/inference_provider): Compare the same open-weight model across inference providers on accuracy, cost, and speed, measured on agentic finance, coding, and legal tasks.
### Legal
- [Legal Research Bench](https://www.vals.ai/benchmarks/legal_research): Evaluating agents on legal research tasks across diverse areas of US law
- [HLAB-Vals-1.1](https://www.vals.ai/benchmarks/hlab): Tests an agent's ability to complete legal work using documents, spreadsheets, presentations, and file-system tools.
- [LegalBench](https://www.vals.ai/benchmarks/legal_bench): Evaluating language models on a wide range of open source legal reasoning tasks.
- [Immigration Bench](https://www.vals.ai/benchmarks/immigration): Evaluating AI models on expert-written U.S. immigration law questions
### Finance
- [Finance Agent v2.1](https://www.vals.ai/benchmarks/fabv2-1): Evaluating agents on core financial analyst tasks
- [Excel Modeling Benchmark v1.1](https://www.vals.ai/benchmarks/emb-v1-1): Evaluating agents on Excel-based financial modeling tasks
- [Tax Agent Bench](https://www.vals.ai/benchmarks/tax_agent_bench): Evaluating agents on research-grade US corporate tax questions
- [ClaimBench](https://www.vals.ai/benchmarks/claimbench): Can a decision model automate checking whether a 10-K excerpt supports, contradicts or says nothing about a claim?
### Healthcare
- [MedCode](https://www.vals.ai/benchmarks/medcode): Can models support the medical billing process?
- [MedScribe](https://www.vals.ai/benchmarks/medscribe): Can models support doctors with their administrative work?
### Mental Health
- [SAFE-Teen](https://www.vals.ai/benchmarks/mental-health): Can models respond safely as teens disclose concerns, resist advice, and ask for help?
### Math
- [ProofBench v1.1](https://www.vals.ai/benchmarks/proof_bench): Can models write math proofs that are formally verified?
### Science
- [Terminal-Bench-Science](https://www.vals.ai/benchmarks/terminal-bench-science): Research workflow tasks contributed by practicing scientists across five domains
- [BioMysteryBench](https://www.vals.ai/benchmarks/biomysterybench): Can agents solve agentic biology data investigations?
- [MysteryMechanism](https://www.vals.ai/benchmarks/mysterymechanism): Can agents rediscover sealed mathematical mechanisms through bounded experiments?
- [Robo-bench](https://www.vals.ai/benchmarks/robo_bench): Can a model control a robot arm and a humanoid in a real-time physics simulation, from motor commands and camera images alone? Most tasks are held out.
### Education
- [SAGE](https://www.vals.ai/benchmarks/sage): Student Assessment with Generative Evaluation
### Voice
- [VoiceCodeBench](https://www.vals.ai/benchmarks/voice-code-bench): How good are speech-to-text models at handling exact structured values in English workplace speech?
### Coding
- [Vibe Code Bench v1.1](https://www.vals.ai/benchmarks/vibe-code): Can models build web applications from scratch?
- [Vibe Code Bench 1-100](https://www.vals.ai/benchmarks/vcb-1-100): Can models extend a working web application across future user requests?
- [Code Migration v1.1](https://www.vals.ai/benchmarks/code-migration-v1-1): Can language models reimplement working programs in another language?
- [ProgramBench](https://www.vals.ai/benchmarks/programbench): Can language models rebuild programs from scratch?
- [Terminal-Bench 4.0](https://www.vals.ai/benchmarks/terminal-bench-4): Frontier-difficulty terminal tasks across software, science, ML, operations, hardware, security, and media
- [SkillsBench](https://www.vals.ai/benchmarks/skillsbench): How important are skills for agents?
- [IOI v2](https://www.vals.ai/benchmarks/ioi): International Olympiad in Informatics
- [SWE-bench Multimodal](https://www.vals.ai/benchmarks/swebench-multimodal): Do AI Systems Generalize to Visual Software Domains?
### Cyber
- [CyberOffense](https://www.vals.ai/benchmarks/cyber_offense): Can AI autonomously compromise realistic enterprise networks?
- [CyberBench v1.1](https://www.vals.ai/benchmarks/cyber): Can AI agents find security bugs and patch them?
- [SRE Bench](https://www.vals.ai/benchmarks/srebench): Can AI agents work out what a real-world binary does without its source code?
### Games
- [CUA-bench](https://www.vals.ai/benchmarks/cua_bench): Can an AI agent play six commercial video games with only a keyboard and a mouse? Half of them are held out.
- [Time Horizon Index: KSP](https://www.vals.ai/benchmarks/time_horizon_index): Can an AI agent build and run a space program in Kerbal Space Program?
- [Poker Agent](https://www.vals.ai/benchmarks/poker_agent): Which model can make the most money playing poker?
### Social Mobility
- [Public Benefits Bench v1.1](https://www.vals.ai/benchmarks/public-benefits-bench): Can AI help people navigate SNAP benefits?
### Archived
No longer run on new models; earlier results stay published.
- [AIME](https://www.vals.ai/benchmarks/aime): Challenging national math exam given to top high-school students
- [CaseLaw v2](https://www.vals.ai/benchmarks/case_law_v2): Private question-answer benchmark over Canadian court-cases.
- [Code Migration](https://www.vals.ai/benchmarks/code-migration): Can language models reimplement working programs in another language?
- [CorpFin v2](https://www.vals.ai/benchmarks/corp_fin_v2): A private benchmark evaluating understanding of long-context credit agreements
- [CyberBench v1](https://www.vals.ai/benchmarks/cyber-v1): Can AI agents find security bugs and patch them?
- [Excel Modeling Benchmark](https://www.vals.ai/benchmarks/emb): Evaluating agents on Excel-based financial modeling tasks
- [Finance Agent v1](https://www.vals.ai/benchmarks/finance_agent_v1): Evaluating agents on core financial analyst tasks
- [Finance Agent v1.1](https://www.vals.ai/benchmarks/finance_agent): Evaluating agents on core financial analyst tasks
- [Finance Agent v2](https://www.vals.ai/benchmarks/fabv2): Evaluating agents on core financial analyst tasks
- [GPQA Diamond](https://www.vals.ai/benchmarks/gpqa): Graduate-level Google-Proof Q&A benchmark evaluating models on questions that require deep reasoning.
- [Harvey's Legal Agent Benchmark v1](https://www.vals.ai/benchmarks/hlab-v1): Tests an agent's ability to complete legal work using documents, spreadsheets, presentations, and file-system tools.
- [IOI v1](https://www.vals.ai/benchmarks/ioi-v1): International Olympiad in Informatics
- [LiveCodeBench](https://www.vals.ai/benchmarks/lcb): Our Implementation of the LiveCodeBench benchmark
- [MATH 500](https://www.vals.ai/benchmarks/math500): Academic math benchmark on probability, algebra, and trigonometry
- [MedQA](https://www.vals.ai/benchmarks/medqa): Evaluating language model bias in medical questions.
- [MGSM](https://www.vals.ai/benchmarks/mgsm): A multilingual benchmark for mathematical questions.
- [MMLU Pro](https://www.vals.ai/benchmarks/mmlu_pro): Academic multiple-choice benchmark covering 14 subjects including STEM, humanities, and social sciences.
- [MMMU Pro](https://www.vals.ai/benchmarks/mmmu): Multimodal Multi-task Benchmark
- [MortgageTax](https://www.vals.ai/benchmarks/mortgage_tax): Evaluating reading and understanding tax certificates as images
- [ProofBench v1](https://www.vals.ai/benchmarks/proof_bench_v1): Can models write math proofs that are formally verified?
- [Public Benefits Bench v1](https://www.vals.ai/benchmarks/public-benefits-bench-v1): Can AI help people navigate SNAP benefits?
- [SAGE v1](https://www.vals.ai/benchmarks/sage-v1): Student Assessment with Generative Evaluation
- [SWE-bench Verified](https://www.vals.ai/benchmarks/swebench): Solving production software engineering tasks
- [TaxEval v2](https://www.vals.ai/benchmarks/tax_eval_v2): A Vals-created set of questions and responses to tax questions
- [Terminal-Bench 1.0](https://www.vals.ai/benchmarks/terminal-bench): State-of-the-art set of difficult terminal-based tasks
- [Terminal-Bench 2.0](https://www.vals.ai/benchmarks/terminal-bench-2): State-of-the-art set of difficult terminal-based tasks
- [Terminal-Bench 2.1](https://www.vals.ai/benchmarks/terminal-bench-2-1): State-of-the-art set of difficult terminal-based tasks
- [Vals Index](https://www.vals.ai/benchmarks/vals_index_v1_1): Benchmark consisting of a weighted performance across finance and coding tasks. Showing the potential impact that LLMs can have on the economy.
- [Vals Index](https://www.vals.ai/benchmarks/vals_index_v1_2): A single measure of AI's potential economic impact — model performance across finance and coding tasks, weighted by each sector's contribution to U.S. GDP.
- [Vals Index](https://www.vals.ai/benchmarks/vals_index_v1): Benchmark consisting of a weighted performance across finance and coding tasks. Showing the potential impact that LLMs can have on the economy.
- [Vals Index](https://www.vals.ai/benchmarks/vals_index_v2_1): A single measure of AI's potential economic impact — agentic model performance across finance, coding, legal, and tax tasks, weighted by each sector's share of U.S. GDP.
- [Vals Index](https://www.vals.ai/benchmarks/vals_index_v2): A single measure of AI's potential economic impact — agentic model performance across finance, coding, and legal tasks, weighted by each sector's share of U.S. GDP.
- [Vals Multimodal Index](https://www.vals.ai/benchmarks/vals_multimodal_index_v1_1): Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLMs can have on the economy.
- [Vals Multimodal Index](https://www.vals.ai/benchmarks/vals_multimodal_index_v1): Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLMs can have on the economy.
- [Vals Multimodal Index](https://www.vals.ai/benchmarks/vals_multimodal_index): Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLMs can have on the economy.
- [Vibe Code Bench v1](https://www.vals.ai/benchmarks/vibe-code-v1): Can models build web applications from scratch?
## Vals Index leaders (updated 2026-11-09)
- [1. Claude Fable 5.2](https://www.vals.ai/models/anthropic_claude-fable-5-2): 71.31%
- [2. Claude Sonnet 5.5](https://www.vals.ai/models/anthropic_claude-sonnet-5-5): 70.85%
- [3. Gemini 4 Carbon](https://www.vals.ai/models/google_gemini-4-carbon): 70.45%
- [4. Claude Opus 5.5](https://www.vals.ai/models/anthropic_claude-opus-5-5): 70.14%
- [5. Gemini 4 Argon](https://www.vals.ai/models/google_gemini-4-argon): 69.39%
- [6. Wonder Full](https://www.vals.ai/models/meta_muse-wonder-full): 66.79%
- [7. GPT-6 Astra](https://www.vals.ai/models/openai_gpt-6-astra): 65.02%
- [8. Wonder](https://www.vals.ai/models/meta_muse-wonder): 64.88%
- [9. GPT-6.1 Sol](https://www.vals.ai/models/openai_gpt-6.1-sol): 63.94%
- [10. Kimi K3.1 Pro](https://www.vals.ai/models/kimi_kimi-k3.1-pro): 62.52%
## Research and updates
- [Blog](https://www.vals.ai/blogs)
- [Inkling 2 Small: An Inkling of Profanity](https://www.vals.ai/blogs/inkling-2-small-swears): We read the reasoning of Thinking Machines' Inkling 2 Small across 2,655 AI agent runs on 13 benchmarks. It rarely swears, and when it does it is usually late in a long run.
- [How Much Does Reasoning Effort Affect Score and Cost?](https://www.vals.ai/blogs/reasoning-effort): At their lowest reasoning setting, frontier models often keep a majority of their performance at a fraction of the cost.
- [Evals First: Getting the Most Out of Fireworks with Vals](https://www.vals.ai/blogs/vals-fireworks-partnership): Vals partners with Fireworks customers to build and maintain custom evals, so they can pick the right model, train their own, and cut costs without cutting quality.
- [Can AI Get Immigration Right?](https://www.vals.ai/blogs/can-ai-get-immigration-right): This report presents an independent evaluation of how leading AI models perform on U.S. immigration law questions, and what it means for the international students, entrepreneurs, employers, and other people who rely on AI to navigate the U.S. immigration system.
- [Why is my local Qwen sending requests to Alibaba Cloud?](https://www.vals.ai/blogs/qwen-alibaba-requests): Maybe it was useful in RL
- [Thomson, One N at a Time](https://www.vals.ai/blogs/thomson-exposition): How the N = 7 proof works, what the proofs so far have in common, and which N we expect to come next.
- [Evaluating Teen Safety](https://www.vals.ai/blogs/teen-safety): We tested 17 AI chatbots on teen safety. Every one failed, and some far more often than others.
- [How Far Will AI Assistants Go for a Refund They Can’t Get?](https://www.vals.ai/blogs/impossible-refund): Six AI assistants pursued an impossible £350 train refund in a simulated world, escalating to government officials, legal clinics and bank chargebacks.
- [Routers Are a New AI Primitive](https://www.vals.ai/blogs/routers-new-ai-primitive): Model routers need an independent, end-to-end evaluation of quality, cost, and latency on real work. We're building one.
- [How pro-China are Chinese models?](https://www.vals.ai/blogs/china-us-analysis): It's complicated.
- [Do Agent Teams Pay Off? A Case Study on Vibe Code Bench](https://www.vals.ai/blogs/multi-agent-vibe-code-bench): Do AI coding-agent teams pay off? Compare GPT 6 Sol and Claude Opus 5.5 on Vibe Code Bench, including cost, reasoning effort and performance.
- [An Evaluation of Inception's Mercury Decide](https://www.vals.ai/blogs/evaluation-of-mercury-decide): Compare Mercury Decide, Jev and frontier LLMs on claim verification and a LegalBench subset, with accuracy, cost, latency and calibration results.
- [Trust but Verify: Our first Public Sector Summit](https://www.vals.ai/blogs/trust-but-verify-public-sector-summit): Reflections from our first Public Sector Summit at STATION DC on youth wellbeing, AI diplomacy with China, and independent evaluation.
- [Two-Thirds of MiMo v2.6's Coding Tasks Leak the Answer. Are Models That Exploit This Misbehaving?](https://www.vals.ai/blogs/mimo-reward-hacking): MiMo v2.6 reward-hacking audit: in two-thirds of Xiaomi's released coding environments, the fix remains in Git history. See how MiMo exploits it.
- [An Independent Evaluation of TypeSafe's Jev](https://www.vals.ai/blogs/independent-evaluation-of-jev): TypeSafe's Jev vs frontier LLMs on claim verification and a LegalBench subset: compare accuracy, cost, latency and calibration in Vals AI's evaluation.
- [Two Room-Temperature Antiferromagnetic Semiconductor Candidates](https://www.vals.ai/blogs/room-temperature-magnetic-semiconductors): Explore two room-temperature antiferromagnetic semiconductor candidates found by Claude Opus 5.5 agents, with calculations, code and caveats.
- [Vals Web Search Index: Evaluating Search for Real Work](https://www.vals.ai/blogs/web-search-index): Why Vals AI built the Vals Web Search Index: evaluating search tools on legal and finance research while holding the model and agent harness fixed.
- [Has the Bitter Lesson Come for AI Detectors?](https://www.vals.ai/blogs/ai-detection-benchmark): Compare AI writing detectors and LLMs on private human and AI-written text. Explore detection accuracy, false positives and adversarial rewrites.
- [A Lean Proof of the Thomson Problem for Seven Electrons](https://www.vals.ai/blogs/thomson-n7-lean-proof): Explore a Lean proof of the Thomson problem for seven electrons, produced by Claude Sonnet 5.5 agents, with the proof code and verification checks.
- [Evaluating AI Safety in Teen Conversations](https://www.vals.ai/blogs/evaluating-ai-safety-in-teen-conversations): How safe are AI chatbots for teens? Explore findings from 648 simulated conversations as teens disclose more, resist advice and seek help.
- [A Faster Shortest Path Algorithm](https://www.vals.ai/blogs/faster-shortest-path-algorithm): How Claude Opus 5.5 agents developed C-HD, a shortest-path algorithm with a Lean proof. Explore the result, assumptions and verification.
- [AI Cheating is on the Rise](https://www.vals.ai/blogs/cheating-on-the-rise): An integrity audit across BioMysteryBench, Terminal-Bench 2.1, and SWE-bench Verified shows that cheating increasingly complicates evaluation.
- [Vals Environmental Impacts Report](https://www.vals.ai/blogs/vals-environmental-impacts-report): Explore estimates of LLM energy use, carbon emissions and water consumption on Vals Index tasks, with model comparisons and methodology.
- [Claude Fable 5.1 Solves the Cyphral Distich](https://www.vals.ai/blogs/fable-solves-cyphral-distich): Explore Claude Fable 5.1's proposed solution to Sir Thomas Urquhart's Cyphral Distich, a centuries-old cipher, with the reasoning behind the result.
- [Series A: Always a Higher Peak](https://www.vals.ai/blogs/series-a): Vals AI announces a $40M Series A at a $400M valuation led by a16z, alongside new releases including Vals Smith, Frontier Risk Benchmarks, and Vals 2.0.
- [Child AI Safety](https://www.vals.ai/blogs/child-ai-safety): How Vals AI is developing youth-specific AI safety evaluations with clinicians and researchers to assess chatbot risks for children and teens.
- [Vals Public Sector Memo](https://www.vals.ai/blogs/vals-public-sector-memo): Introducing Vals Public Sector: independent AI evaluation for government, with expert-built benchmarks, frontier risk measurement and national security work.
- [Measuring Healthcare AI Where It Actually Works](https://www.vals.ai/blogs/measuring-healthcare-ai-where-it-actually-works): Compare healthcare AI on clinical SOAP notes and medical coding. Explore MedScribe and MedCode findings, including the gap between documentation and billing.
- [Behind the Scenes of Vibe Code Bench](https://www.vals.ai/blogs/b