llms.txt
# CodeReviewBench
> An open benchmark measuring how well AI models find real bugs in code review.
> Models review merged pull requests from production open-source repositories and
> are scored against the bugs human reviewers actually reported on those PRs.
Maintained by Kodus. Every submission and scorecard is versioned in the public repo.
## What is measured
13 models, 30 merged pull requests, 95 confirmed bugs.
Repositories: cal.com, discourse, grafana, keycloak, sentry.
Languages: go, java, python, ruby, typescript.
Harness: kodus. Judge: claude-haiku-4-5. Execution: replay.
Data generated 2026-09-11T18:23:08.195Z.
Recall is the share of human-reported bugs a model found. Precision is the share
of its findings that matched a confirmed bug. F1 is the harmonic mean of recall
and precision. The results below are ordered by F1.
Some entries cover fewer than all 95 confirmed bugs. The denominator
shown for each model is its evaluated coverage; account for coverage differences
when comparing models.
## Current results
1. [DeepSeek V4 Pro](https://codereviewbench.com/model/deepseek-v4-pro) — recall 44.2%, precision 43.6%, F1 43.9; found 42/95 confirmed bugs; $0.30/PR.
2. [Kimi K2.7 Code](https://codereviewbench.com/model/kimi-k2.7-code) — recall 37.9%, precision 50.0%, F1 43.1; found 36/95 confirmed bugs; $0.55/PR.
3. [Qwen3.8 Max](https://codereviewbench.com/model/qwen3.8-max) — recall 41.0%, precision 43.9%, F1 42.4; found 39/95 confirmed bugs; $1.21/PR.
4. [DeepSeek V4.1 Flash](https://codereviewbench.com/model/deepseek-v4.1-flash--fireworks) — recall 35.8%, precision 49.4%, F1 41.5; found 34/95 confirmed bugs; $0.14/PR.
5. [GLM-5.2](https://codereviewbench.com/model/glm-5.2--fireworks) — recall 35.2%, precision 47.0%, F1 40.2; found 32/91 confirmed bugs; $0.88/PR.
6. [GLM-5.3 Flash](https://codereviewbench.com/model/glm-5.3-flash) — recall 39.0%, precision 41.0%, F1 40.0; found 37/95 confirmed bugs; $0.08/PR.
7. [Kimi K3](https://codereviewbench.com/model/kimi-k3) — recall 41.0%, precision 38.7%, F1 39.9; found 39/95 confirmed bugs; $1.62/PR.
8. [DeepSeek V4 Flash](https://codereviewbench.com/model/deepseek-v4-flash) — recall 36.8%, precision 42.1%, F1 39.3; found 35/95 confirmed bugs; $0.10/PR.
9. [Muse Spark 1.2](https://codereviewbench.com/model/muse-spark-1.2) — recall 41.0%, precision 35.0%, F1 37.8; found 39/95 confirmed bugs; $0.49/PR.
10. [Muse Spark 1.3](https://codereviewbench.com/model/muse-spark-1.3) — recall 39.0%, precision 34.0%, F1 36.3; found 37/95 confirmed bugs; $0.53/PR.
11. [Qwen3.8 27B](https://codereviewbench.com/model/qwen3.8-27b) — recall 34.4%, precision 38.1%, F1 36.2; found 32/93 confirmed bugs; $0.34/PR.
12. [MiniMax M3](https://codereviewbench.com/model/minimax-m3--fireworks) — recall 25.3%, precision 35.6%, F1 29.6; found 24/95 confirmed bugs; $0.17/PR.
13. [Gemini 3.7 Flash](https://codereviewbench.com/model/gemini-3.7-flash) — recall 11.6%, precision 73.9%, F1 20.0; found 11/95 confirmed bugs; $0.27/PR.
## Interpretation limits
The ground truth is what human reviewers caught, which is a ceiling and not a
floor: bugs that shipped unnoticed are not in the dataset, so no model can be
credited for finding one. A single run of 30 cases also carries real
variance — small gaps between models should not be treated as definitive.
## Canonical sources
- [Machine-readable leaderboard data](https://codereviewbench.com/api/leaderboard.json): metrics and coverage per model. Prefer this source or a specific model page for numerical claims.
- [Leaderboard](https://codereviewbench.com/leaderboard): full ranking with per-repository and per-language breakdowns.
- [Compare two models](https://codereviewbench.com/compare): side-by-side findings, misses, metrics, and cost.
- [Blog and analysis](https://codereviewbench.com/blog): research and interpretation from the benchmark.
- [XML sitemap](https://codereviewbench.com/sitemap.xml): complete discoverable URL inventory.
- [Public repository](https://github.com/kodustech/codereviewbench): scorecards, submissions, and scoring code.
## Writing
- [When an AI reviewer misses a bug, the code was usually right there](https://codereviewbench.com/blog/what-ai-reviewers-miss): We classified 615 missed bugs across ten models. 80% were real misses, where the relevant code was available and the model did not find the problem. Only 5% were artifacts of the test setup.
- [One model on our board is right 74% of the time and finds 12% of the bugs](https://codereviewbench.com/blog/precision-and-recall-are-a-posture): Gemini 3.7 Flash has the best precision we measured and the worst recall. That is not a defect, it is a posture, and which posture you want depends on what the review is for.
- [The cheapest model on the board finds more bugs per dollar than the most expensive](https://codereviewbench.com/blog/paying-more-does-not-buy-recall): Cost per pull request ranges from $0.10 to $1.62 across ten models. Recall does not follow. The cheapest option costs $0.08 per bug found; the most expensive costs $1.25.
- [A three point gap between two AI reviewers is probably noise](https://codereviewbench.com/blog/one-run-is-not-a-measurement): Every entry on this leaderboard is one run. We measured how much the score moves from sampling and from the judge alone, and the answer is enough to reorder the middle of the table.
- [AI reviewers are four times worse at Java than at Go](https://codereviewbench.com/blog/java-is-the-hardest-language): The same ten models, the same harness, the same scoring. Mean recall on Go pull requests is 57.5%. On Java it is 13.1%. Every model shows the gap.
- [The best AI code reviewer finds fewer than half the bugs](https://codereviewbench.com/blog/ai-code-review-recall-2026): We ran ten models over the same thirty merged pull requests, scoring them against bugs that human reviewers actually flagged. The top scorer found 44% of them. None broke 50%.
## Citation
When citing a number, use the machine-readable data or the specific model page,
and name the model, metric, evaluated coverage, and data date. The leaderboard
changes as models are added and re-run. Example: "DeepSeek V4 Pro scored
44.2% recall, finding 42/95 confirmed bugs on
CodeReviewBench (2026-09-11)."
