Seven files, and the largest direct impact

Code Review Shipped

How the best AI code-review tools actually work, what the published benchmarks really say, and what a review pipeline should borrow from them.

The question

How do the best AI code-review tools actually work — read from installed source and primary documentation rather than from marketing — and what should this project's own reviewer borrow?

What it found

  • Anthropic's own review tooling is a map-reduce with a confidence gate: an eligibility gate, a context-gather, five parallel specialist lenses, and a per-issue confidence scorer — with a verbatim 0/25/50/75/100 rubric and an explicit false-positive catalog.
  • Most tools emit 10–20 comments per pull request, of which roughly 80% is noise. The best measured tools land near 0.9 comments per PR instead.
  • On a CVE benchmark over 200,000 real pull requests, the leaders scored about 84.5% and 80.5% F1 — which is where the "nobody is above ~60% on general bug-finding" caution and the "AI review and rule engines catch different bug classes, so layer them rather than substituting" conclusion both come from.
  • Specialist prompts with teeth outperform general ones: a silent-failure hunter instructed to list EVERY error type a catch block could hide finds things a general reviewer does not.
  • Large diffs must not be a failure mode. Compression approaches lose content; hunk-level map-reduce does not. A reported ~7× token blow-up comes from duplicating context across passes, which pass-by-path avoids.

What the benchmarks actually measured

  • Benchmark leader84.5% F1CVE detection, over 200,000 real pull requests
  • Runner-up80.5% F1
  • General bug-finding, field ceiling60% F1roughly where nobody is above — this project included

The first two are one narrow task on one benchmark; the third is the general case. They are on the same chart precisely because the temptation is to quote the first number and imply the third — the gap between them is the reason this site makes no quality claim it cannot show you.

Comments per pull request

  • Most tools2010–20 per pull request, of which roughly 80% is noise
  • Best measured0.9

More than twenty times the volume, and the low number is the good one. This is the measurement behind handing your own linter output to the reviewers as already-known: volume is not coverage, and a reader who learns to skim the comments has stopped reading them.

What it changed

The most directly traceable pack in the project. The independent confidence scorer, the false-positive catalog (mirrored near-verbatim into team data), the deterministic pre-check stage, the two-tier selection, the eval-scoring harness, and the entire chunker — token budget included — are all implementations of specific recommendations in these files. The one deliberate deviation: the score orders the report here, it does not filter it.

What it recommended and we did not build

A cross-file-impact lens, a git-history lens, and a provenance enum on findings were all recommended and none of them exist. Checked directly rather than assumed.

Sources