Software economics and Korea inbound research, from Seoul.

Two-panel benchmark chart: precision against recall for 14 rows, and tokens burned per real defect for six model pairs

Alibaba Open Code Review vs Claude Code 2026: 4.7x Precision at 1/15 Tokens

30-Second Verdict — Direct Answer

Short answer: Alibaba open-sourced its internal AI reviewer on Apache-2.0, and it publishes a fraction of a general agent’s comments. On its 200-PR benchmark with Claude-4.6-Opus on both sides, it posts 33.90% precision at 385K tokens and 1m23s; Claude Code posts 7.23% at 5,664K and 13m06s. The bill is recall: 20.00% of 1,505 defects against 28.90%.

What your review volume costs in comments to read

interactive

Per-PR comment counts come from the 14-row AACR-Bench table below (Claude-4.6-Opus under each scaffold). Seconds per comment is your number, not the vendor’s.

PRs
sec
Open Code Review comments / month—889 comments across 200 PRs
of which false alarms—33.9% precision on the same model
OCR triage hours / month—False alarms only
Claude Code comments / month—5,980 comments across 200 PRs
of which false alarms—7.2% precision, same model
Claude Code triage hours / month—False alarms only
OCR tokens / month—385K per review, 200-PR run
Claude Code tokens / month—5,664K per review, same run

Per-PR constants are arithmetic on the published benchmark columns: 889 and 5,980 comments, 301 and 435 true positives, across 200 PRs. Triage seconds are a modelling assumption and the input that moves the answer most. Tokens are the vendor chart’s average per review.

Alibaba's internal AI code review assistant served tens of thousands of developers for two years before the company published it as open source. That history is the interesting part, because the tool is not a prompt. It is a pipeline with hard gates in front of the model, and the benchmark shipped with the repository is built to show what those gates buy.

What Alibaba actually open-sourced

The repository is alibaba/open-code-review, a Go CLI installed as @alibaba-group/open-code-review and invoked as ocr. As of 16 November 2026 it holds 52,622 stars, 4,226 forks and 340 open issues, with the last push the same day. Apache-2.0, no seat count, no licence key, no paid tier behind a feature flag. That last sentence is why a one-person studio can try it this afternoon without asking anyone for budget.

The shape is a normal diff reviewer with three additions. ocr review reads a workspace, a commit or a branch range. ocr scan reviews whole files when there is no useful diff, which is what you want when you inherit a codebase. Delegation mode lets your existing coding agent's model do the thinking, so a Claude Code or Codex subscriber needs no second API key. Requirements are small: Git 2.41 or newer, Node 18 or newer, one model endpoint.

What you getDetail read from the repo on 16 Nov 2026
LicenceApache-2.0, Copyright 2026 Alibaba
Installnpm install -g @alibaba-group/open-code-review, or install script, release binary, source
PrerequisitesGit >= 2.41, Node >= 18, one LLM key (none in delegation mode)
Built-in rule files65 language and file-type rule sets (go.md, java.md, terraform.md, solidity.md, protobuf.md, and 60 more) plus a default
IntegrationsClaude Code, Codex, Cursor, Kimi Code, OpenCode plugins; GitHub Actions, GitLab CI, Gerrit, Bitbucket, Codeup, GitFlic
OutputHuman text, JSON for another agent, SARIF, resumable sessions, a local session viewer

The benchmark: 50 repositories, 200 pull requests, 1,505 defects

The numbers below come from the chart published in the repository, not from a press release. AACR-Bench draws 200 real pull requests from 50 popular open-source repositories across 10 languages, and 80-plus senior engineers cross-validated 1,505 annotated ground-truth issues. Every row is the same set of PRs, so precision, recall and token counts are comparable across models and across scaffolds.

#ModelVendorScaffoldF1PrecisionRecallAvg timeTokensComments (real / published)
1Claude-4.6-OpusAnthropicOpen Code Review25.10%33.90%20.00%1m23s385K301 / 889
2Qwen3.8-MaxAlibabaOpen Code Review23.00%33.90%17.40%5m14s334K262 / 774
3GLM-5.2Zhipu AIOpen Code Review21.30%32.30%15.90%7m58s682K239 / 741
4Qwen3.7-MaxAlibabaOpen Code Review21.20%25.20%18.30%4m41s625K276 / 1,096
5GPT-5.5OpenAIOpen Code Review21.00%32.10%15.50%2m51s422K234 / 728
6GLM-5.1Zhipu AIOpen Code Review20.40%28.90%15.70%4m11s743K237 / 820
7Claude-4.8-OpusAnthropicOpen Code Review17.90%37.80%11.70%1m06s352K176 / 465
8Deepseek-V4-ProDeepSeekOpen Code Review17.90%30.60%12.70%6m28s394K191 / 624
9Claude-4.8-OpusAnthropicClaude Code v2.1.16914.13%15.93%12.70%5m38s2,062K191 / 1,200
10Qwen3.7-MaxAlibabaClaude Code v2.1.16912.17%8.23%23.37%8m06s5,153K351 / 4,260
11GLM-5.1Zhipu AIClaude Code v2.1.16911.93%8.37%20.80%14m10s4,038K313 / 3,742
12Claude-4.6-OpusAnthropicClaude Code v2.1.16911.57%7.23%28.90%13m06s5,664K435 / 5,980
13Deepseek-V4-ProDeepSeekClaude Code v2.1.16910.93%8.27%16.13%14m24s5,450K243 / 2,945
14GPT-5.5OpenAICodex v0.140.08.36%27.82%4.92%2m58s525K74 / 266

Read the top and the bottom of that table together and the pattern is not "open source beats commercial". It is that the scaffold dominates the model. The same Claude-4.6-Opus weights sit at rank 1 with 25.10% F1 and at rank 12 with 11.57%. Qwen3.7-Max moves from 21.20% to 12.17%. The model did not change between those rows. The instructions around it did.

Same weights, two scaffolds: the six pairs that matter

Six models appear twice in the run: once behind Open Code Review, once behind a general-purpose agent (Claude Code v2.1.169, or Codex v0.140.0 for GPT-5.5). Those pairs are the only fair comparison in the table, and they are where the trade-off becomes legible.

Same weights, two scaffoldsOpen Code Review
F1 / precision / recall
General agent
F1 / precision / recall
Tokens
OCR vs agent
Time
OCR vs agent
Claude-4.6-Opus
Claude Code v2.1.169
25.10% / 33.90% / 20.00%11.57% / 7.23% / 28.90%385K vs 5,664K (14.7x)1m23s vs 13m06s (9.5x faster)
Claude-4.8-Opus
Claude Code v2.1.169
17.90% / 37.80% / 11.70%14.13% / 15.93% / 12.70%352K vs 2,062K (5.9x)1m06s vs 5m38s (5.1x faster)
Qwen3.7-Max
Claude Code v2.1.169
21.20% / 25.20% / 18.30%12.17% / 8.23% / 23.37%625K vs 5,153K (8.2x)4m41s vs 8m06s (1.7x faster)
GLM-5.1
Claude Code v2.1.169
20.40% / 28.90% / 15.70%11.93% / 8.37% / 20.80%743K vs 4,038K (5.4x)4m11s vs 14m10s (3.4x faster)
Deepseek-V4-Pro
Claude Code v2.1.169
17.90% / 30.60% / 12.70%10.93% / 8.27% / 16.13%394K vs 5,450K (13.8x)6m28s vs 14m24s (2.2x faster)
GPT-5.5
Codex v0.140.0
21.00% / 32.10% / 15.50%8.36% / 27.82% / 4.92%422K vs 525K (1.2x)2m51s vs 2m58s (1.0x faster)

The median pair spends 7.1x more tokens per review on the general-agent side and gets 3.3x worse precision. The vendor's README says roughly one-ninth of the tokens and significantly higher precision, and the per-model table supports that direction, with one honest exception: GPT-5.5 under Codex already reports few comments, so its token ratio is only 1.24x and its precision is close to Open Code Review's. Where the general agent is verbose, the gap is 14.7x.

Two routes from the same diff to a comment

diagram
Two routes from the same diff to a comment
Figure 1. Open Code Review gates, bundles and re-checks positions before publishing; a general agent prompts, loops and publishes. Comment counts per PR are the Claude-4.6-Opus pair from the benchmark table.

Where the precision comes from: gates before the model

The design idea is simple to state and hard to ship: steps that must not go wrong are handled by code, and the model is left to decide only what needs judgement. Four of those steps are visible in the published documentation, with their actual constants.

File selection is a six-gate filter. Binaries, secret paths (.env, .ssh, .npmrc), your own excludes, unsupported extensions and built-in default paths are dropped before a token is spent. Test files, vendored code and generated protobuf are excluded by default, which is why a 400-file pull request does not become a 400-file prompt. A single file whose diff alone exceeds 80% of the token ceiling is excluded as too_large rather than truncated.

Files are bundled, not queued. One LLM call looks at file metadata only, never diff content, and groups related changes into review units of at most 10 files: handler plus service plus test, or a rename plus its call sites. Each bundle runs as its own sub-agent with its own context, in parallel, eight at a time by default. That is what makes coverage survive a large changeset.

Rules are matched per file, not pasted into one prompt. A Go file gets go.md, a Java mapper gets the SQL-injection rules, an unfamiliar extension falls back to default.md. There are 65 of those files in the binary, and a project can add its own .opencodereview/rule.json with per-path rules, optionally merged with the built-in ones.

Positions are checked after the model writes. Independent positioning and reflection passes clamp every comment to a real line in the diff. That is the difference between "this looks risky" and a comment that lands on the line the author can actually fix, and it is the part general agents most often get wrong on large changesets.

“Compared to general-purpose agents (Claude Code), Open Code Review achieves significantly higher Precision and F1 with the same underlying model, while consuming only ~1/9 of the tokens and completing reviews faster. Note that its Recall is lower than general-purpose agents — a deliberate trade-off favoring precision over noise.”

The project README, alibaba/open-code-review, read 16 November 2026

The Honest Catch: what the missing recall costs

Recall is the number the marketing does not lead with, and it is the one that decides whether this is safe for your team. On the same weights, the tool gives up 8.9 points of recall against Claude Code. In absolute terms: 301 real defects found instead of 435, out of the same 1,505. That is 134 defects per 200 pull requests that a general agent would have flagged and this pipeline did not.

The other side of the same accounting is what your team reads. Claude Code published 5,980 comments to land those 435 true positives, which is 13.75 comments per real defect. Open Code Review published 889 to land 301, or 2.95 per real defect. Per review that is 29.9 comments against 4.4.

Claude-4.6-Opus, 200 PRsComments per PRReal defects per PRFalse alarms per PRComments per real defectTokens per real defect
Open Code Review4.451.502.942.951,279
Claude Code v2.1.16929.902.1727.7313.7513,021

Two things follow. First, the cost comparison is not the token line on your invoice; it is the hours your engineers spend deciding whether a comment matters. At 30 seconds per comment and 300 pull requests a month, the false alarms alone are 69 hours of reading for the general agent and 7.4 hours for Open Code Review. If your engineers cost more than the model does, the ROI of that gap is immediate and it shows up in the sprint, not in the invoice.

Second, and this is the part that argues for the tool: a pipeline that misses 134 defects per 200 PRs is only acceptable if something else catches them. Usually "something else" is a human reviewer who now spends that attention on 5,545 fewer false alarms in the same 200 pull requests.

What surprised me is how directly the benchmark reads as an argument about attention rather than about model quality. The strongest model in the pair, by F1, is not the one with the best precision. Precision is a property of the process that decides what gets published.

What it costs to run, in the units you pay in

Per review, the smallest token count in the table is Qwen3.8-Max at 334K, the fastest wall-clock is Claude-4.8-Opus at 1m06s, and the most expensive general-agent row is Claude-4.6-Opus at 5,664K tokens and 13m06s. Wall-clock matters for CI: a 13-minute review job blocks a merge queue in a way an 83-second job does not. On a busy repository that difference is the whole argument for a dedicated reviewer over a general agent with a review skill.

Model pricing is yours to plug in, and this article does not quote a rate card it cannot verify. What the table gives you is the token volume and the defect yield, which is enough: divide your provider's input and output prices into the token column and you have a per-review cost. The calculator above does the same arithmetic on comments instead of tokens, because that is the cost line most teams forget to budget.

Who should skip it

To be honest, three teams should not switch this week. The first is anyone whose main complaint is that the reviewer misses bugs. This pipeline trades recall for precision on purpose; if you need the widest net, a general agent on the same model finds more, and the cheaper alternative is not a different tool but a different setting of the same one.

The second is a team that will not maintain the rule files. The precision comes from 65 language rule sets plus whatever you write for your own paths. Nobody owning that configuration means you are paying the token cost of a deterministic scaffold and getting the noise of an unconfigured prompt. The software is free forever under Apache-2.0, but the configuration is not free.

The third is a repository where PR volume sits under about 60 a month and nobody triages comments today. At that volume the time saved is minutes, and a reviewer nobody reads is worse than none: it teaches the team to ignore bot comments on the PRs that matter.

How to put it in CI this week

The install is one npm line, and the interesting flags are the ones that let you see the filters before you spend money. ocr review --preview prints which files would be reviewed and which were dropped, without a token leaving your machine. Run that against your three most painful pull requests first; the output is the argument for or against the tool, before any model bill.

Four copy-ready blocks: install, configure, review, wire into CI

copy-ready

Commands, variable names and flags are the ones in the project's README and QuickStart page, read on 16 November 2026. Swap the model ids for the ones your provider offers.

install.sh
# Git 2.41+ and Node 18+ first, then:
npm install -g @alibaba-group/open-code-review
ocr version

# other routes: install script, GitHub release binary, from source
# see open-codereview.ai/docs/installation
provider.sh
# interactive: pick provider, paste key, model, then a live connectivity test
ocr config provider
ocr config model
ocr llm test

# non-interactive (CI, no TUI): writes the same config file
ocr config set provider                    anthropic
ocr config set model                       your-model-id
ocr config set providers.anthropic.api_key sk-ant-xxxxxxxxxx
review.sh
cd path/to/your-repo

# what would be reviewed, without spending a token
ocr review --preview

# workspace: staged + unstaged + untracked
ocr review

# branch range (merge-base of main and the feature branch)
ocr review --from main --to feature-branch

# whole files, no diff needed: audit an unfamiliar repo or a directory
ocr scan --path internal/agent

# machine-readable output for another agent or a CI step
ocr review --format json --audience agent > review.json

# delegation mode: the host agent's model does the review, no OCR key
ocr delegate preview
ocr-review.yml
# .github/workflows/ocr-review.yml (from examples/github_actions)
name: OpenCodeReview PR Review
on:
  pull_request_target:
    types: [opened, synchronize, reopened]
permissions:
  contents: read
  pull-requests: write
jobs:
  code-review:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: alibaba/open-code-review@main
        with:
          llm_url: ${{ secrets.OCR_LLM_URL }}
          llm_auth_token: ${{ secrets.OCR_LLM_AUTH_TOKEN }}
          llm_model: ${{ vars.OCR_LLM_MODEL }}
          llm_use_anthropic: ${{ vars.OCR_LLM_USE_ANTHROPIC }}

The action reads the diff only and does not execute code from the pull request, which is why the example runs on pull_request_target with secrets available. Names in capitals are yours to set.

The action also exposes four comment-posting modes in action.yml, including a sticky summary that updates one comment as the pull request changes. That matters on a busy repository: a bot that stacks a fresh review on every push trains people to collapse the thread, and a collapsed review is a review nobody reads.

Left panel plots precision against recall for all 14 benchmark rows split into Open Code Review and general-agent scaffolds; right panel shows tokens burned per real defect on a log scale, with Open Code Review lower in every pair
Figure 2. Every row of the vendor's benchmark, placed by precision against recall, then priced in tokens per real defect. The blue cluster is one scaffold, the amber cluster another; the weights are shared inside each pair. Sources and the arithmetic are printed at the top of the figure.

Frequently asked questions

Is Alibaba Open Code Review free?

The CLI is Apache-2.0 and free forever: no seat count, no licence key, no paid tier gating the reviewer. You pay the model provider for tokens, or nothing extra if you use delegation mode with a coding agent you already subscribe to. The 65 built-in rule files and the file filter ship with the binary.

How is this different from running Claude Code with a review skill?

The difference is where the constraints live. A skill is a prompt: the model decides which files to read and when to stop. Open Code Review puts file selection, bundling, rule matching, position checking and the token ceiling in code. On the same Claude-4.6-Opus weights that shows up as 33.90% precision against 7.23%, at 385K tokens against 5,664K, with recall 8.9 points lower.

What does the lower recall mean in practice?

301 of 1,505 annotated defects found instead of 435, per the same 200 pull requests. Per PR that is about 1.5 real defects caught against 2.2. Treat the reviewer as a filter that removes noise from human review, not as the only reviewer.

Does it send my code to Alibaba?

No. It is a CLI you run locally: it reads your git diff, calls the model endpoint you configure (OpenAI, Anthropic, Bedrock or a custom provider), and writes comments to your terminal or a JSON file. The published security case documents the trust boundaries, including TLS-only transport, a host-header allowlist on the local viewer, and path validation before any file write.

Can I review code without a git history?

Yes. ocr scan reviews whole files rather than diffs, which is how you audit an unfamiliar repository or a directory that has no meaningful diff. ocr review --preview shows what would be sent before anything is.

Is the benchmark independent?

No. AACR-Bench was built and published by the same team that wrote the tool, with a public dataset on Hugging Face for anyone to rerun. It is a vendor benchmark with unusually transparent mechanics: 50 repositories, 200 pull requests, 1,505 cross-validated defects, all 14 rows published including the ones where a competitor wins on a metric. That is more than most tools publish, and less than an independent audit.

Two more pieces of the same accounting are already on the site. The teardown of why agent code review misses whole files explains the coverage failure this pipeline is built to prevent, and the cost-per-session calculator handles the part that follows: what a token ceiling does to a flat plan. If you are pricing a review job across a whole engineering team, the TCO simulator takes the same numbers further, and the reasoning-token billing guide covers why the output column is not the one to optimise.

Methodology & Disclosure: Repository statistics, licence, prerequisites, CLI flags, file-filter rules and the 65 rule files were read on 16 November 2026 from github.com/alibaba/open-code-review and its documentation pages for architecture, review rules and quick start. All 14 benchmark rows, including precision denominators, are transcribed from the benchmark chart published in that repository; the AACR-Bench dataset is public at huggingface.co/datasets/Alibaba-Aone/aacr-bench. Per-review and per-defect figures are arithmetic on those columns, not vendor claims. The only modelling assumption in this article is triage time per comment, and it is labelled wherever it appears. Every table and both panels of the figure are generated by scripts/make_ocr_bench_figures_20261116.py from data/ocr_benchmark_20261116.json, so no number was typed by hand.

No first-person production testing is claimed: the benchmark numbers are Alibaba's, run on their own pipeline, and this article has not re-run them. Any partner link on this publication carries rel="sponsored nofollow". Review rules live in the Editorial Methodology and verified corrections are logged in the Corrections Log.

Leave a Reply