Short answer: Alibaba open-sourced its internal AI reviewer on Apache-2.0, and it publishes a fraction of a general agent’s comments. On its 200-PR benchmark with Claude-4.6-Opus on both sides, it posts 33.90% precision at 385K tokens and 1m23s; Claude Code posts 7.23% at 5,664K and 13m06s. The bill is recall: 20.00% of 1,505 defects against 28.90%.
What your review volume costs in comments to read
interactivePer-PR comment counts come from the 14-row AACR-Bench table below (Claude-4.6-Opus under each scaffold). Seconds per comment is your number, not the vendor’s.
Per-PR constants are arithmetic on the published benchmark columns: 889 and 5,980 comments, 301 and 435 true positives, across 200 PRs. Triage seconds are a modelling assumption and the input that moves the answer most. Tokens are the vendor chart’s average per review.
Alibaba's internal AI code review assistant served tens of thousands of developers for two years before the company published it as open source. That history is the interesting part, because the tool is not a prompt. It is a pipeline with hard gates in front of the model, and the benchmark shipped with the repository is built to show what those gates buy.
What Alibaba actually open-sourced
The repository is alibaba/open-code-review, a Go CLI installed as @alibaba-group/open-code-review and invoked as ocr. As of 16 November 2026 it holds 52,622 stars, 4,226 forks and 340 open issues, with the last push the same day. Apache-2.0, no seat count, no licence key, no paid tier behind a feature flag. That last sentence is why a one-person studio can try it this afternoon without asking anyone for budget.
The shape is a normal diff reviewer with three additions. ocr review reads a workspace, a commit or a branch range. ocr scan reviews whole files when there is no useful diff, which is what you want when you inherit a codebase. Delegation mode lets your existing coding agent's model do the thinking, so a Claude Code or Codex subscriber needs no second API key. Requirements are small: Git 2.41 or newer, Node 18 or newer, one model endpoint.
| What you get | Detail read from the repo on 16 Nov 2026 |
|---|---|
| Licence | Apache-2.0, Copyright 2026 Alibaba |
| Install | npm install -g @alibaba-group/open-code-review, or install script, release binary, source |
| Prerequisites | Git >= 2.41, Node >= 18, one LLM key (none in delegation mode) |
| Built-in rule files | 65 language and file-type rule sets (go.md, java.md, terraform.md, solidity.md, protobuf.md, and 60 more) plus a default |
| Integrations | Claude Code, Codex, Cursor, Kimi Code, OpenCode plugins; GitHub Actions, GitLab CI, Gerrit, Bitbucket, Codeup, GitFlic |
| Output | Human text, JSON for another agent, SARIF, resumable sessions, a local session viewer |
The benchmark: 50 repositories, 200 pull requests, 1,505 defects
The numbers below come from the chart published in the repository, not from a press release. AACR-Bench draws 200 real pull requests from 50 popular open-source repositories across 10 languages, and 80-plus senior engineers cross-validated 1,505 annotated ground-truth issues. Every row is the same set of PRs, so precision, recall and token counts are comparable across models and across scaffolds.
| # | Model | Vendor | Scaffold | F1 | Precision | Recall | Avg time | Tokens | Comments (real / published) |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude-4.6-Opus | Anthropic | Open Code Review | 25.10% | 33.90% | 20.00% | 1m23s | 385K | 301 / 889 |
| 2 | Qwen3.8-Max | Alibaba | Open Code Review | 23.00% | 33.90% | 17.40% | 5m14s | 334K | 262 / 774 |
| 3 | GLM-5.2 | Zhipu AI | Open Code Review | 21.30% | 32.30% | 15.90% | 7m58s | 682K | 239 / 741 |
| 4 | Qwen3.7-Max | Alibaba | Open Code Review | 21.20% | 25.20% | 18.30% | 4m41s | 625K | 276 / 1,096 |
| 5 | GPT-5.5 | OpenAI | Open Code Review | 21.00% | 32.10% | 15.50% | 2m51s | 422K | 234 / 728 |
| 6 | GLM-5.1 | Zhipu AI | Open Code Review | 20.40% | 28.90% | 15.70% | 4m11s | 743K | 237 / 820 |
| 7 | Claude-4.8-Opus | Anthropic | Open Code Review | 17.90% | 37.80% | 11.70% | 1m06s | 352K | 176 / 465 |
| 8 | Deepseek-V4-Pro | DeepSeek | Open Code Review | 17.90% | 30.60% | 12.70% | 6m28s | 394K | 191 / 624 |
| 9 | Claude-4.8-Opus | Anthropic | Claude Code v2.1.169 | 14.13% | 15.93% | 12.70% | 5m38s | 2,062K | 191 / 1,200 |
| 10 | Qwen3.7-Max | Alibaba | Claude Code v2.1.169 | 12.17% | 8.23% | 23.37% | 8m06s | 5,153K | 351 / 4,260 |
| 11 | GLM-5.1 | Zhipu AI | Claude Code v2.1.169 | 11.93% | 8.37% | 20.80% | 14m10s | 4,038K | 313 / 3,742 |
| 12 | Claude-4.6-Opus | Anthropic | Claude Code v2.1.169 | 11.57% | 7.23% | 28.90% | 13m06s | 5,664K | 435 / 5,980 |
| 13 | Deepseek-V4-Pro | DeepSeek | Claude Code v2.1.169 | 10.93% | 8.27% | 16.13% | 14m24s | 5,450K | 243 / 2,945 |
| 14 | GPT-5.5 | OpenAI | Codex v0.140.0 | 8.36% | 27.82% | 4.92% | 2m58s | 525K | 74 / 266 |
Read the top and the bottom of that table together and the pattern is not "open source beats commercial". It is that the scaffold dominates the model. The same Claude-4.6-Opus weights sit at rank 1 with 25.10% F1 and at rank 12 with 11.57%. Qwen3.7-Max moves from 21.20% to 12.17%. The model did not change between those rows. The instructions around it did.
Same weights, two scaffolds: the six pairs that matter
Six models appear twice in the run: once behind Open Code Review, once behind a general-purpose agent (Claude Code v2.1.169, or Codex v0.140.0 for GPT-5.5). Those pairs are the only fair comparison in the table, and they are where the trade-off becomes legible.
| Same weights, two scaffolds | Open Code Review F1 / precision / recall | General agent F1 / precision / recall | Tokens OCR vs agent | Time OCR vs agent |
|---|---|---|---|---|
| Claude-4.6-Opus Claude Code v2.1.169 | 25.10% / 33.90% / 20.00% | 11.57% / 7.23% / 28.90% | 385K vs 5,664K (14.7x) | 1m23s vs 13m06s (9.5x faster) |
| Claude-4.8-Opus Claude Code v2.1.169 | 17.90% / 37.80% / 11.70% | 14.13% / 15.93% / 12.70% | 352K vs 2,062K (5.9x) | 1m06s vs 5m38s (5.1x faster) |
| Qwen3.7-Max Claude Code v2.1.169 | 21.20% / 25.20% / 18.30% | 12.17% / 8.23% / 23.37% | 625K vs 5,153K (8.2x) | 4m41s vs 8m06s (1.7x faster) |
| GLM-5.1 Claude Code v2.1.169 | 20.40% / 28.90% / 15.70% | 11.93% / 8.37% / 20.80% | 743K vs 4,038K (5.4x) | 4m11s vs 14m10s (3.4x faster) |
| Deepseek-V4-Pro Claude Code v2.1.169 | 17.90% / 30.60% / 12.70% | 10.93% / 8.27% / 16.13% | 394K vs 5,450K (13.8x) | 6m28s vs 14m24s (2.2x faster) |
| GPT-5.5 Codex v0.140.0 | 21.00% / 32.10% / 15.50% | 8.36% / 27.82% / 4.92% | 422K vs 525K (1.2x) | 2m51s vs 2m58s (1.0x faster) |
The median pair spends 7.1x more tokens per review on the general-agent side and gets 3.3x worse precision. The vendor's README says roughly one-ninth of the tokens and significantly higher precision, and the per-model table supports that direction, with one honest exception: GPT-5.5 under Codex already reports few comments, so its token ratio is only 1.24x and its precision is close to Open Code Review's. Where the general agent is verbose, the gap is 14.7x.
Where the precision comes from: gates before the model
The design idea is simple to state and hard to ship: steps that must not go wrong are handled by code, and the model is left to decide only what needs judgement. Four of those steps are visible in the published documentation, with their actual constants.
File selection is a six-gate filter. Binaries, secret paths (.env, .ssh, .npmrc), your own excludes, unsupported extensions and built-in default paths are dropped before a token is spent. Test files, vendored code and generated protobuf are excluded by default, which is why a 400-file pull request does not become a 400-file prompt. A single file whose diff alone exceeds 80% of the token ceiling is excluded as too_large rather than truncated.
Files are bundled, not queued. One LLM call looks at file metadata only, never diff content, and groups related changes into review units of at most 10 files: handler plus service plus test, or a rename plus its call sites. Each bundle runs as its own sub-agent with its own context, in parallel, eight at a time by default. That is what makes coverage survive a large changeset.
Rules are matched per file, not pasted into one prompt. A Go file gets go.md, a Java mapper gets the SQL-injection rules, an unfamiliar extension falls back to default.md. There are 65 of those files in the binary, and a project can add its own .opencodereview/rule.json with per-path rules, optionally merged with the built-in ones.
Positions are checked after the model writes. Independent positioning and reflection passes clamp every comment to a real line in the diff. That is the difference between "this looks risky" and a comment that lands on the line the author can actually fix, and it is the part general agents most often get wrong on large changesets.
“Compared to general-purpose agents (Claude Code), Open Code Review achieves significantly higher Precision and F1 with the same underlying model, while consuming only ~1/9 of the tokens and completing reviews faster. Note that its Recall is lower than general-purpose agents — a deliberate trade-off favoring precision over noise.”
The project README, alibaba/open-code-review, read 16 November 2026
The Honest Catch: what the missing recall costs
Recall is the number the marketing does not lead with, and it is the one that decides whether this is safe for your team. On the same weights, the tool gives up 8.9 points of recall against Claude Code. In absolute terms: 301 real defects found instead of 435, out of the same 1,505. That is 134 defects per 200 pull requests that a general agent would have flagged and this pipeline did not.
The other side of the same accounting is what your team reads. Claude Code published 5,980 comments to land those 435 true positives, which is 13.75 comments per real defect. Open Code Review published 889 to land 301, or 2.95 per real defect. Per review that is 29.9 comments against 4.4.
| Claude-4.6-Opus, 200 PRs | Comments per PR | Real defects per PR | False alarms per PR | Comments per real defect | Tokens per real defect |
|---|---|---|---|---|---|
| Open Code Review | 4.45 | 1.50 | 2.94 | 2.95 | 1,279 |
| Claude Code v2.1.169 | 29.90 | 2.17 | 27.73 | 13.75 | 13,021 |
Two things follow. First, the cost comparison is not the token line on your invoice; it is the hours your engineers spend deciding whether a comment matters. At 30 seconds per comment and 300 pull requests a month, the false alarms alone are 69 hours of reading for the general agent and 7.4 hours for Open Code Review. If your engineers cost more than the model does, the ROI of that gap is immediate and it shows up in the sprint, not in the invoice.
Second, and this is the part that argues for the tool: a pipeline that misses 134 defects per 200 PRs is only acceptable if something else catches them. Usually "something else" is a human reviewer who now spends that attention on 5,545 fewer false alarms in the same 200 pull requests.
What surprised me is how directly the benchmark reads as an argument about attention rather than about model quality. The strongest model in the pair, by F1, is not the one with the best precision. Precision is a property of the process that decides what gets published.
What it costs to run, in the units you pay in
Per review, the smallest token count in the table is Qwen3.8-Max at 334K, the fastest wall-clock is Claude-4.8-Opus at 1m06s, and the most expensive general-agent row is Claude-4.6-Opus at 5,664K tokens and 13m06s. Wall-clock matters for CI: a 13-minute review job blocks a merge queue in a way an 83-second job does not. On a busy repository that difference is the whole argument for a dedicated reviewer over a general agent with a review skill.
Model pricing is yours to plug in, and this article does not quote a rate card it cannot verify. What the table gives you is the token volume and the defect yield, which is enough: divide your provider's input and output prices into the token column and you have a per-review cost. The calculator above does the same arithmetic on comments instead of tokens, because that is the cost line most teams forget to budget.
Who should skip it
To be honest, three teams should not switch this week. The first is anyone whose main complaint is that the reviewer misses bugs. This pipeline trades recall for precision on purpose; if you need the widest net, a general agent on the same model finds more, and the cheaper alternative is not a different tool but a different setting of the same one.
The second is a team that will not maintain the rule files. The precision comes from 65 language rule sets plus whatever you write for your own paths. Nobody owning that configuration means you are paying the token cost of a deterministic scaffold and getting the noise of an unconfigured prompt. The software is free forever under Apache-2.0, but the configuration is not free.
The third is a repository where PR volume sits under about 60 a month and nobody triages comments today. At that volume the time saved is minutes, and a reviewer nobody reads is worse than none: it teaches the team to ignore bot comments on the PRs that matter.
How to put it in CI this week
The install is one npm line, and the interesting flags are the ones that let you see the filters before you spend money. ocr review --preview prints which files would be reviewed and which were dropped, without a token leaving your machine. Run that against your three most painful pull requests first; the output is the argument for or against the tool, before any model bill.
Four copy-ready blocks: install, configure, review, wire into CI
copy-readyCommands, variable names and flags are the ones in the project's README and QuickStart page, read on 16 November 2026. Swap the model ids for the ones your provider offers.
# Git 2.41+ and Node 18+ first, then:
npm install -g @alibaba-group/open-code-review
ocr version
# other routes: install script, GitHub release binary, from source
# see open-codereview.ai/docs/installation
# interactive: pick provider, paste key, model, then a live connectivity test
ocr config provider
ocr config model
ocr llm test
# non-interactive (CI, no TUI): writes the same config file
ocr config set provider anthropic
ocr config set model your-model-id
ocr config set providers.anthropic.api_key sk-ant-xxxxxxxxxx
cd path/to/your-repo
# what would be reviewed, without spending a token
ocr review --preview
# workspace: staged + unstaged + untracked
ocr review
# branch range (merge-base of main and the feature branch)
ocr review --from main --to feature-branch
# whole files, no diff needed: audit an unfamiliar repo or a directory
ocr scan --path internal/agent
# machine-readable output for another agent or a CI step
ocr review --format json --audience agent > review.json
# delegation mode: the host agent's model does the review, no OCR key
ocr delegate preview
# .github/workflows/ocr-review.yml (from examples/github_actions)
name: OpenCodeReview PR Review
on:
pull_request_target:
types: [opened, synchronize, reopened]
permissions:
contents: read
pull-requests: write
jobs:
code-review:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: alibaba/open-code-review@main
with:
llm_url: ${{ secrets.OCR_LLM_URL }}
llm_auth_token: ${{ secrets.OCR_LLM_AUTH_TOKEN }}
llm_model: ${{ vars.OCR_LLM_MODEL }}
llm_use_anthropic: ${{ vars.OCR_LLM_USE_ANTHROPIC }}
The action reads the diff only and does not execute code from the pull request, which is why the example runs on pull_request_target with secrets available. Names in capitals are yours to set.
The action also exposes four comment-posting modes in action.yml, including a sticky summary that updates one comment as the pull request changes. That matters on a busy repository: a bot that stacks a fresh review on every push trains people to collapse the thread, and a collapsed review is a review nobody reads.

Frequently asked questions
Is Alibaba Open Code Review free?
The CLI is Apache-2.0 and free forever: no seat count, no licence key, no paid tier gating the reviewer. You pay the model provider for tokens, or nothing extra if you use delegation mode with a coding agent you already subscribe to. The 65 built-in rule files and the file filter ship with the binary.
How is this different from running Claude Code with a review skill?
The difference is where the constraints live. A skill is a prompt: the model decides which files to read and when to stop. Open Code Review puts file selection, bundling, rule matching, position checking and the token ceiling in code. On the same Claude-4.6-Opus weights that shows up as 33.90% precision against 7.23%, at 385K tokens against 5,664K, with recall 8.9 points lower.
What does the lower recall mean in practice?
301 of 1,505 annotated defects found instead of 435, per the same 200 pull requests. Per PR that is about 1.5 real defects caught against 2.2. Treat the reviewer as a filter that removes noise from human review, not as the only reviewer.
Does it send my code to Alibaba?
No. It is a CLI you run locally: it reads your git diff, calls the model endpoint you configure (OpenAI, Anthropic, Bedrock or a custom provider), and writes comments to your terminal or a JSON file. The published security case documents the trust boundaries, including TLS-only transport, a host-header allowlist on the local viewer, and path validation before any file write.
Can I review code without a git history?
Yes. ocr scan reviews whole files rather than diffs, which is how you audit an unfamiliar repository or a directory that has no meaningful diff. ocr review --preview shows what would be sent before anything is.
Is the benchmark independent?
No. AACR-Bench was built and published by the same team that wrote the tool, with a public dataset on Hugging Face for anyone to rerun. It is a vendor benchmark with unusually transparent mechanics: 50 repositories, 200 pull requests, 1,505 cross-validated defects, all 14 rows published including the ones where a competitor wins on a metric. That is more than most tools publish, and less than an independent audit.
Two more pieces of the same accounting are already on the site. The teardown of why agent code review misses whole files explains the coverage failure this pipeline is built to prevent, and the cost-per-session calculator handles the part that follows: what a token ceiling does to a flat plan. If you are pricing a review job across a whole engineering team, the TCO simulator takes the same numbers further, and the reasoning-token billing guide covers why the output column is not the one to optimise.
Methodology & Disclosure: Repository statistics, licence, prerequisites, CLI flags, file-filter rules and the 65 rule files were read on 16 November 2026 from github.com/alibaba/open-code-review and its documentation pages for architecture, review rules and quick start. All 14 benchmark rows, including precision denominators, are transcribed from the benchmark chart published in that repository; the AACR-Bench dataset is public at huggingface.co/datasets/Alibaba-Aone/aacr-bench. Per-review and per-defect figures are arithmetic on those columns, not vendor claims. The only modelling assumption in this article is triage time per comment, and it is labelled wherever it appears. Every table and both panels of the figure are generated by scripts/make_ocr_bench_figures_20261116.py from data/ocr_benchmark_20261116.json, so no number was typed by hand.
No first-person production testing is claimed: the benchmark numbers are Alibaba's, run on their own pipeline, and this article has not re-run them. Any partner link on this publication carries rel="sponsored nofollow". Review rules live in the Editorial Methodology and verified corrections are logged in the Corrections Log.


Leave a Reply