We benchmarked 12 models on 1,913 real vulnerabilities. Here's the scoreboard.

Picking a model to drive a security scanner ought to be a benchmark question, and there are public benchmarks for it. The trouble is what they measure.

CyberGym is the careful version of the genre. It builds 1,507 instances out of real OSS-Fuzz vulnerabilities across 188 projects, and the task is genuinely hard: the agent reads a vulnerability description, gets the codebase at its pre-patch commit, and has to write a proof-of-concept that crashes the unpatched build but not the patched one.

It’s also a poor guide for application security. The corpus is C and C++, so it says nothing about the Python, Java, PHP, and JavaScript most product code is written in. The bugs are overwhelmingly memory safety, and a sanitizer crash is not the same thing as a vulnerability someone can exploit in a web app. And it has been public long enough that the models we’d be grading have most likely trained on it.

Corgea runs an AI engine (“fusion”) that scans code, triages findings, and writes fixes. The hardest question we get asked is which model should drive the scan, and nothing public answered it for the kind of code our customers actually ship. So we built our own: a fixed corpus, ground truth that’s curated and versioned, and mechanical scoring.

We ran 12 model profiles against the same corpus. Which model won turned out to be less interesting than which column of the table people usually read first.

What we ran

Only one variable changes between runs: the model bundle that drives the vulnerability scanner. The fusion code is pinned to a single commit, and the harness hard-aborts a run if the checkout has drifted, because a moving engine would turn an apples-to-apples comparison into noise.

Each profile is a flat list of roles mapped to model IDs. The vulnerability scanner is the role under test:

# Swap the detector, leave every triage role on the default bundle
vuln_scan = "gpt-5.6-luna"

# Flagship single-model profile
vuln_scan = "claude-opus-4-8"

Each profile runs across the full corpus. We collect every finding and score it against ground truth, and we fix the list of files that count before the run instead of trimming it once we’ve seen the results.

The corpus

The benchmark scores source code rather than config. The harness owns an allowlist of extensions (.py, .java, .js, .php, .html, .twig, .c, .h, .cs, .cshtml) and drops anything else from both sides of the join. IaC and config files (.tf, .yaml, .properties) belong to a separate scanner and stay out of scope. We say so up front because “we found 4,000 issues” is easy when you count every YAML key.

There are 1,913 ground-truth cases across 55 fixtures, 8 languages, and 95 distinct CWEs. The fixtures come from three places. Some are apps we built ourselves, like an insecure payment API and a coffee-shop app ported across Java, Node, PHP, and C#, where we know exactly what was planted and where. Others are real open-source vulnerable apps: pygoat, DVGA, VAmPI, deliberately-vulnerable Flask and Tornado training apps, and dozens more, all messy in the way production code is messy. The rest are imported CVE-derived cases, woven from public vulnerability datasets.

Every expected case is one row in a versioned inventory with a stable ID, a canonical CWE, the file path, the target line, and whether it’s a true positive (a real vulnerability the scanner should flag) or a hard negative (a spot that looks vulnerable but isn’t, bait for false positives).

That last column is what keeps the benchmark honest. A scanner that flags everything scores perfect recall and is still useless. By design the corpus holds 1,562 expected true positives (82%) and 351 hard negatives (18%), which gives us something to measure precision against.

The CWE spread covers the bugs that actually ship: cross-site scripting, SQL injection, missing authorization, CSRF, SSRF, path traversal, command injection, hardcoded secrets, open redirects.

The headline numbers use only the independent slice of the corpus. 565 cases were minted while reviewing the scanners’ own output, and we exclude all of them, because ground truth derived from the tools you’re grading is circular.

How we score

We match each scanner finding against the expected cases with a strict join. A finding credits a case only when all four of these hold:

1. Same fixture. A finding in insecure-api/main.py can’t match a case in insecure-payment-api/main.py, so basename collisions never cross-match.
2. The finding’s CWE equals the case’s canonical CWE, or one of its listed acceptable alternates.
3. The finding’s file matches the case’s path.
4. The finding lands within 25 lines of the target, or its line span brackets the target.

Matching is a 1:1 assignment. Each finding credits at most one case; each case is claimed by at most one finding. Exact-CWE matches always win over alternate-CWE matches, and the closest finding wins. The result is deterministic: re-run it and you get a byte-identical scoreboard.

Every case lands in one of four cells:

A scan that runs and finds nothing writes a sentinel row, so we can tell “scanned, found nothing” (a real FN or TN) from “never scanned” (blank). That way we never have to infer coverage.

Scanners also disagree on taxonomy. A SQL injection labeled CWE-89 is still a real catch if the scanner calls it CWE-943. So each case can list acceptable alternate CWEs, and we mark those matches with an asterisk (TP* / FP*, meaning right location but alternate CWE) to keep the drift visible. The lenient view counts TP* as a hit; the strict view reclassifies every asterisked match as a miss. The leaderboard above is lenient.

From the cells we compute precision, recall, and F1. Non-overlapping confidence intervals aren’t enough for us to call a difference significant, since the cases cluster by fixture. For significance we run pairwise McNemar tests over each pair’s common coverage, adjust for the fixture-cluster design effect, apply a Holm correction, and confirm the ranking survives leaving any single fixture out. We publish a ranking claim only when it clears that bar.

The scoreboard

Lenient scoring, independent corpus slice. Profiles below 90% coverage are held out of the ranking.

Start with the precision column. Every ranked profile clears 90%, and all twelve sit between 94.8% and 96.8%, which is far too narrow a range to separate them. On this corpus, precision is effectively a solved problem.

What does separate the profiles is recall and cost, and the top one isn’t the flagship model. It’s GPT-5.6 Luna (gpt-5-6-luna-vuln-scan, vuln-scan only), which hits 40.0% recall at $0.133 per fixture, one of the cheapest bills on the board. Claude Opus 4.8 lands ninth, at 34.9% recall. The whole ranked field fits in a 7.5-point recall band, from DeepSeek V4 Flash at 32.5% to Luna at 40.0%.

baseline and claude-opus-4-8 are a statistical tie on recall. Over their common coverage of 1,174 cases across 54 fixtures, the two differ by 0.003. If you’re paying for the flagship expecting it to beat the default bundle, this corpus doesn’t show that.

Cost

Per fixture, GPT-5.6 Luna costs $0.133 and claude-opus-4-8 costs $2.203, about 17× more. Across the full corpus that works out to $118.97 for Claude Opus 4.8 against $7.17 for GPT-5.6 Luna.

For that 17× premium you get lower recall (34.9% versus Luna’s 40.0%) and precision that’s a rounding error apart. The floor goes lower still. DeepSeek V4 Flash scans a fixture for $0.018, two orders of magnitude under Opus, and still finishes within 2.4 points of it on recall. The cheap profiles and the most expensive one land in the same quality band, so on this corpus the extra spend isn’t buying extra catches.

Limitations

We’d rather state these than have you find them.

• These are SAST detection scores on a curated corpus, so don’t read them as field guarantees. Strong recall here does not promise strong recall on your code, because vulnerable-by-design apps bury the same bug under far less noise than production does. Treat these numbers as a ceiling on what you’d see in your own repo.
• 565 of the 1,913 cases were minted from the scanners’ own output. Grading a tool on ground truth it helped create is circular, so those stay out of the headline numbers.
• Our cost figures cover input and output spend, stamped with that basis. Token counts can be approximate for backends that don’t return a usage block, and GLM 5.2 reads n/a because no verified rate is configured for it.
• IaC and config files are out of scope here by design.
• Ground-truth labels are human judgments, and a curated label is still a label. We version the inventory, so relabeling a case leaves a trail instead of quietly moving a score.
• These results describe one pinned version of fusion. A prompt change can move every number in the table, which is exactly why we pin the commit and snapshot the run.

What’s next

We want a wider corpus: more languages, more real CVE-derived cases, and more hard negatives. The hard negatives are the expensive part to build, and also the part that keeps a benchmark honest.

Detection is only the first leg of what fusion does, so we’re extending the harness to score triage accuracy (did the model correctly call a finding a false positive?) and fix quality (did the patch close the bug without breaking the build?). A model that finds bugs but writes bad patches is a different tool than the scoreboard above suggests.