How to Evaluate SAST Tools: A Buyer Pilot Scorecard

Vendor demos are useful for seeing product workflows. They are a poor basis for selecting a SAST tool. The vendor chooses the repository, vulnerability class, configuration, and time limit. Your team inherits the scan time, alert volume, missed findings, and developer response after the contract is signed.

A buyer pilot should answer a narrower question: does this tool produce enough trusted security signal in our repositories and delivery workflow to justify operating it? That requires the same repositories, commits, labels, and scoring rules for every product.

The published Corgea vs. Snyk benchmark and Corgea vs. Aikido benchmark use a fixed deliberately vulnerable repository and count precision, recall, and F1 against reviewed issues. Those tests are useful product evidence, but they do not replace a pilot on your architecture. Read the BLAST SAST whitepaper for Corgea’s analysis approach, then use this process to compare it with the other tools on your shortlist.

Set the rules before any vendor scans

Write a one-page pilot charter before granting source access. Name the repositories, commit SHAs, tools and editions, enabled rules, scan modes, time window, reviewers, and decision owner. Freeze the corpus and configuration during scoring. If a vendor receives tuning help, give every vendor the same type and amount of help, then record it.

Do not compare a deep, manually triggered audit from one product with another product’s default PR check. Package level, scan trigger, execution environment, exclusions, and custom rules can change the result. The Aikido comparison shows why buyers need to distinguish a default scan from a deeper analysis path.

Record the following for every run:

• repository, branch, commit SHA, and scan timestamp
• tool edition, version, configuration, rules, exclusions, and any vendor tuning
• scan trigger and environment, including build command, cache state, and required secrets or generated sources
• raw finding export and the normalized finding-to-case mapping
• reviewer, disposition date, evidence, and a reason for every false-positive or unknown label

Treat a vendor benchmark as a hypothesis, not a procurement result. Corgea’s six-model benchmark, hard-negatives analysis, and reproducible evaluation guide explain why the corpus, negative cases, and exact analysis version matter as much as the headline score.

Choose representative repositories

Use enough code to expose the conditions that affect your decision, without creating a pilot so large that no one can review the output. Three to five repositories often works when they deliberately cover different operating conditions.

Include repositories that contain:

• the languages and frameworks your teams ship, including the primary web, API, worker, and data-processing stacks
• a monorepo with shared packages, service boundaries, code owners, and independent build targets
• each relevant build system, package manager, and generated-source step
• generated code, migrations, test fixtures, vendored code, and the exclusion rules you expect the tool to apply
• a legacy service with custom framework wrappers, older dependencies, incomplete tests, or known scanner noise
• a security-sensitive service with authorization, tenant boundaries, payments, administration, sensitive data access, or internal service-to-service calls

Do not let a tool earn broad language-coverage credit by parsing a small example while failing the build or ignoring the paths your developers change. Capture whether the tool can understand your build graph, resolve dependencies, and handle generated sources. The best SAST tools guide has a practical list of framework, multi-file, generated-code, and custom-sanitizer questions to use in the pilot.

Build a labeled test set

The test set is the source of truth for detection scoring. It should include a case ID, repository and commit, file and line range, vulnerability class, expected behavior, severity or business impact, owner, and review evidence. Keep the evidence private if it contains sensitive code.

Build the set from four groups:

1. Known vulnerabilities. Include recently remediated vulnerabilities, confirmed penetration-test results, accepted bug reports, and carefully seeded cases in a non-production pilot branch.
2. Historical findings. Add issues that prior scanners reported and your team later confirmed, closed, or suppressed. They reveal how a tool behaves in the code your organization actually maintains.
3. Safe patterns. Include code that resembles risky patterns but is safe because of framework behavior, a custom sanitizer, trusted input, an authorization boundary, test-only use, or an unreachable path. These cases measure noise.
4. Business-logic cases. Include authorization, tenancy, workflow-state, price or entitlement, and other application-specific controls. A scanner can perform well on generic injection cases and still miss the conditions that matter most to your product.

Do not put all test cases in one synthetic repository. A deliberately vulnerable project can show whether a tool catches known classes, as the Snyk test does, but your pilot also needs the abstractions, conventions, and safe patterns that generate real triage work.

Use explicit labels

Every reported finding and every known test case needs one final label. Do not call uncertain findings false positives just to improve precision.

Label	Apply it when	Scoring treatment
Confirmed TP	The reported weakness is present and the report maps to a labeled case or independently reviewed vulnerability.	Counts toward precision and recall.
Confirmed FP	The report is unsupported, the protection is effective, the path is not relevant, or the report adds no distinct coverage.	Counts against precision.
Exploitability unknown	Review evidence is incomplete, environment context is unavailable, or the team cannot yet determine whether the weakness is exploitable.	Report separately. Do not count it as TP or FP.
Missed	A labeled known vulnerability has no equivalent finding from the tool.	Counts against recall.
Duplicate	Multiple alerts map to the same root cause or case without providing a distinct remediation path.	Report separately and count toward duplicate rate.

Keep both the tool’s original finding ID and the canonical case ID. One canonical case may map to several alerts, and one alert may overlap a broader root cause. Resolve those mappings in review rather than letting finding count become the score.

Score the results

Use confirmed labels for the core detection metrics:

Metric	Formula or review question
Confirmed true positives	How many distinct labeled cases did the tool find?
Confirmed false positives	How many reviewed reports were unsupported or safe?
Missed known issues	How many labeled vulnerabilities had no equivalent report?
Precision	Confirmed TP / (confirmed TP + confirmed FP)
Recall	Confirmed TP / (confirmed TP + missed)
F1	2 × precision × recall / (precision + recall)
Duplicate rate	Duplicate alerts / all alerts reviewed
Exploitability-unknown rate	Unknown findings / all findings reviewed
Time to first useful result	Time from scan start to the first confirmed, actionable finding
Scan duration	Time from the accepted trigger to completed results, measured separately by scan mode
Mean time to triage	Total review time / findings reviewed
Developer workflow friction	Did the PR check, IDE result, ticket, and suppression flow fit the existing review process?
Fix acceptance and regression rate	For proposed fixes, how many developers accepted them and how many failed tests, required material edits, or were reverted?

Precision and recall only mean something when the denominator is fixed. The ground-truth curation article and hard-negatives article are useful references for separating real weaknesses from safe code that resembles them. When cases remain unknown, publish the count alongside the score and schedule their review. Do not silently remove them.

Compare scan modes, not just one scan

SAST is operated through several workflows. Run each candidate through the workflows you intend to buy.

Scan mode	What to test	What to record
Pull request	A small change that includes a known case and a normal developer change.	Time to result, changed-code coverage, check behavior, inline evidence, duplicate comments, and whether the check blocks or informs.
Full repository	The fixed representative commit with the full build context.	Coverage across services and shared packages, baseline import, source and generated-code handling, scan duration, and triage load.
Nightly	A scheduled full scan after the baseline exists.	Incremental behavior, reliability, newly discovered findings, rerun stability, and alert routing.
Release	The release candidate and production build configuration.	Release-gate timing, required evidence, policy exceptions, ownership, reporting, and whether findings map to a realistic release decision.

The release scan may be slower and broader than the PR scan. That is acceptable if the tool makes the distinction clear and the release gate is reliable. It is not acceptable to claim PR protection from a scan that only runs overnight, or to judge a release gate by a vendor’s small demo repository.

Set go or no-go thresholds

Set the thresholds before looking at vendor scores. They should reflect your risk tolerance and delivery budget, not a vendor’s published benchmark. The buyer-pilot scorecard includes fields for the values and owners you choose.

Use these gates:

Decision gate	Go when	No-go or follow-up when
Known release-blocking cases	The tool reports every case that the pilot team designated release blocking.	Any release-blocking case is missed without an approved compensating control.
Precision and duplicates	Confirmed precision meets the precommitted target and duplicate rate stays within the team’s triage budget.	Reviewers spend the planned budget on unsupported or repeated alerts before reaching actionable findings.
Unknown findings	Unknown cases have evidence, an owner, and a deadline.	A score depends on a large unreviewed unknown set.
PR workflow	Results arrive within the agreed PR feedback window and give developers enough evidence to act.	Results arrive after review is complete, create repeated comments, or cannot be governed with the team’s normal checks.
Full and release scans	The tool covers the chosen repositories and completes inside the agreed full-scan and release-gate windows.	It needs exclusions, build changes, or manual intervention that the operating team will not maintain.
Remediation	Proposed fixes pass the required checks, preserve behavior, and are accepted by developers at the target rate.	Fixes create regressions, require routine rewrites, or cannot be reviewed in the normal PR flow.
Operating model	Security and engineering can own configuration, exceptions, reporting, and cost at rollout scale.	The pilot depends on vendor-only tuning, undocumented configuration, or a one-off integration.

An inconclusive pilot is a valid result. Extend it only to resolve a specific missing label, integration question, or scan-mode failure. Do not extend it to search for a more favorable score.

Download the buyer-pilot scorecard

Download the SAST buyer-pilot scorecard and open it in a spreadsheet. It has rows for test cases, finding reviews, scan modes, formula definitions, and decision gates. Duplicate the rows for each tool and preserve the raw vendor export outside the scorecard.

For adjacent buying questions, compare SAST, SCA, and DAST, review how to reduce false positives in SAST, and use the Snyk alternative, Semgrep alternative, Checkmarx alternative, and GitHub Advanced Security alternative pages to frame vendor-specific questions. For a broader shortlist, see the best SAST tools comparison.

Run the same pilot on your repository before you buy another scanner.