🔍 Read the full analysis: 3 Ways CWE-bench V2 Improves AI Benchmarking on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Collinear AI has released CWE-bench v2, a benchmark of 100 security-fix tasks from open-source repositories, along with its grader and evaluation harness. The release reports a top pass@1 score of 67.8% across 11 evaluated models, but says every model failed some tasks on every attempt.
Collinear AI has released CWE-bench v2, a public evaluation set of 100 security-fix tasks drawn from real open-source repositories, together with a grader and evaluation harness. The release gives developers a shared way to test whether coding agents can locate and repair software weaknesses while preserving existing behavior; its reported results also show that even the leading evaluated model did not pass every task, a challenge also explored by real-world coding benchmarks.
The tasks cover 91 CWE weakness categories. For each one, an agent receives a checkout of a repository and a short instruction to find and fix a security problem. The authors say tasks are packaged as self-contained Harbor environments, with a registry file for running them through Hugging Face. The dataset is gated, so users must accept its access conditions before running it.
Collinear AI reports results for 11 models. Gemini 4 Argon led at 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5 at 67.3% and 78.0%, respectively. Pass@1 measures success on a single attempt; pass@4 reports results across four attempts, metrics that also feature in coding-model benchmarks. The release says at least one model solved 95 of the 100 tasks, but that does not mean one model solved all 95.
The announcement also compares cost and performance. DeepSeek V4.1 Flash (1017) scored 57.0% pass@1 at a reported $0.04 per rollout; Claude Opus 5.5 scored about ten percentage points higher at $1.83, while Gemini 4 Argon scored slightly higher at $3.92. These are figures reported by the release, not fixed costs: pricing and evaluation conditions can affect the comparison.
What Security-Fix Scores Can Show
Security work requires more than generating code that looks plausible. An agent must interpret a task, find the relevant code, address the weakness and avoid breaking behavior that should remain unchanged. A benchmark spanning 91 weakness categories can test more kinds of repair than a set concentrated on one vulnerability pattern, while giving teams a common set of tasks for comparing systems.
The results also place a limit on claims about readiness. The leading reported pass@1 score is below 70%, and the release says the leading model fails 19% of the set on every attempt. That is evidence of unresolved failures within this test, not proof that the same error rate applies in production. A benchmark score can inform model selection and research, but it cannot establish that an agent is safe to handle security work across arbitrary repositories, threat models or deployment conditions.
For readers building or buying coding agents, public tasks and a repeatable grader offer a way to check claims against shared tests rather than relying only on vendor demonstrations. The cost figures may also help teams consider trade-offs, provided they treat the estimates as specific to the release’s setup and not as universal prices.
Products Worth Considering
Anker SOLIX C300 Portable Power Station, 288Wh
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black
As an affiliate, we earn on qualifying purchases.
How Version Two Tests Repairs
Version 2 retains the prior task format and expands the public set to 100 tasks. According to Collinear AI, each task was reviewed by hand for realistic instructions, checkable behavior and verifiers that assess whether a fix works rather than whether it follows one particular code structure. The authors say they checked that starter code fails, a reference fix passes, existing functionality remains intact, and verifiers reject memorized solutions such as known upstream patches.
The release says LLM judge panels helped shape programmatic checks for task difficulty and fairness. Its authors describe scoring as fully programmatic and intended to support repeatable evaluations. Results are tracked on two Hugging Face channels: a dataset-page leaderboard for open-weight models and a leaderboard Space for broader comparisons, including API-only models. Contributors can submit results through pull requests.
Collinear AI also maintains a separate held-out set for tracking performance on tasks not seen by evaluated models. The company says interested teams can contact it to evaluate models on that set, which is not publicly accessible. That arrangement may help measure performance beyond the visible tasks, but outside readers cannot directly inspect or reproduce held-out evaluations.
“A defender has to anticipate the ways an attacker could get in, and get it right every single time.”
— Collinear AI, in its CWE-bench v2 announcement on Hugging Face
Limits of the Published Comparisons
The supplied announcement does not give enough methodological detail to independently assess every comparison, including the exact model configurations and the conditions used for each cost estimate. Public leaderboard results may also change as more submissions arrive. The reported scores should therefore be read as results from the release’s evaluation, not as a settled ranking across all possible setups.
It is also unclear how well performance on these 100 curated tasks predicts outcomes in production security work. The authors describe task review and verifier checks, but a benchmark cannot cover the full variety of software and deployment conditions. The held-out set is not public, limiting independent inspection and reproduction of results on those tasks. The release does not establish whether benchmark performance tracks success in live security reviews.
More Runs and Community Review
Researchers and developers can run models on the public tasks with Harbor after accepting the dataset conditions and setting up Hugging Face access. Collinear AI invites feedback on the benchmark, and contributors can submit leaderboard results through pull requests. Further runs may add comparisons or expose issues in task design and grading.
Teams interested in the private held-out set must contact Collinear AI, according to the release. A clearer picture of the benchmark’s value will depend on additional evaluations and independent reproduction of public results. Whether success on either set predicts performance in real security work remains an open question.
Key Questions
What is CWE-bench v2?
CWE-bench v2 is a public benchmark from Collinear AI comprising 100 security-fix tasks based on real open-source repositories. It includes a grader and evaluation harness intended to let users run and compare coding agents.
Which model had the highest reported score?
The release reports that Gemini 4 Argon led at 67.8% pass@1 and 81.0% pass@4 among 11 evaluated models. These figures are the announcement’s results and may depend on model configuration and evaluation conditions.
Does the top score mean an agent can handle security work reliably?
No. The top reported pass@1 score was 67.8%, and the authors say the leading model failed 19% of the tasks on every attempt. Performance on this curated set does not establish reliability across production software or all security scenarios.
Can anyone run the benchmark?
The public tasks are available through Hugging Face and are packaged for Harbor, but the dataset is gated. Users must accept its access conditions and set up the required access before running evaluations.
Are the held-out tasks publicly available?
No. Collinear AI says the held-out set is separate and not public; teams interested in evaluating models on it can contact the company. Its restricted access means outside researchers cannot directly reproduce evaluations on those tasks.
Primary source: Hugging Face · via ThorstenMeyerAI.com
