🔍 Read the full analysis: FrontierCode: 5 Questions To Ask Before Submitting Your AI’s Pull Request on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added FrontierCode, a benchmark that evaluates whether AI-generated code changes meet maintainers’ review criteria, and tested 23 models. Claude Opus 5.5 led with a 52.2% score, but cleared all blocking checks in 57.7% of attempts, underscoring that human review remains necessary.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that scores whether a proposed repository change meets maintainers’ acceptance criteria rather than merely passing tests. Claude Opus 5.5 ranked first with a 52.2% score, but passed every blocking check in 57.7% of attempts, indicating that even the highest-ranked system still needs human review before a pull request is merged.
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, using the same open-source agent harness, Stirrup, for each model. Each attempt was capped at 250 turns, with up to three attempts per task. The review stage used GPT-6 Luna (medium) as the judge. The results cover the benchmark’s 100-task Main set, drawn from a larger 150-task collection.
Claude Opus 5.5 (max with fallback) scored 52.2%, ahead of GPT-6.1 Sol (max) at 49.1% and Claude Fable 5.1 (max with fallback) at 49.0%. The pass-rate measure is separate: Opus cleared all blocking checks in 57.7% of attempts. Xiaomi’s MiMo-V2.6-Pro led the listed open-weight models at 45.4%, followed by DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%.
FrontierCode checks patches for hidden-test correctness, regressions and cleanliness, whether an agent’s tests distinguish the original bug, behavior on adaptive tests, scope and diff size, and an LLM-based review against maintainers’ criteria. Failing any blocking check gives an attempt a score of zero; passing the blocking checks produces a score from 70% to 100%, adjusted for quality checks. The source reports that nine of the 100 tasks were not solved by any model on any attempt.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results address a practical question for engineering teams using coding agents: whether a generated patch is fit to submit, not just whether it can satisfy a test suite. FrontierCode’s scoring includes scope limits, useful tests and reviewer criteria, reflecting common reasons maintainers may reject a patch even when its core behavior appears correct.
The reported pass rate also puts a limit on claims that AI can handle pull requests autonomously. The leading model failed to clear at least one blocking check in a substantial share of attempts. That does not establish how a model will perform on every team’s codebase, but it supports treating generated changes as work for a developer to inspect rather than automatically merge.
Cost varies sharply among the models, according to the source. GPT-6 Luna (max) scored 41.9% at $0.09 per task; MiMo-V2.6-Pro scored 45.4% at $0.18; GPT-6.1 Sol scored 49.1% at $0.81; and Opus scored 52.2% at $10.15. On these results, moving from Sol to Opus adds 3.1 percentage points at roughly 12.5 times the reported task cost. Those figures are benchmark-specific and do not establish the cost or quality teams will see in their own workflows.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Repository Changes
FrontierCode was developed by Cognition with open-source maintainers. It was announced in June 2026, with version 1.1 following in July and adding rules against searching online for task solutions. According to the source, more than 20 maintainers wrote tasks based on 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task. The tasks are private, reducing the chance that models encountered them during training.
The benchmark’s approach differs from evaluations that mainly count passing tests. Its six assessment areas include correctness, regression checks, test quality, adaptive behavior, change scope and a language-model review. Cognition’s trajectory scanner can also assign a zero to an attempt flagged for looking up a solution. The source attributes the benchmark’s review-oriented framing to Tomer Nosrati, a Celery maintainer: “Where others grade like a CI, FrontierCode grades like a tech lead.”
Artificial Analysis’ results also show that rankings on a general intelligence index do not necessarily predict results here. The source says Claude Sonnet 5.5 ranked second among the compared models on that index but fifth on FrontierCode, while Grok 4.7 fell to 18th on FrontierCode. These comparisons suggest different evaluations measure different capabilities; they do not, by themselves, explain why each model’s ranking changed.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results do not establish how models perform across all repositories or in day-to-day team workflows. Artificial Analysis tested a 100-task Main set from FrontierCode’s 150 tasks, under a particular agent harness, turn limit and review setup. The source does not provide enough detail here to determine how results might change with different tools, prompts, model settings or human review procedures.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published leaderboard is not a complete comparison of the models named in the source. The source also does not specify in this account how uncertainty across attempts was calculated, or how closely the LLM judge’s decisions match human maintainers’ judgments.
As an affiliate, we earn on qualifying purchases.
Models Still Awaiting Results
The next point to watch is whether the models missing from the October 13 leaderboard are evaluated on FrontierCode, particularly the systems named by the source as having no result. Additional results could change the ranking and help readers compare a wider range of systems under the same benchmark setup.
For now, the published data supports using FrontierCode as one measure of repository-level coding performance, not as a guarantee that a model’s patch is ready to merge. Developers and maintainers will still need to inspect changes, test them in their own environments and decide whether they meet local standards. No later benchmark update or release date is specified in the source.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether an AI-generated repository change meets a set of blocking checks and code-quality criteria, including correctness, regression checks, test quality, change scope and a review against maintainers’ standards.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in the reported results with a 52.2% FrontierCode score. It passed every blocking check in 57.7% of attempts.
Does a passing FrontierCode result mean a pull request should be merged?
No. The benchmark reports performance on its tasks and criteria. It does not replace review by developers familiar with the repository or testing in a team’s own environment.
How many tasks and models were included?
Artificial Analysis tested 23 models on the 100-task Main set. FrontierCode has 150 tasks in total, according to the source.
Were all major models included?
No. As of October 13, 2026, the source said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
