TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
MaxxBench modified the instructions or starting code in 89 tasks from the DeepSWE coding benchmark and tested 15 model configurations. Its report found wide score differences on the revised tasks among models whose original DeepSWE results were often close, though the results come from MaxxBench’s evaluation and do not establish how models perform across all real-world coding work.
MaxxBench has reported that leading coding models with similar scores on DeepSWE separated sharply when tested on 89 revised tasks, a result that raises questions about whether familiar benchmark performance reflects the ability to follow changed requirements. In the report, Claude Opus 5.5 scored 76.4% on MaxxBench, while Gemini 3.8 Flash scored 53.4%, despite both scoring about 74% on DeepSWE.
MaxxBench says it altered the instructions or starting code for 89 medium-duration feature-implementation tasks from DeepSWE, then revised the reference solutions and tests to match. The changes were intended to resemble ordinary requests made after an initial implementation, while preserving checks for behavior that still applied. The report says the reference patches grew by a median of 17 lines, or 3.5%.
Examples include changing browser report filenames to lowercase while retaining the reporting feature, and requiring FastAPI route retirement timestamps to use UTC with a trailing “Z” rather than preserving the supplied time-zone offset. MaxxBench presents the changes as a test of whether agents can adapt to revised specifications rather than rely on expectations associated with the original tasks.
Across the 15 model configurations in the report, DeepSWE scores ranged from 53.3% to 74.2%; MaxxBench scores ranged from 28.1% to 76.4%. The reported spread between the highest and lowest results widened from 20.9 percentage points on DeepSWE to 48.3 points on MaxxBench. Among models near 74% on DeepSWE, Claude Opus 5.5 rose from 74.2% to 76.4%, while DeepSeek V4.1 Flash fell from 74.2% to 55.6% and Gemini 3.8 Flash from 73.8% to 53.4%.
Revised Tasks Separate Similar Scores
The report’s central implication is that a score on a familiar coding benchmark may not capture how consistently a model responds when requirements change. If a model has learned patterns associated with a task or its expected tests, it might perform well without being equally reliable at implementing a new constraint. MaxxBench argues that its revisions expose differences hidden by the original results.
That matters to teams comparing coding agents for software work: choosing solely by a headline benchmark score could obscure differences in how models handle follow-up requests. The results also offer a reason to evaluate agents on variations of existing tasks. However, this is evidence from one benchmark study, not proof that every DeepSWE score is misleading or that the same ranking applies to production codebases.
Products Worth Considering
Bose SoundLink Plus Portable Bluetooth Speaker, Black
DJI Mini 5 Pro Fly More Combo with DJI RC 2
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor, 240Hz
As an affiliate, we earn on qualifying purchases.
How MaxxBench Changed DeepSWE
MaxxBench says benchmark tasks can influence model training, checkpoint selection, and prompt or agent development. It calls gains tied to a benchmark’s specific tasks and conventions “benchmaxxing.” That is the report’s framing, not a finding that any named model was trained directly on the tasks or deliberately optimized to exploit their tests.
The project chose DeepSWE’s medium-duration feature tasks because it wanted to examine how models handled revised versions of work they had already scored well on, rather than immediately moving to longer engineering projects. Its mutations changed requirements or, in some cases, the surrounding code and available interfaces. The source material does not provide independent validation of the evaluation or enough detail here to establish how representative the 89 tasks are of software work outside the benchmark.
“We changed requirements in a coding benchmark and found models talking themselves out of following them.”
— MaxxBench report
Limits of the Reported Results
The report describes observed runs and score differences, but the supplied material does not state how many attempts were made per task, whether results were averaged across repeated runs, or how statistical uncertainty was calculated. It also does not provide independent replication or a comparison with performance on live software projects. Those details would help readers judge how stable and generalizable the rankings are.
MaxxBench’s explanation that agents may be trying to satisfy an imagined grader is an interpretation of model behavior, not a demonstrated cause of every score decline. The results establish differences on the modified tasks as reported; they do not by themselves show why each model missed a requirement or how often the same behavior occurs outside this evaluation.
Replication and Broader Testing
The next useful step is to compare the results with independent evaluations using the same task mutations, disclosed run procedures, and repeated trials. More testing across different repositories and coding tasks could show whether the gaps persist beyond this set of DeepSWE-derived examples. The source material does not announce a follow-up release or a schedule for additional results.
Until further evidence is available, readers can treat MaxxBench as a reported stress test of adaptation, not a replacement measure for all coding-agent performance. The report’s practical question remains open: whether the differences it found predict how reliably models handle changing specifications in everyday development.
Key Questions
What is MaxxBench?
MaxxBench is an evaluation that modifies tasks from DeepSWE to test how coding models handle changed instructions or starting code. The report covers 89 tasks and 15 model configurations.
What were the main results?
MaxxBench reported scores from 28.1% to 76.4% on its revised tasks, compared with 53.3% to 74.2% on DeepSWE for the configurations listed. Some models with similar original scores diverged substantially after the revisions.
Does this prove DeepSWE scores are unreliable?
No. The report indicates that performance on these revised tasks differed from performance on the original benchmark. It does not establish that DeepSWE scores are generally unreliable or that MaxxBench predicts results across all coding work.
Why did MaxxBench change the tasks?
Its authors wanted to test whether models could preserve a feature while adapting to a follow-up requirement, such as changing output filenames to lowercase. They say the modifications were small relative to the original tasks.
What remains uncertain about the findings?
The supplied report material does not specify repeated-run procedures or statistical uncertainty, and it does not show independent replication or performance on live projects. Those limits make the results’ broader reach uncertain.
Source: maxxbench.unlap.ai
