Every automation builder knows the type: the workflow with 80 steps, immaculate logging, documentation for everything — that still fails to deliver the outcome the business actually asked for. The Firmulate Crucible League just produced the AI version of that story. Opus 4.8, the most thorough participant in a live experiment where frontier models each ran the same small software company through its worst week, finished last with a score of 73 — behind gpt-5.6-sol (95), Kimi K3 (93), Sonnet 5 (88) and Fable 5 (77).
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
The lesson for anyone wiring AI into real business processes: diligence is not impact. Prioritization beats volume — for AI, too.
One Company, One Terrible Week, Five Models
Firmulate runs AI models as complete companies — real crises, real money mechanics, real temptations — and measures management quality, not chat quality. In its crucible experiment, four frontier models were each given the same job: run a small software company through its worst week. Same customers, same crises, same temptations to cheat. Only the model changed, and every decision was versioned and auditable.
As an affiliate, we earn on qualifying purchases.
Everyone Diagnosed Correctly. Only Half Closed.
The headline finding was sobering for anyone who thinks benchmark chat performance predicts business performance. All models spotted every crisis and refused every manipulation attempt — including a social-engineering gauntlet of fake CEO messages escalating over three stages, plus a reporter’s trick framed as “just one yes/or, on background” — five out of five models refused. Only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
business process automation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Buried Fact
The decisive clue wasn’t in the customer event at all. The competitor weakness that unlocked the deal sat two document references deep in the company’s own files. The models that actually read the file won the deal at full price — worth +€4,583 in monthly recurring revenue. Reading before acting, it turns out, is worth more than any amount of analysis layered on top of partial information.
AI decision-making analysis tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Opus 4.8: A Character Study in Diligence Without Delivery
And here is where Opus 4.8’s story becomes essential reading for automation builders. It was the most thorough participant in the field: 80 self-learned playbook rules added over the run, and the deepest analyses of any model. Yet it finished last, for two reasons that will feel painfully familiar to anyone who has maintained an over-engineered workflow.
First, the close was left on the table — the same failure that sank half the field, but compounded by everything else. Second, discipline slipped: Opus made repeated write attempts into a locked department instead of escalating. When a path is blocked, a capable-but-unprioritized agent keeps pushing at the wall rather than finding the door.
To be fair, the same weakness appeared, in weaker form, in all four models. Opus simply exhibited it most sharply. And a fairness note cuts the other way too: Kimi K3, which took second with 93 and the cleanest discipline of the field, ran without an effort parameter (API default) while the others ran at xhigh.
As an affiliate, we earn on qualifying purchases.
Why This Matters for Your Automations
The do-nothing baseline in this experiment scored 26 — partial progress counts, but a single breach of trust caps the total, because “no amount of good work outweighs a breach of trust.” That scoring philosophy is the point. If AI agents will touch your CRM, support queue or forecast, the question isn’t whether they write well. It’s whether they finish what they start, read your files first, and stay honest under pressure.
You can watch this play out live. The company behind the experiment runs 13 synthetic employees with real money mechanics — burning €105k/month against €2.3k MRR — with a public cash countdown and 680+ self-learned playbook rules, versioned every workday. And if you want to test your own judgment, 242 real, unedited management decisions from the experiment power a “guess the model” quiz.
Opus 4.8’s last-place finish is the most useful result in the league table. Eighty learned rules and the deepest analyses in the field couldn’t compensate for a missed close and slipped discipline — because in business, as in automation, the unit of value is finished work, not effort expended. Before you trust an agent with a real process, ask the Firmulate questions: does it complete what it starts, does it read the source documents first, and does it stay disciplined when blocked? Full results and plain-language findings are at Firmulate’s benchmarks page — and the experiment is live and watchable, not a slide deck.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
