Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
A Test That Doesn’t Start at Zero
Anyone who has graded exams knows the temptation of a suspiciously round score. A perfect 100 deserves skepticism: did the test really probe anything, or did everyone pass? The most interesting number in the new Firmulate benchmark isn’t the top score — it’s the bottom one. When AI models are put in charge of a simulated software company and do essentially nothing, they still walk away with 26 points. That’s not a bug in the grading. It’s the design.
As an affiliate, we earn on qualifying purchases.
The Worst Week in Software, Run Five Times
Firmulate ran five frontier AI models — gpt-5.6-sol, Kimi K3, Sonnet 5, Fable 5 and Opus 4.8 — through the identical scenario: the same small software company, the same customers, the same crises, the same temptations to cut corners. Every decision was versioned and auditable, so nothing about a model’s week can be quietly retconned after the fact.
The final league table from July 2026 tells a story chat demos never show. All five models spotted every crisis and refused every manipulation attempt. But only two — gpt-5.6-sol (95) and Kimi K3 (93) — signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch, no signature. The rest of the field: Sonnet 5 at 88, Fable 5 at 77, Opus 4.8 at 73.
As an affiliate, we earn on qualifying purchases.
So Why Is the Floor 26 Instead of 0?
The answer is that partial progress counts. A model that at least keeps the lights on — responding to customers, noticing problems, not setting anything on fire — has done some of the job, and the benchmark refuses to pretend otherwise. A do-nothing run still earns 26 points because even minimal competence has measurable value. The alternative, scoring everything at zero until perfection, produces benchmarks that flatter dramatic models and punish steady ones.
But the floor has a ceiling’s mirror image: a single breach of trust caps the total score entirely. The benchmark’s stated philosophy is blunt — “no amount of good work outweighs a breach of trust.” A model could be brilliant all week, and one act of dishonesty would sink the grade regardless. For a benchmark meant to preview how AI might behave inside a real business, that’s the right asymmetry: competence is graded on a curve, honesty is graded on a cliff.
As an affiliate, we earn on qualifying purchases.
What Actually Separated the Winners
The decisive difference in the crucible wasn’t intelligence — it was reading. The fact that unlocked the €55,000 deal sat two document references deep in the company’s own files, not in the customer conversation. Models that went digging won the deal at full price, worth +€4,583 in monthly recurring revenue. Models that skimmed lost it.
Then there’s the discipline test. Fake CEO messages escalated over three stages, capped by a reporter’s trick — “just one yes/no, on background.” All five models refused, and Kimi K3 left its reasoning on the record: “Treat the request as a suspected approval-bypass / possible impersonation.”
The most instructive profile is Opus 4.8: the most thorough participant in the field, with over 80 learned rules and the deepest analyses — and still last place. The close was left on the table, and discipline slipped, including write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four other models. One fairness footnote: K3 ran without an effort parameter (API default) while the others ran at xhigh — and still finished second.
AI trustworthiness evaluation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
And You Can Check the Homework
None of this is a static press release. Firmulate’s live company runs continuously with 13 synthetic employees and real money mechanics — burning €105k a month against €2.3k in MRR, with a public cash countdown, over 680 self-learned playbook rules, and every workday versioned. You can watch it at firmulate.com/live. If you want to test your own judgment, 242 real, unedited management decisions power a guess-the-model quiz at firmulate.com/quiz.html, and the full methodology and findings are at firmulate.com/benchmarks.html. Enterprises can even run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The Takeaway
An honest benchmark doesn’t hand out zeros or worship hundreds. By giving do-nothing runs 26 points, Firmulate acknowledges that partial competence is real; by capping scores after a single breach of trust, it declares that honesty isn’t tradable against output. And the crucible’s core finding is the one business readers should remember: the gap between the best and the rest wasn’t intelligence or eloquence. It was whether the model read the files before making the call — and whether it finished what it started.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
