📊 Full opportunity report: The Ultimate AI Leaderboard Starts Once The Demo Concludes on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
The final phase of the Firmulate experiment has concluded, revealing a management-focused AI leaderboard. The models were tested on real-world management tasks under crisis conditions, emphasizing trust and decision quality. The results challenge traditional AI benchmarks by prioritizing management capabilities.
The final results of the Firmulate management AI experiment have been announced, revealing how different models perform when tasked with running a simulated company during its most challenging week. The original analysis can be explored in this detailed report. The experiment tested five frontier models on their ability to diagnose crises, make decisions, and maintain trust under pressure. This development underscores a shift in AI evaluation from simple response quality to management and trustworthiness, with significant implications for enterprise deployment.
The Firmulate experiment placed five AI models—gpt-5.6-sol, Kimi K3, Sonnet 5, Fable 5, and Opus 4.8—against each other in a simulated management crisis. The models were scored on their ability to identify issues, communicate effectively, and execute decisions, with the top model, gpt-5.6-sol, earning a score of 95. The evaluation was rigorous, including a trust standard: a single breach of trust capped the overall score. Despite all models detecting crises and resisting manipulation attempts, only two successfully secured a key €55,000 deal, highlighting a gap between diagnosis and action. This emphasizes the importance of comprehensive AI evaluation, as discussed in the original analysis.
For example, models that read the company’s files more thoroughly were more successful in closing deals, but some still failed to present the critical fact needed for the sale, illustrating that effective management involves more than just technical competence. The experiment also tested responses to social engineering, with all models refusing to disclose sensitive information, demonstrating strong safety posture. However, even the most comprehensive model struggled with completing managerial tasks that required disciplined escalation and follow-through, exposing limitations in current AI management capabilities.
The Ultimate AI Leaderboard Starts Once The Demo Concludes
Five frontier models were tested on real-world management tasks during a simulated company’s worst week — diagnosing crises, making decisions, and maintaining trust under pressure. The results challenge traditional AI benchmarks by making management capability the core measure of evaluation.
Diagnosis Was Universal — Execution Was Not
Every model detected the crisis and resisted manipulation, but only two secured the critical €55,000 deal. A single breach of trust capped the overall score under the experiment’s strict trust standard.
| Rank | Model | Score | Performance | Deal Closed | Trust Standard |
|---|---|---|---|---|---|
| 01 | gpt-5.6-sol | 95 | ✓ Yes | ✓ Passed | |
| 02 | Sonnet 5 | — | ✓ Yes | ✓ Passed | |
| 03 | Opus 4.8 | — | ✗ No | ✓ Passed | |
| 04 | Kimi K3 | — | ✗ No | ✓ Passed | |
| 05 | Fable 5 | — | ✗ No | ✓ Passed |
Models that read the company’s files more thoroughly were more successful in closing deals — yet some still failed to present the critical fact needed for the sale.
More Than a Good Answer
Effective management means completing the full cycle — deciding, communicating, escalating — without compromising trust. Even the most comprehensive model struggled with disciplined escalation and follow-through.
Diagnose
Identify the crisis and read organizational context thoroughly — files, stakes, and history.
Decide
Weigh consequences, prioritize organizational goals, and commit under time pressure.
Communicate
Present the critical facts — the difference between closing and losing the €55,000 deal.
Escalate
Follow proper channels with discipline and complete tasks through to follow-through.
Why Management Skills Are the Next Benchmark
Traditional benchmarks measure isolated tasks like coding or conversational accuracy. Firmulate simulates a company’s worst week with real money mechanics and versioned decision logs — a far more authentic test of enterprise readiness.
Trust as a Hard Constraint
A single breach of trust capped the overall score. All five models refused to disclose sensitive information during social engineering tests — a strong safety posture vital for enterprise deployment.
Real Money, Real Stakes
The experiment’s design uses real money mechanics and versioned decision logs, forcing models to resist shortcuts and handle consequences rather than optimize for polished responses.
Operational Competence Over Time
Models must operate reliably over sustained periods — handling messy, unpredictable management realities rather than narrow, well-defined benchmark tasks.
What The Researchers Said
“Management quality, not just chat quality, deserves its own category of AI evaluation. Managing a crisis requires discipline, trust, and decision-making under pressure.”
— Thorsten Meyer, Lead Researcher, Firmulate“Our model refused to disclose sensitive information during social engineering tests, demonstrating strong safety posture, which is vital for enterprise deployment.”
— Kimi K3 Model Developer“The real challenge isn’t just diagnosing crises but completing the management cycle — deciding, communicating, escalating — without compromising trust or effectiveness.”
— Firmulate Project LeadRemaining Questions & Next Steps
What was the main purpose?
To evaluate AI models’ ability to manage a company’s crises, make decisions, and maintain trust during a simulated worst week — shifting focus from response quality to management skills.
Which model performed best?
gpt-5.6-sol finished first with a score of 95, demonstrating strong crisis management and decision-making capabilities.
What are the limitations?
It remains uncertain whether models can handle diverse, unpredictable real-world scenarios, and how they perform over longer periods or in different organizational cultures.
What comes next?
More sophisticated benchmarks with long-term decision tracking, trust audits, and multi-stakeholder interactions — plus supervised integration into real organizational workflows, with management established as a formal AI evaluation category.
Why Management Skills Are the Next Benchmark in AI
This experiment highlights that management quality—not just chat or coding proficiency—should become a core measure for AI evaluation in enterprise contexts. The ability to diagnose, decide, communicate, and trustworthiness under pressure directly impacts real-world outcomes, such as closing deals or maintaining trust. The results suggest that AI models must be assessed on their capacity to handle consequences, prioritize organizational goals, and operate reliably over time, which are critical for deploying AI at scale in complex business environments.
AI Essentials for Managers: Practical Ways to Boost Productivity, Make Better Decisions, and Lead High-Performing Teams (Self-Learning Management Series)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Evolution from Response Benchmarks to Management Testing
Traditional AI benchmarks focus on isolated tasks like coding or conversational accuracy, which do not reflect the realities of managing organizations or crises. The Firmulate experiment is a pioneering effort to simulate a company’s worst week, forcing models to navigate multiple crises, make strategic decisions, and maintain trust. This approach builds on prior developments in AI safety and reliability, emphasizing that effective management involves reading organizational context, resisting shortcuts, and completing tasks through proper channels. The experiment’s design, involving real money mechanics and versioned decision logs, provides a more authentic test of AI’s readiness for enterprise management.
Previous efforts have shown that models can perform well on narrow benchmarks, but the challenge remains whether they can handle the messy, unpredictable nature of real-world management. This experiment bridges that gap by creating a controlled but realistic environment where AI must demonstrate operational competence and trustworthiness over sustained periods.
“Management quality, not just chat quality, deserves its own category of AI evaluation. Our experiment shows that managing a crisis requires discipline, trust, and decision-making under pressure.”
— Thorsten Meyer, lead researcher at Firmulate
AI crisis management simulation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Remaining Questions About AI Management Performance
While the experiment provides valuable insights, several questions remain. It is not yet clear whether these models can consistently perform in diverse, real-world organizational contexts beyond the simulated environment. The long-term reliability of AI management, especially under unpredictable or novel crises, remains to be tested. Additionally, the impact of different organizational structures and cultures on AI decision-making is still unknown. The experiment also does not fully address how models handle ethical dilemmas or complex human relationships in management scenarios.
AI trustworthiness assessment tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Future Steps for AI Management Benchmarks and Deployment
Following these results, researchers and enterprises are likely to develop more sophisticated management benchmarks that include long-term decision tracking, trust audits, and multi-stakeholder interactions. Companies considering AI assistants for management roles should begin testing models in simulated environments tailored to their specific operations, focusing on trust, escalation protocols, and decision accountability. The next phase involves integrating these models into real organizational workflows under close supervision, with ongoing evaluation of their management quality and safety. The ultimate goal is to establish management as a formal category in AI assessment, ensuring models can reliably handle the complexities of running a business.
AI enterprise management solutions
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main purpose of the Firmulate experiment?
The experiment aims to evaluate AI models’ ability to manage a company’s crises, make decisions, and maintain trust during a simulated worst week, shifting focus from response quality to management skills.
Why is management performance important in AI evaluation?
Management performance reflects an AI’s capacity to handle real-world organizational tasks, including decision-making, trustworthiness, and accountability, which are critical for enterprise deployment.
Which model performed the best in the final leaderboard?
gpt-5.6-sol finished first with a score of 95, demonstrating strong crisis management and decision-making capabilities.
What are the limitations of the current experiment?
It remains uncertain whether these models can handle diverse, unpredictable real-world scenarios, and how they will perform over longer periods or in different organizational cultures.
What are the next steps for AI in management roles?
Future efforts will focus on developing more comprehensive benchmarks, testing models in enterprise environments, and establishing management as a formal AI evaluation category.
Source: ThorstenMeyerAI.com
