TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A live AI management experiment tested five models on handling a simulated company’s worst week. The results reveal that management skills, not just chat quality, are crucial. The leaderboard shows which models excel at decision-making and trustworthiness under pressure.
In a groundbreaking live experiment, Firmulate has ranked AI models based on their ability to manage a simulated company’s worst week, emphasizing management skills over typical chat or coding benchmarks. The final July 2026 Crucible League results reveal that gpt-5.6-sol topped the leaderboard with a score of 95, outperforming competitors in decision-making, trust, and crisis management. This experiment underscores the importance of evaluating AI not just on response quality but on how effectively it manages real-world organizational challenges, a shift that could reshape AI evaluation standards.
The Firmulate live management benchmark involved five AI models operating within a simulated small software company’s environment, facing real-time crises, customer negotiations, and internal decision-making. The models were scored on their ability to diagnose issues, communicate effectively, escalate when necessary, and maintain trust—key elements of managerial competence. The top performer, gpt-5.6-sol, achieved a score of 95, while others like Kimi K3 and Sonnet 5 followed. Notably, all models identified crises accurately and rejected manipulation attempts, but only two successfully closed deals, highlighting a gap between diagnosis and execution. The experiment also enforced a strict trust standard: any breach caps the score, emphasizing integrity over superficial performance.
One critical finding was that models could sound informed yet fail to retrieve specific facts necessary for decisive action. For example, a model that read the company’s files but failed to reference a key document lost a €55,000 deal, illustrating that effective management requires more than surface-level responses. Additionally, models demonstrated resilience against social engineering tactics, refusing to disclose sensitive information or approve bypass requests, which reassures organizations concerned about security. However, the experiment revealed that more thorough models, despite their depth, did not necessarily perform better in closing deals or managing escalation channels, exposing a disconnect between effort and effective management.
Management Skills as a New AI Evaluation Standard
This experiment shifts the focus from traditional AI benchmarks—such as coding accuracy or conversational fluency—to management quality. As AI tools become integrated into organizational decision-making, their ability to diagnose, prioritize, escalate appropriately, and maintain trust will determine their real-world value. The findings suggest that evaluating AI models on management tasks could lead to more reliable, trustworthy implementations in business environments, especially where trust and accountability are paramount. This approach also highlights the importance of transparency and integrity, as breaches of trust directly impact the model’s score and, by extension, its suitability for critical roles.
For companies, this means moving beyond superficial performance metrics and adopting testing frameworks that simulate actual operational challenges. The experiment underscores that success in management tasks involves more than generating plausible responses; it requires consistent, accurate, and honest decision-making under pressure. As AI models improve, this management-centric evaluation could become a standard, ensuring that AI tools are truly prepared for organizational responsibilities rather than just answering well in isolated tests.
Products Worth Considering
Apple Watch SE 3 GPS 44mm Smartwatch
Samsung 27 Odyssey OLED G5 (G50SF) QHD QD-OLED Gaming Monitor
Apple iPad Air 11-inch (M2), 512GB, Wi-Fi, Space Gray
As an affiliate, we earn on qualifying purchases.
The Evolution of AI Benchmarks Toward Management Competence
Traditional AI benchmarks have focused on coding competitions, chat-based responses, and specific task completions. These measures, while useful, do not capture the complexities of real-world management, which involves triaging crises, making strategic decisions under pressure, and maintaining organizational trust. The Firmulate experiment is part of a broader shift toward evaluating AI models in operational scenarios that mirror actual business environments. This approach emerged from recognizing that models can perform well on isolated tasks but fall short when managing ongoing, interconnected challenges.
The experiment’s roots trace back to the limitations of existing benchmarks, which often reward superficial performance rather than true managerial competence. By embedding models in a simulated company facing authentic crises—such as customer churn, PR issues, and financial pressures—researchers aim to measure how well AI can handle the complexities of management. The July 2026 results represent a significant milestone, demonstrating that AI evaluation must include trustworthiness, decision accuracy, and escalation behavior—core elements of effective management that are often overlooked in traditional benchmarks.
“This experiment shows that management quality, not just chat performance, should be a new category of AI evaluation. It’s about how AI handles real-world organizational challenges.”
— Thorsten Meyer, lead researcher at Firmulate
Unclear Aspects of Long-Term Management Performance
It remains uncertain how these models will perform in extended, real-world organizational settings beyond the controlled simulation. The experiment measures immediate decision-making and trust, but the long-term consistency of AI management skills, especially under evolving crises or organizational changes, is still untested. Additionally, the impact of integrating these models into live systems—such as CRM or support platforms—may reveal further limitations or require new evaluation metrics. Researchers acknowledge that real-world management involves unpredictable human factors and complex negotiations that may not be fully captured in the current simulation.
Next Steps in AI Management Evaluation and Deployment
Following the July 2026 results, researchers plan to refine the management benchmark, incorporating longer-term scenarios and more complex decision trees. Organizations interested in deploying AI for management roles are encouraged to run their own wargames, simulating crises and evaluating models’ ability to read context, escalate appropriately, and maintain trust. Future developments may include integrating AI models into live operational systems with real-time feedback, enabling continuous assessment of management skills. Additionally, expanding the leaderboard to include more models and real-world organizational data will help establish industry standards for AI management performance.
Ultimately, the goal is to develop evaluation frameworks that go beyond isolated responses, focusing instead on AI’s capacity to handle consequences and sustain organizational integrity over time. The live experiment sets a precedent for how AI’s role in management can be rigorously tested before widespread adoption.
Key Questions
How does this new leaderboard differ from traditional AI benchmarks?
This leaderboard evaluates AI models based on their ability to manage a simulated company’s crises, make decisions, escalate appropriately, and maintain trust, rather than just generating accurate responses or code.
Why is management ability important for AI models?
Management skills determine whether AI can effectively handle real-world organizational challenges, prioritize tasks, and preserve trust—crucial for deploying AI in operational roles.
Can these models be trusted to manage real companies?
While the experiment shows promising resilience against manipulation and good decision-making in simulation, long-term reliability and performance in live environments still require further testing and validation.
What are the main limitations of this experiment?
The current test is limited to short-term crisis management within a simulated environment. Real-world management involves unpredictable human factors and evolving scenarios that are not yet fully captured.
What should companies consider before using AI for management tasks?
Organizations should evaluate whether AI models can read organizational context, escalate properly, and maintain trust, rather than relying solely on superficial response quality.
Source: ThorstenMeyerAI.com
