🔍 Read the full analysis: How ThinkingBox Connects Database Grading And Agent Post-Training on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub, using executable checks of final database states as rewards for agent post-training. In one reported 50-step GRPO run, Qwen3.8-27B’s average success rate rose from 51.70% to 60.78%, and more benchmark tasks passed all 20 attempts; seven fewer tasks were solved at least once.
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub, turning executable checks of an agent’s final database state into rewards for post-training. In one reported 50-step GRPO run, Qwen3.8-27B’s average success rate increased on a 507-task benchmark, although the model solved seven fewer tasks at least once after training.
Across 507 benchmark tasks, each evaluated 20 times, pass@1 rose from 5,242 of 10,140 attempts, or 51.70%, before training to 6,163 attempts, or 60.78%, afterward. The teams also reported that the number of tasks passed in every one of the 20 attempts increased from 38 to 107. They label this measure Observed 20/20.
The results were not an across-the-board expansion in task coverage. Pass@20—the share of tasks solved at least once in 20 attempts—fell from 453 tasks, or 89.35%, to 446, or 87.97%. That is a decline of seven tasks. The figures are the teams’ reported results from this evaluation; they do not establish how the model would perform in production.
The training used a separate pool of ThinkingBox-format practice tasks, rather than the 507-task evaluation benchmark. The authors say they selected training tasks using an estimate of how difficult each was for the base model, then kept the pool fixed during training. At the end of a task, an executable check inspected whether expected records were present in the backend and supplied the reward signal.
Why Final Database State Matters
For agents that can change business records, a convincing conversation or a plausible series of tool calls is not proof that the requested action was completed correctly. A check of the final database state can score the actual outcome, including cases where an agent claims success but leaves a required record wrong.
The reported pattern may matter to teams weighing reliability against task coverage. The average success rate rose, and more tasks passed all 20 trials, while the number solved at least once dipped. That suggests the measured gains were associated with more repeatable outcomes on a larger set of tasks, not with broader coverage across the full benchmark. It is a reported benchmark result, not evidence that the same tradeoff will hold in other models or workflows.
Because the checks target the required result rather than one prescribed sequence of tool calls, the approach could allow different valid ways to complete a task. Whether that flexibility improves dependable behavior outside the reported evaluation remains an open question.
As an affiliate, we earn on qualifying purchases.
From Grading to Post-Training
ThinkingBox was developed as a way to evaluate agents by examining the database records they leave behind. Its benchmark tasks include a backend, MCP tools and a simulated user with private context. Executable checks compare the final database state with the expected outcome once the task ends.
The joint post describes extending those checks from evaluation into training. For most tasks, the authors say, the executable outcome can return a direct one-or-zero reward from the final state. Their central idea is to use that outcome as a training signal, rather than rewarding an agent for following a particular tool-call path.
The authors describe the evaluation benchmark as out of distribution because its domains, databases and most MCP tools were not encountered during training. They also give a hotel-booking example: a traveler requested a room downgrade that required manager approval. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. Afterward, it preserved booking and payment state in all 20 attempts; the authors say expected records matched in 19. This is one reported task, not a general rate for hotel bookings.
“For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.”
— Microsoft and Hugging Face, in their joint post
As an affiliate, we earn on qualifying purchases.
Open Questions on Reliability
The supplied account reports one 50-step training run and does not give enough information to judge how sensitive the results are to random seeds, the choice of training tasks or the number of steps. It also does not provide a full comparison with other reward designs or training methods, so the results do not isolate how much of the change came specifically from database-state rewards.
The authors report that the benchmark was separate from training, but no live production results are included. Performance in operational workflows, generalization beyond this benchmark and the cost of running the training process remain unknown. The reported pass@20 decline is clear, but the account does not explain why fewer tasks were solved at least once.
reinforcement learning database rewards
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Reproduction and Broader Testing
With the stack available on GitHub, researchers and developers can inspect the implementation and try their own runs. The next useful evidence would include independent reproductions and tests across additional models, training pools, run lengths and random seeds.
For deployment decisions, teams would also need evaluations that reflect their intended workflows, including checks for correct final records and unauthorized or incomplete changes. Until those results are available, the reported benchmark improvements should be read as an initial evaluation of the method, not a measure of production reliability.
machine learning evaluation benchmark
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is ThinkingBox-Training?
It is a training stack from Microsoft and Hugging Face that uses executable checks of an agent’s final database state as a reward signal during post-training.
What changed in the reported Qwen3.8-27B results?
After a reported 50-step GRPO run, pass@1 on the 507-task benchmark rose from 51.70% to 60.78%. The number of tasks passed in all 20 attempts increased from 38 to 107.
Did the model solve more benchmark tasks overall?
Not by the pass@20 measure. The number of tasks solved at least once fell from 453 to 446, a decrease of seven, even as average success and the all-20-attempt count rose.
Do the results show how the system performs in production?
No. The reported figures come from a benchmark evaluation. The supplied account does not report deployment results or establish reliability in live business workflows.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
