Don’t just trust “done”: Microsoft trains agents on database outcomes
3-line summary
- Microsoft and Hugging Face presented ThinkingBox training with rewards based on backend outcomes.
- Tasks passing every trial increased, while tasks solved at least once declined.
- Training code is public; the reported results concern synthetic business tasks.
Microsoft and Hugging Face introduced ThinkingBox-Training and released its code on October 22. The approach rewards an agent for what remains in the system after a task, rather than relying on its claim of completion.
The official repository describes fresh backends, tools and simulated users for each attempt. Checks of final database state and side effects return success or failure rewards. Most checks are executable; some narrow requirements also use a separate AI judge.
After 50 training steps for Qwen3.8-27B, the team evaluated 507 separate tasks twenty times each. Average success rose from 51.70% to 60.78%; tasks passing all twenty attempts rose from 38 to 107. Yet tasks passing at least once fell from 453 to 446. More repeatable success and broader coverage are different outcomes.
Evaluation tasks were excluded from training, but both sets share a framework and similar task styles. The public sample list checks integration; it is not the full private training pool behind the results. Customers and tasks are synthetic. These figures do not guarantee performance in real businesses, and AI Ping has not reproduced the training run.
Why it matters
AI Ping sees a reason to separate convincing completion reports from completed work. Before deployment, teams should examine repeated outcomes and unexpected changes alongside individual successes.
Who it's for
- Developers
- Define the expected final state and records that must remain unchanged, then compare outcomes across repeated agent trials.
