Skip to content

Research

MLPerf adds its first benchmark for AI agent post-training

Edited and reviewed by AI Ping Editorial Team

3-line summary

  1. MLPerf Training 6.1, released November 16, adds the suite’s first test of reinforcement learning for AI agents.
  2. The test starts from a shared Qwen checkpoint and measures how long a system takes to reach a target score on held-out coding tasks.
  3. Google’s 640-chip Ironwood submission reached the target in 18.02 minutes. NVIDIA submitted Rubin and GB300 runs with 64 GPUs apiece.
PracticalDevelopers
A fictional metal-covered AI accelerator beside a stopwatch
AI-generated conceptual image: A stopwatch beside a fictional accelerator represents the time to a quality target. · AI Ping

MLCommons has added a way to compare how long systems take to train a coding agent to a fixed quality target using reinforcement learning. The nonprofit released MLPerf Training 6.1 on November 16 with the suite’s first test of agentic reinforcement learning. Systems start from the same saved model state and are timed against a common quality target. NVIDIA’s Rubin GPUs and Google’s Ironwood TPUs also made their first appearances in MLPerf Training results.

Pretraining builds a model’s general capabilities from a large body of data. The new test covers a later stage, when reinforcement learning changes how an existing model responds to tasks. Teams comparing training systems can now look up results for that stage alongside the suite’s pretraining tests. MLPerf measures the speed of serving a model’s replies in its separate inference tests.

Four attempts per coding task

The test starts from a shared Qwen3.5-397B-A17B checkpoint, a saved state of the model. It uses a reinforcement learning method known as GRPO. During training, the model produces several candidate answers to a task. The method uses their relative rewards to adjust the model toward higher-scoring responses. The system does work both to generate those candidates and to update the model from the resulting feedback.

To check progress, the benchmark uses 251 held-out tasks from the easy split of the R2E-Gym subset. The rules call for four generated answers per task. A task counts toward the pass@4 score when at least one of the four answers passes its tests. The benchmark sets the quality target at 0.69 on that measure. The score refers to this task set and the four-attempt evaluation. Teams save checkpoints late in training and assess them after the run. The clock stops at the weight update that produced the qualifying checkpoint. Saving that final checkpoint and running the evaluation happen outside the scored time.

MLCommons discards the fastest and slowest times from the submitted runs and averages the rest. For a submission with three runs, the middle time is the one that remains. The public logs give the start and stop times for each run, along with its recorded evaluation score.

A fictional accelerator and a clip on cards with abstract shapes representing candidate answers
AI-generated conceptual image: A clip on a set of patterned cards represents checking candidate answers for a passing result. · AI Ping

Ironwood reaches the target in 18.02 minutes with 640 chips

Google Cloud reported 18.02 minutes for the post-training test on 640 Ironwood TPUs. The public logs show that the second of its three runs produced the middle time and scored about 0.765. Google’s submission describes how it split the job. It assigned 128 chips to updating the model and 512 to generating model outputs. Google says its system ran the two parts of the job concurrently.

NVIDIA submitted results on both Rubin and Blackwell Ultra GB300 systems using 64 GPUs each. The middle elapsed times in the corresponding run logs were 25.93 minutes for Rubin and 36.99 minutes for GB300. NVIDIA reported a roughly 1.4-fold performance difference for this test in its comparison table. The system descriptions mark the Rubin setup as available on premises and the GB300 setup as available in the cloud.

One fictional accelerator beside a chassis holding several accelerator modules
AI-generated conceptual image: A single module and a fictional multi-chip chassis illustrate how accelerator counts can differ across submissions. · AI Ping

Chip count and availability accompany the time

The 18.02-minute Ironwood entry used 640 chips. The cited Rubin entry used 64 GPUs. Both times describe complete systems of the specified size. Measuring either platform’s speed at a common chip count would require results for those configurations. The submissions also identify how each system is available, which matters to teams comparing a cloud service with a system they would install. The time-to-target column by itself leaves equipment costs and energy use unanswered.

The same release replaces an earlier recommendation benchmark with DLRMv4-HSTU, which measures training for a model that uses sequences of user activity. It uses the Yambda music-service dataset in place of the previous test’s advertising click data. The target is 0.75 AUC, a measure of how well the model ranks positive examples ahead of negative examples. The two tests therefore report times against different measures of quality.

Sources