MLPerf Training Adds First LLM Post-Training Benchmark

MLCommons is adding its first LLM post-training benchmark to MLPerf Training, beginning with the v6.1 submission round this month. Rather than measuring how quickly a system can pretrain a model, the new test tracks how long it takes to train a 397-billion-parameter model to fix real software bugs using reinforcement learning. The change reflects where much of the recent progress in coding and agentic models is happening: post-training rather than pretraining. Until now, MLPerf Training did not include a standard test for that stage of model development.

What the benchmark measures

According to the MLCommons announcement, dated September 24, the task is agentic software engineering. An LLM-driven agent works on real coding problems inside sandboxed environments, and the system is scored on wall-clock time to reach a fixed quality target: a pass@4 score of 0.69, meaning the model solves at least 69% of validation problems in at least one of four attempts.

The dataset is R2E-Gym, released under the Apache 2.0 license, with 700 training problems and 251 validation problems drawn from Python projects including Aiohttp, NumPy and Pandas. A separate review of the benchmark says the tasks are based on GitHub commits and require agents to generate code fixes inside containerized environments. Each run is limited to a context window of 65,536 tokens and a maximum of 30 agent turns.

MLCommons says the reference implementation takes 140 to 160 minutes per run on GB300 NVL72 racks. The task force designed the benchmark so a full run fits within 1,024 GB300 hours, according to the review, which keeps the cost of participation bounded.

The training method and the model

The benchmark uses reinforcement learning with verifiable rewards, or RLVR. The task force chose it, per the second source, because it suits software tasks: an agent attempt, called a rollout, either passes the tests or does not. The system samples multiple rollouts per problem, assigns a binary pass/fail reward, and updates the model with Group Relative Policy Optimization (GRPO), which compares rewards within a group of attempts at the same problem.

The model is Qwen 3.5 397B, which the review describes as an open-weight mixture-of-experts model under the Apache 2.0 license, released in February 2026. MLCommons describes it as having 397 billion total parameters and 17 billion active per token. The review adds that it is the first to productize a hybrid architecture that combines Gated DeltaNet with sparse mixture-of-experts layers, activating one shared expert and 10 of 512 routed experts in each MoE layer. That architecture claim comes from the review and was not independently checked against the model's own documentation for this article.

Why post-training is hard to benchmark

The design choices reveal what makes this workload different from pretraining. MLCommons says post-training has driven major recent advances in LLMs, and that the benchmark is meant to test complete system integration. In its description, a single run combines mixture-of-experts policy training, high-throughput inference to generate trajectories, asynchronous trajectory collection, agent sandbox execution, and long multi-turn interactions with heavy key-value cache demands.

That mix of workloads is what makes the benchmark different. A pretraining benchmark stresses a relatively uniform loop of forward and backward passes. An RLVR run alternates between generating many long agent trajectories, executing code in sandboxes to get rewards, and updating the policy. A fast accelerator is necessary but not sufficient: scheduling between inference and training, sandbox throughput and memory behavior can each become the bottleneck. By measuring the whole pipeline to a quality target, the benchmark rewards systems that balance all of these parts, not just peak compute.

The review notes that real-world post-training combines distillation, reinforcement learning and supervised fine-tuning, spanning math, reasoning and agentic tasks. The benchmark deliberately narrows this to a single RL-centered workload so that results are comparable. That is a reasonable trade-off for a benchmark, but it also means the result describes one recipe on one model, and the sources do not claim that it predicts performance on every post-training pipeline.

Quality as a gate, not a footnote

The new benchmark times a run only to a fixed quality target. The MLCommons post sums up the philosophy in a few words: results must be "both correct and fast." In practice, a target like this stops a system from posting a quick run that never reaches the pass@4 target. For reinforcement learning, where training can be noisy and sensitive to settings, a quality gate is plausibly a demanding constraint, and run-to-run variance could become a point of discussion once results arrive.

What it means for builders

For teams that train or fine-tune large models, the benchmark offers a shared reference for a workload that has mostly been described in company blog posts and ad hoc reports. It specifies a concrete open model, an open dataset, a stopping criterion and a budget. Hardware vendors and cloud providers that submit will have to publish numbers on the same task, which may help buyers compare systems for RL-heavy work. The reference baseline of 140 to 160 minutes on GB300 NVL72 racks gives submitters a starting point to beat, though the sources do not say how much optimization headroom MLCommons expects.

It also reflects a quiet change in what counts as an ML engineering problem. Evaluating infrastructure by pretraining throughput alone leaves out inference-heavy training loops, where generation, tool execution and memory pressure matter as much as matrix multiplication. Putting that in the benchmark suite creates an incentive for vendors and framework authors to optimize it.

Limits and open questions

• No results yet. The sources describe the benchmark's design and reference baseline, but neither identifies who has submitted results or provides any published scores, so it is too early to compare systems.

• Narrow task. The workload covers Python software repair. Math, reasoning and other agentic settings are mentioned as parts of real post-training but are not in this first benchmark.

• Small evaluation set. With 251 validation problems, a 0.69 pass@4 target is a coarse measure, and the sources do not discuss how run-to-run variability is handled.

• Participation. The review says details on participating organizations were not specified.

What to watch next

The first thing to watch is the v6.1 results themselves: which vendors submit, how far they get below the 140 to 160 minute reference time, and what they disclose about their software stacks. A second signal will be whether the benchmark expands beyond software repair to other post-training tasks in later rounds. Last, observers should check whether the reported times track real-world RL training speeds, since a benchmark is only useful if gains on it carry over to production pipelines.