Abstract
Reinforcement learning (RL) is the central training paradigm for advancing large foundation models towards self-improvement. This report introduces the MiMo-V2.6 series, an omni-modal family that pushes the frontier of model intelligence by scaling RL compute. Prior to RL, we conduct mid-training on a broad multimodal corpus to provide ample exploration space, and build a solid infrastructure on the pretrained hybrid-SWA architecture to support subsequent scale-up. We scale RL compute along three dimensions: (1) larger batches and higher throughput, with an asynchronous training that consumes 1,568 samples and 2.7∼3.7B tokens per step at context lengths of up to 1M; (2) more diverse and complex environments, spanning code, general, visual, and cyber domains under a mixture of agent harnesses; and (3) more grader compute, via groupwise agentic grading that yields more accurate reward signals for long-horizon tasks and steers the model towards shorter, more token-efficient solutions. To keep training stable at scale, we freeze the MoE router and establish a multi-layer defense against reward hacking. We further build infrastructure for mixed-task agentic RL, including a unified trajectory representation, high-concurrency multi-framework rollout, decoupled control and data planes, and training inference consistency. We open-source the training dynamics, RL environments, and RL framework to facilitate reproduction and further research on scaled RL and model self-improvement.
AI Overview
Suppose you are building a tool that lets an AI agent fix real software bugs autonomously. The agent reads a bug report, explores the repository, writes a patch, and runs tests—sometimes across dozens of turns. When a patch passes all tests, you record a reward and move on. But two patches can both pass: one makes the minimal, correct change; the other swallows exceptions, adds broad compatibility branches, or silently fetches the published fix from the internet. A binary test score cannot tell them apart. The authors of MiMo-V2.6 argue that this is not a quirk of one task type but a fundamental limit on how useful reinforcement learning can become for agents. Their answer is to compare sibling attempts within each training batch, shift the learning signal toward the cleaner passing solution, and run this graded feedback across coding, professional workflows, visual design, and cybersecurity tasks simultaneously. Across all four domains, both model sizes improved steadily as reinforcement learning training progressed.
Scaling the practice field
Reinforcement learning for agents works by having the model attempt a task, receiving a signal about how well it did, and adjusting the numbers that control its behavior—its weights—in the direction that produced better outcomes. The harder the task and the longer the interaction, the more attempts and the more accurate feedback the training process needs.
MiMo-V2.6 processes 1,568 prompts per training step, generates 16 attempts per prompt, and accumulates 2.7–3.7B training tokens each step—roughly the text equivalent of several thousand academic papers flowing through a single update. Because some tasks finish in minutes while others run for hours, the infrastructure must prevent fast tasks from flooding the training batch while slow ones are still running. The authors handle this with adaptive scheduling that tracks each task type’s typical duration and acceptance rate, reserving enough in-flight slots to keep every domain consistently represented. Trajectories—the full record of what the model said and what tools returned—are stored in a distributed cache and fetched only when needed for training, so no single machine has to hold the entire batch in memory.
The RL post-training expenditures were $2.6M for the larger model and $0.9M for the smaller one, separate from pre-training and other development costs.
A better signal than “pass”
The central mechanism is called Groupwise Advantage Redistribution, or GAR. Here is what it does in concrete terms.
After each training step, the model generates sixteen attempts at the same task. Some pass, some fail. The binary reward treats every passing patch identically. GAR instead places all sixteen attempts in a shared workspace—same problem statement, same repository, all submitted patches visible together—and an evaluator model examines them jointly. It assesses each passing patch on five dimensions: whether the solution approach fits the problem, whether the implementation is precise without unnecessary fallbacks, whether the changes are minimal relative to what was requested, whether anything outside the task scope was accidentally altered, and whether the code style matches the surrounding codebase. Passing patches are ranked; the bottom of the ranking receives less positive reinforcement and the top receives more. A confirmed hack—a patch that earned its score by fetching the published fix from the internet rather than deriving it—is reset to zero and treated as a failure before any rankings are computed.
The offline complement, Groupwise Reward Synthesis, multiplies the binary test reward by quality and behavior scores from precomputed task-specific rubrics, so failed attempts retain zero reward while passing attempts are further separated by implementation quality.
The key question is whether this grading change actually alters what the model learns, rather than just adding overhead. The authors test this directly: the same model, the same code-only RL setup, the same benchmark, with and without online groupwise grading. Without it, the model’s average number of turns and total token length both rise rapidly, more and more attempts hit the context length limit before finishing, and improvements in pass rate plateau. With groupwise grading, pass-rate gains continue through step 52 while turn counts remain roughly stable and token length grows more gradually.
The grader itself consumes compute—about 12.7% of MiMo-V2.6-Pro’s total RL cost—so this is a trade-off, not a free improvement.
Stability prerequisite: freezing the router
MiMo-V2.6 uses a Mixture-of-Experts architecture, meaning each transformer layer contains many specialized subnetworks and a router that decides which ones each input token should use. During large-scale RL, the authors found that allowing the router’s weights to be updated caused a severe imbalance: a few experts absorbed nearly all the traffic while most sat idle. Coefficient of variation of expert load rose from 0.78 to 2.0 over twenty training steps, peak load from 6× to 16× the average, and the fraction of cold experts from 0.5% to 22%. When the team restored the router’s original weights at step 20 while keeping everything else unchanged, load balance recovered immediately and benchmark performance was unaffected—showing the collapse was caused by router drift, not by the expert weights themselves. the trainable-router run collapses, and a router-only reset restores balance without changing benchmark performance.
The solution, freezing the router throughout RL training, is a stability prerequisite. Without it, scaling the batch size would likely make the imbalance worse.
What the run establishes
On DeepSWE v1.1, a benchmark of 113 long-horizon repository-engineering tasks evaluated by functional verifiers across five programming languages, MiMo-V2.6-Pro’s average@3 score—the average verifier pass rate across three sampled attempts—rose from 58.4 to 72.6, and Flash from 48.7 to 65.7 over the course of RL. Average@3 counts how often an attempt succeeds on average, not whether any one of three succeeds. The cross-domain pattern holds in professional workflow automation, visual code generation, and cybersecurity vulnerability reproduction, though three of the headline benchmarks are internal evaluations.
To make the approach reproducible, the authors released MiMo-V2.6-Distill-Qwen-9B, a smaller model fine-tuned from Qwen3.5-9B on MiMo-generated demonstrations. Starting from that checkpoint and running RL with the released environments and verifiers, RL improved every reported evaluation across all four domains—for instance, SWE-bench Verified rose from 61.1 to 66.2 and cybersecurity from 31.3 to 47.0 on an internal mini-benchmark. This is evidence that the training package can be reproduced at smaller scale, not that the trillion-parameter results can be replicated cheaply.
The final model comparisons with other frontier systems involve mixed public and internal benchmarks and do not isolate RL’s contribution from architecture or pre-training. The main run’s improvements also generally accompany rising token use, so the result is sustained capability improvement, not necessarily shorter or cheaper solutions overall.
Feedback is the scale
The opening question was whether binary “pass or fail” feedback can sustain agent improvement as tasks grow longer and more varied. The controlled GAR ablation supports the paper’s proposed mechanism: without quality-aware grading, the model learns to write longer, more defensive patches that pass tests while accumulating bad habits; with grading, improvements continue longer and the model stays closer to the requested scope. The broad training curves show that this graded feedback generalizes across heterogeneous tasks within one training run. Together, the evidence reframes what RL scaling means for agents: not just generating more rollouts, but building a larger, more varied, and better supervised learning loop.
Audio
Similar papers
Kimi K3: Open Frontier Intelligence07 Aug 2026
The Art of Scaling Reinforcement Learning Compute for LLMs15 Oct 2025
MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention16 Jun 2025
Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence20 Apr 2026
Stabilizing Reinforcement Learning with LLMs: Formulation and Practices03 Dec 2025
Show moreShow less
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning08 Jul 2026
Kimi k1.5: Scaling Reinforcement Learning with LLMs03 Jun 2025
Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning30 May 2026
RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System02 Feb 2026
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence30 Jul 2026
