Reading notes on:

Multi-teacher on-policy distillation is the obvious answer to a deployment problem: if math, code, and instruction-following each want their own specialist teacher, distill them all into one student and ship a single model. Frontier systems already do variants of this, from Nemotron 3 Ultra to Kimi K3.

What Open-MOPD contributes is a sharper diagnosis of why the naive recipe underdelivers. The failure is not primarily teacher conflict. It is optimization budget distortion: long domains consume token budget, fast-converging domains keep getting overfunded, and stale student-dependent rewards drift away from the actual current policy.

Open-MOPD: prompt share, token share distortion, and the three-part budget correction pipeline

1. The Capability Integration Gap Is Real

On the controlled SmolLM3-3B-Base testbed, naive M-OPD reaches an average score of 28.05, while distilling each domain separately and keeping them routed as independent experts reaches 31.55. In other words, the mixed student only recovers 35.6% of the improvement between mixed-domain SFT and RouteRL.

The gap is not uniform:

  • Instruction following is the worst-hit domain, finishing 6.16 points below its RouteOPD reference.
  • It is also the only domain whose performance actually drops in the middle of training.
  • Along the training curve, math closes 64% of its headroom, code closes 40%, and instruction following closes only 26%.

That asymmetry is important. It already suggests a resource-allocation problem rather than a simple “the teachers disagree” story.

2. First Diagnosis: It Is Not Teacher Conflict

The paper measures teacher disagreement on each token as

\[c_t = \max_{d \in D} \log \pi_{\phi_d}(y_t \mid x, y_{<t}) - \min_{d \in D} \log \pi_{\phi_d}(y_t \mid x, y_{<t}).\]

If conflicting gradients were the main bottleneck, $c_t$ should be large and frequent. Instead:

  • mean disagreement stays around 0.126 nat;
  • only 0.62% of tokens exceed 1 nat on average;
  • masking or replacing high-conflict tokens actually hurts the final score by 0.52 to 0.83 points.

So the high-conflict tokens that do exist are informative, not noise. The student is not failing because the teachers are fighting each other token by token.

3. The Real Problem: Budget Distortion Happens in Three Different Ways

3.1 Long responses hijack the token budget

With domain prompt counts $(39.8\%, 39.8\%, 20.3\%)$ for math, code, and instruction following, you might expect the update budget to be fairly balanced. But token-mean training does not care about prompt counts. It cares about response length:

\[s_{tok}^d = \frac{n_d L_d}{\sum_{j \in D} n_j L_j}.\]

Math and code responses average around 10,500 tokens. Instruction-following responses average about 409. The result is brutal:

  • Math: 49.7% of gradient tokens
  • Code: 49.3%
  • IF: 0.99%

To equalize token shares by prompt oversampling alone, IF would need to be oversampled by 33.6x, which would destroy the coverage of the long-horizon domains.

3.2 Even balanced token shares drift as domains converge at different speeds

Define the per-token reward magnitude proxy

\[\bar{m}_d = \mathbb{E}_{t \in d}[|r_t|].\]

Early in training, the domains start at very different gaps:

  • Math: 0.019
  • Code: 0.063
  • IF: 0.091

If you artificially force equal token shares, the effective update budget still drifts because $\bar{m}_d$ shrinks at different rates. In the paper’s measurements, IF’s budget share collapses from 48.7% to roughly 9% within 25 steps, while code rises to 63.8%.

3.3 Reusing rollouts makes the reward stale

Under $K$ inner updates per rollout batch, the frozen teacher log-probs stay fixed while the student changes after each inner step. The mismatch grows quickly:

  • rollout-to-current KL reaches 0.059 at $K = 4$ and 0.216 at $K = 32$;
  • PPO clipping fraction rises to 0.576 at $K = 4$ and peaks around 0.860 at $K = 32$.

So even if the budget were perfect, the supervision becomes mathematically inconsistent as soon as the student moves away from the rollout state.

4. Open-MOPD’s Three Fixes

4.1 Token-share balancing

If the target budget share is $g_d^*$, set

\[w_{share}^d = \frac{g_d^*}{s_{tok}^d}.\]

This makes the weighted contribution of each domain exactly match the target share. With equal one-third targets, the observed weights become approximately:

  • Math: 0.69
  • Code: 0.66
  • IF: 32.7

The short IF trajectories get amplified without sacrificing prompt diversity.

4.2 Gap-following allocation

Static balancing is not enough, because domains converge at different speeds. Open-MOPD therefore scales by the running gap proxy instead of its inverse:

\[\tilde{w}_d = w_{share}^d \cdot \text{Clamp} \left( \left(\frac{m_d}{m_{ref}}\right)^\alpha, 0.05, 20 \right),\]

then normalizes:

\[w_{gap}^d = \frac{\tilde{w}_d}{\sum_j \tilde{w}_j s_{tok}^j}.\]

The sign on $\alpha$ is the important bit. An inverse rule would create positive feedback: a domain that is already converging gets even more weight, converges faster, and collapses training. The paper reports exactly that pathology when the wrong sign is used: IF weight explodes from 24.4 to 80.9, and training crashes at step 74.

4.3 Student reward refresh

Teacher log-probs are cached once, but the student-dependent part of the reward is recomputed at every inner step:

\[\delta_t^{(k)}(v) = \text{sg}\!\left[ \log \pi_{\phi_{d(x)}}(v \mid x, y_{<t}) - \log \pi_\theta^{(k)}(v \mid x, y_{<t}) \right],\] \[r_t^{(k)}(v) = \delta_t^{(k)}(v)\, \tilde{\pi}_\theta^{(k)}(v \mid x, y_{<t}).\]

This is the nice engineering trick in the paper: PPO already needs the actor forward pass, so refreshing the reward adds essentially zero extra forward passes.

5. The Results Are Close to the Routed Oracle

The end-to-end comparison is striking:

Method Total score Recovery rate
RouteRL 32.35 100.0%
RouteOPD 31.55 88.0%
Naive M-OPD 28.05 35.6%
Open-MOPD 31.24 83.4%

So the open recipe lands within 0.31 points of RouteOPD while keeping a single deployable student.

The ablation ladder makes the contribution of each component visible:

  • naive M-OPD: 29.28
  • +share + gap: 30.43
  • full Open-MOPD: 31.24

The compute overhead is also modest. Reward prefill plus refresh accounts for only about 2.2% of step time, because generation and actor updates dominate the runtime budget.

If you compare this with The Distributional Lens of Post-Training or Revisiting On-Policy Distillation, the theme is consistent: the load-bearing ingredient is still on-policy data, but on-policy data alone is not enough when multi-domain optimization budgets drift this badly. For the orthogonal question of how coarse or fine the teacher target should be, the right companion read is SOPD.

Takeaways

  1. Prompt ratios are the wrong accounting unit. In mixed-domain post-training, physical token volume is what actually spends gradient budget.
  2. Budget balancing must be dynamic. Equal shares at step 0 do not stay equal once domains learn at different rates.
  3. The stable direction is gap-following, not inverse normalization. More unsolved domains should get more budget; the reverse causes runaway feedback.
  4. Student-dependent rewards must stay fresh. Reusing rollouts without refresh quickly turns dense distillation rewards into stale supervision.
  5. The multi-teacher problem is less mystical than it looks. Most of the gap comes from systems-level budgeting errors, not from irreconcilable teacher disagreement.