Breaking the Reward Trade‑off: Flow‑OPD Brings Multi‑Teacher OPD to Image Generation

Problem with GRPO in multi‑task post‑training

Single‑scalar GRPO reward pushes a flow‑matching model to the performance ceiling on an isolated task but causes severe degradation on other tasks, leading to “reward hacking” where the model fails at text rendering or style‑conditioned generation.

Mixing several scalar rewards (mixed‑reward GRPO) introduces catastrophic forgetting and parameter interference each time a new reward is added, as shown by drops in basic visual generation and text rendering abilities.

Flow‑OPD framework

Flow‑OPD introduces on‑policy distillation (OPD) to flow‑matching diffusion models. It consists of three stages:

Train task‑specific teacher models with single‑reward GRPO.

Cold‑start a student model either by supervised fine‑tuning (SFT) or by model merging, providing strong initial performance.

Perform multi‑teacher OPD distillation: the student generates an image trajectory step‑by‑step; at each step a hard‑routing module selects the expert teacher whose specialty matches the current instruction (e.g., text‑rendering specialist or basic visual element specialist). The student receives dense supervision from the selected teacher.

Multi‑teacher OPD distillation

The reward is the negative mean‑squared error between the student’s velocity field and the teacher’s velocity field, replacing the scalar GRPO reward. Updates use a PPO‑style algorithm. This dense, expert‑driven signal resolves gradient conflicts that arise with sparse scalar rewards.

Manifold Anchor Regularization (MAR)

To avoid background‑mode collapse and semantic redundancy, Flow‑OPD anchors optimization to a frozen “Aesthetic Teacher” model. The teacher provides high‑fidelity KL regularization, preserving visual quality and diversity while the student follows semantic instructions.

Experimental validation

Using stable‑diffusion‑3.5‑medium as the baseline and the Flow‑GRPO data pipeline, Flow‑OPD was evaluated on tasks such as text rendering and overall image quality. The student matched or exceeded each specialist teacher on all reported metrics, eliminating the seesaw degradation observed with prior methods. In edge cases where all teachers failed, the student still produced coherent results, demonstrating a “student surpasses teachers” effect.

Cold‑start ablation

SFT offers extensibility and the ability to absorb heterogeneous teacher knowledge; model merging incurs zero training cost and yields perfect alignment for homogeneous teachers. Both strategies accelerate convergence compared with training from scratch.

MAR image‑quality regularization

Standard GRPO optimization, with coarse reward granularity, often leads to background‑mode collapse. MAR’s KL regularization on the frozen aesthetic manifold provides full‑process supervision, improving structural diversity, visual fidelity, and alignment with human preference, as confirmed by quantitative tables.

Why online multi‑expert supervision works

Dense expert signals anchored to a high‑fidelity manifold eliminate single‑model bias and gradient interference, allowing the student to integrate multiple capabilities without catastrophic forgetting.

Future directions

Dynamic scheduling of heterogeneous teachers across modalities and architectures.

Self‑evolving cross‑manifold trajectories that go beyond any teacher’s expertise.

Lightweight online distillation algorithms (e.g., MoE‑style teacher clusters or parameter‑sharing) to reduce compute and memory overhead.

Paper: https://arxiv.org/abs/2605.08063

Code: https://github.com/CostaliyA/Flow-OPD