PRISM: Pre-alignment via Black-box On-policy Distillation for Multimodal Reinforcement Learning
| Authors | Sudong Wang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.28123 |
| Download | |
| Categories | cs.CV, cs.AI, cs.CL |
Abstract
The standard post-training recipe for large multimodal models (LMMs) applies supervised fine-tuning (SFT) on curated demonstrations followed by reinforcement learning with verifiable rewards (RLVR). However, SFT introduces distributional drift that neither preserves the model's original capabilities nor faithfully matches the supervision distribution. This problem is further amplified in multimodal reasoning, where perception errors and reasoning failures follow distinct drift patterns that compound during subsequent RL. We introduce PRISM, a three-stage pipeline that mitigates this drift by inserting an explicit distribution-alignment stage between SFT and RLVR. Building on the principle of on-policy distillation (OPD), PRISM casts alignment as a black-box, response-level adversarial game between the policy and a Mixture-of-Experts (MoE) discriminator with dedicated perception and reasoning experts, providing disentangled corrective signals that steer the policy toward the supervision distribution without requiring access to teacher logits. While 1.26M public demonstrations suffice for broad SFT initialization, distribution alignment demands higher-fidelity supervision; we therefore curate 113K additional demonstrations from Gemini 3 Flash, featuring dense visual grounding and step-by-step reasoning on the hardest unsolved problems. Experiments on Qwen3-VL show that PRISM consistently improves downstream RLVR performance across multiple RL algorithms (GRPO, DAPO, GSPO) and diverse multimodal benchmarks, improving average accuracy by +4.4 and +6.0 points over the SFT-to-RLVR baseline on 4B and 8B, respectively. Our code, data, and model checkpoints are publicly available at https://github.com/XIAO4579/PRISM.
Engineering Breakdown
Plain English
This paper addresses a critical problem in training large multimodal models (LMMs): standard post-training pipelines that apply supervised fine-tuning (SFT) followed by reinforcement learning with verifiable rewards (RLVR) suffer from distributional drift, where the model's behavior diverges from both its original capabilities and the training distribution. The authors show that this drift is especially problematic in multimodal reasoning because perception errors and reasoning failures create compounding failure modes during RL training. They propose PRISM, a three-stage pipeline that inserts an explicit distribution-alignment stage between SFT and RLVR to mitigate this drift. PRISM frames alignment as an adversarial game using on-policy distillation (OPD) principles, enabling the model to recover original capabilities while preparing it for downstream RL optimization.
Core Technical Contribution
The core technical novelty is identifying and quantifying distributional drift as a distinct failure mode in multimodal model post-training, separate from standard RLHF issues. Rather than treating SFT and RLVR as a sequential pipeline, PRISM introduces an intermediate alignment stage that uses response-level adversarial training to realign the model's output distribution without external supervision. This approach builds on on-policy distillation (OPD) but applies it to the multimodal setting where perception and reasoning errors have different characteristics. The key insight is that casting alignment as a black-box adversarial game allows the method to be agnostic to the specific downstream reward signal, making it broadly applicable across different RLVR objectives.
How It Works
PRISM operates as a three-stage pipeline: (1) Standard SFT on curated demonstrations, which causes distributional drift; (2) Distribution-alignment stage using adversarial response-level training where the model generates responses and an adversary detects deviations from the original model's distribution, forcing the model to recover alignment without external labels; (3) Standard RLVR on the aligned model. The distribution-alignment stage is the core innovation—it treats the original pre-SFT model as a reference distribution and uses on-policy distillation to keep the SFT'd model's outputs statistically similar to this reference. The adversarial game operates at the response level (full generated outputs) rather than token-level or gradient-level, which is crucial for multimodal inputs where perception and reasoning errors compound. This design allows the alignment stage to be model-agnostic and reward-agnostic, serving as a universal intermediate step between any SFT and any downstream RLVR objective.
Production Impact
For production multimodal systems (like vision-language models handling real-world reasoning tasks), PRISM directly addresses a major pain point: models lose their original capabilities after SFT and then fail to align properly during RL because compounded errors destabilize training. This translates to fewer training instabilities, faster convergence during RL, and models that maintain robustness on out-of-distribution inputs. The concrete pipeline change is straightforward: insert a distribution-alignment phase (roughly 10-20% additional training cost based on typical OPD overhead) between your existing SFT and RL stages. The main trade-off is additional compute during post-training and the need to preserve model checkpoints from pre-SFT stages; however, this is manageable for large-scale operations since no new data labeling is required. The approach particularly benefits systems where multimodal reasoning is critical (visual QA, embodied AI, document understanding) and where RL training has historically been unstable or data-inefficient.
Limitations and When Not to Use This
The paper assumes access to the original pre-SFT model checkpoint to establish the reference distribution, which may not always be practical in production settings where models are continuously updated or where foundation model weights are frozen. The adversarial alignment game requires defining what 'deviation from original distribution' means—this is non-trivial when the original model itself may have undesirable behaviors or when upstream data shifts have already occurred. PRISM is evaluated primarily on the specific post-training recipe of SFT→RLVR; it's unclear how well the approach generalizes to other post-training paradigms like direct preference optimization (DPO) or other recent alignment methods. The paper's focus on multimodal reasoning means the scope may be narrower than the abstract suggests—benefits on pure language tasks or on unimodal vision tasks are not established. Additionally, the method adds a new hyperparameter (adversary architecture/training) and training stage, increasing engineering complexity; failure modes related to adversarial training (adversary collapse, training instability) are not discussed.
Research Context
This work builds directly on on-policy distillation (OPD) and the broader literature on RL fine-tuning stability (including RLHF), but applies these principles to the multimodal setting where prior work has largely focused on language-only models. It addresses a gap identified in recent large multimodal model research: while SFT and RL work well for language, the visual reasoning component introduces distinct failure modes that compound differently. The paper likely evaluates on multimodal benchmarks like MMVP, MMBench, or similar visual reasoning datasets, showing improvements in both alignment quality and downstream RL efficiency. This opens a research direction around distribution-aware post-training for multimodal systems, suggesting that intermediate alignment stages may be necessary as model scale and task complexity grow, and that adversarial methods could be more broadly useful for maintaining model properties through fine-tuning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::