You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass
| Authors | Yinuo Yang et al. |
| Year | 2026 |
| HF Upvotes | 11 |
| arXiv | 2604.10966 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We present a discriminative multimodal reward model that scores all candidate responses in a single forward pass. Conventional discriminative reward models evaluate each response independently, requiring multiple forward passes, one for each potential response. Our approach concatenates multiple responses with separator tokens and applies cross-entropy over their scalar scores, enabling direct comparative reasoning and efficient N-way preference learning. The multi-response design also yields up to Ntimes wall-clock speedup and FLOPs reduction over conventional single-response scoring. To enable N-way reward evaluation beyond existing pairwise benchmarks, we construct two new benchmarks: (1) MR^2Bench-Image contains human-annotated rankings over responses from 8 diverse models; (2) MR^2Bench-Video is a large-scale video-based reward benchmark derived from 94K crowdsourced pairwise human judgments over video question-answering spanning 19 models, denoised via preference graph ensemble. Both benchmarks provide 4-response evaluation variants sampled from the full rankings. Built on a 4B vision-language backbone with LoRA fine-tuning and a lightweight MLP value head, our model achieves state-of-the-art results on six multimodal reward benchmarks, including MR^2Bench-Image, MR^2Bench-Video, and four other existing benchmarks. Our model outperforms existing larger generative and discriminative reward models. We further demonstrate that our reward model, when used in reinforcement learning with GRPO, produces improved policy models that maintain performance across standard multimodal benchmarks while substantially improving open-ended generation quality, outperforming a single-response discriminative reward model (RM) baseline by a large margin in both training stability and open-ended generation quality.
Engineering Breakdown
Plain English
This paper introduces a more efficient way to evaluate multiple AI response candidates using a single neural network forward pass instead of one pass per response. Conventional reward models score each response independently, which requires N separate computations for N candidates—wasteful and slow. The authors' approach concatenates all responses with separator tokens and applies cross-entropy loss across their scalar scores simultaneously, achieving up to N-times wall-clock speedup while maintaining or improving accuracy. They validate this on two new benchmarks they constructed: MR²Bench-Image with human-ranked responses from 8 diverse models, enabling direct N-way preference learning instead of just pairwise comparisons.
Core Technical Contribution
The core novelty is reformulating discriminative reward modeling from single-response scoring to multi-response scoring in one forward pass. Instead of running the reward model N times to score N candidates, the model processes all candidates concatenated together in a single pass, using cross-entropy loss over their relative scalar scores to enable direct comparative reasoning. This is a simple but powerful architectural change that exploits the model's ability to compare multiple sequences in parallel, rather than serial evaluation. The insight is that reward models should naturally perform N-way preference ranking simultaneously rather than reducing it to repeated pairwise or singleton comparisons.
How It Works
The input is a prompt followed by multiple candidate responses, all concatenated with learned separator tokens in a single sequence. This concatenated sequence passes through a standard transformer-based reward model encoder. The model outputs one scalar score per response (extracted from position-specific output heads or pooled representations). The training objective is cross-entropy loss computed over all N scores simultaneously—typically using a softmax over the scores with ground-truth rankings converted to soft targets (e.g., position 1 gets highest weight, position N gets lowest). At inference, all N responses are evaluated in one forward pass; you extract their scores and rank them. The efficiency gain comes from amortizing the transformer's computation cost across all candidates rather than running the transformer N times.
Production Impact
undefined
Limitations and When Not to Use This
The approach assumes ground-truth rankings exist or can be reliably annotated; real production data often has incomplete or inconsistent human preferences across 3+ options, which the paper doesn't deeply address. Scalability is limited by context window—comparing 100 responses or ultra-long outputs becomes infeasible due to transformer length constraints; the paper doesn't propose hierarchical or chunking strategies. The method requires retraining reward models on N-way ranked data, which is more expensive and less common than pairwise comparison data (a chicken-and-egg problem); their two benchmarks are helpful but narrow in domain coverage (image responses only from the abstract). There's also an open question about whether the model truly learns true comparative reasoning or exploits position biases when all responses are concatenated; ablation studies on response ordering and position-specific artifacts are not mentioned. Finally, the paper doesn't address how this scales to open-ended ranking (e.g., where two responses are genuinely tied or incomparable).
Research Context
This work builds on the growing literature in discriminative reward modeling for language models and extends it from pairwise (Bradley-Terry style) to N-way ranking. It sits at the intersection of efficient inference (batching multiple evaluations) and preference learning, complementing recent work on direct preference optimization (DPO) and other RLHF alternatives that also move away from scalar reward models. The paper directly addresses a practical bottleneck in model evaluation pipelines that has been largely ignored—most research focuses on the quality of rewards, not the efficiency of computing them. By constructing MR²Bench-Image and framing N-way ranking as a first-class problem, the authors open a new evaluation paradigm that more closely mirrors real-world comparison tasks (e.g., human judges ranking multiple outputs simultaneously) rather than artificially decomposing them into pairs.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::