C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
| Authors | Akira Kawabata & Saku Sugawara |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.13618 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification. However, most existing methods require costly rubric annotations, limiting scalability. Moreover, we find that rubric generation is vulnerable to a failure of cooperation; low-quality rubrics actively mislead reward models rather than help. Inspired by the principle of cooperative communication, we propose Cooperative yet Critical reward modeling (C2), a framework that significantly improves reward model judgments by having the reward model critically collaborate with a rubric generator trained solely from binary preferences. In C2, we synthesize helpful and misleading rubric pairs by measuring how each rubric shifts the reward model toward or away from the correct preference. Using these contrastive pairs, we train a cooperative rubric generator to propose helpful rubrics, and a critical verifier to assess rubric validity before making its judgment, following only rubrics it deems helpful at inference time. C2 outperforms reasoning reward models trained on the same binary preferences, with gains of up to 6.5 points on RM-Bench and 6.0 points length-controlled win rate on AlpacaEval 2.0. Without external rubric annotations, C2 enables an 8B reward model to match performance achieved with rubrics from a 4times larger model. Overall, our work demonstrates that eliciting deliberate cooperation in rubric-augmented verification makes reward models more trustworthy in a scalable way.
Engineering Breakdown
Plain English
This paper addresses a critical bottleneck in reward modeling: rubrics (explicit evaluation criteria) help models make better judgments, but generating and annotating high-quality rubrics at scale is expensive and often fails. The authors discovered that low-quality rubrics actively harm reward models instead of helping them. They propose C2 (Cooperative yet Critical reward modeling), a framework that trains a rubric generator using only binary preference data (no expensive rubric annotations), then synthesizes pairs of helpful and misleading rubrics to teach the reward model to make more reliable judgments. The key innovation is treating rubric generation as a learned process guided by how rubrics actually impact reward model behavior, rather than requiring human-annotated rubric data.
Core Technical Contribution
The core novelty is a cooperative-yet-critical framework that decouples rubric generation from expensive human annotation. Instead of requiring humans to write good rubrics upfront, C2 trains a rubric generator end-to-end using only binary preference signals, then measures each rubric's effect on the reward model's predictions to identify which rubrics help vs. mislead. This inverts the typical pipeline: rather than rubrics guiding the reward model in isolation, the reward model's behavior feedback guides rubric quality. The authors explicitly identify and solve the 'failure of cooperation' problem—where poorly-generated rubrics degrade performance—by synthesizing contrastive rubric pairs that teach the reward model to distinguish helpful from harmful criteria.
How It Works
C2 operates in a cycle of three interacting components: (1) a reward model that learns from binary preference pairs, (2) a rubric generator trained to produce evaluation criteria without any human rubric annotations, and (3) a scoring mechanism that measures how each generated rubric shifts the reward model's predictions. Given a preference pair (preferred output, dispreferred output), the rubric generator creates multiple candidate rubrics explaining the preference. The framework then evaluates each rubric by checking whether it correctly predicts the preference direction—rubrics that consistently align with actual preferences are deemed 'helpful,' while those that point the wrong way are 'misleading.' The reward model is trained with both types as contrastive examples: it learns to recognize when a rubric is trustworthy vs. when it should ignore or down-weight a misleading criterion. Over iterations, both the rubric generator and reward model improve: the generator learns which rubric types the reward model finds useful, and the reward model becomes more robust by learning to critically evaluate rubrics rather than blindly following them.
Production Impact
For teams building RLHF systems or preference-based learning pipelines, this approach directly reduces data annotation costs by eliminating the need for expensive human-written rubrics. Instead of hiring domain experts to craft rubrics (often requiring domain knowledge and multiple rounds of iteration), you only need binary preference labels, which are cheaper and faster to collect at scale. In a production system, this means faster model iteration cycles and lower labeling overhead, particularly valuable when scaling to new domains or tasks where good rubrics aren't obvious. The trade-off is increased computational complexity: you now need to train both a reward model and a rubric generator in a coupled feedback loop, which increases training time and memory overhead compared to simple reward modeling. Integration is straightforward if you already have a preference-based RL pipeline—you essentially swap in C2's training loop—but you'll need to monitor that the learned rubrics remain interpretable and aligned with your actual evaluation criteria, since the framework optimizes for predictive usefulness rather than human readability.
Limitations and When Not to Use This
The paper assumes binary preference data is available and reliable, but in practice, preference labeling can be noisy or subjective, which could degrade both the reward model and rubric generator. The framework doesn't explicitly address how to handle distribution shift: if the test domain differs significantly from the training preferences, the learned rubrics may not transfer well. There's also a potential interpretability gap—since rubrics are learned end-to-end, they may optimize for predictive power rather than being human-understandable, making it harder to debug failures or ensure the model is capturing the right concepts. The paper doesn't thoroughly explore failure modes when the rubric generator collapses to trivial or gaming solutions, nor does it provide clear guidance on hyperparameter tuning for the rubric-reward model interaction. Finally, computational cost is glossed over; training two coupled models simultaneously likely increases wall-clock time and memory significantly compared to baseline reward modeling, and this trade-off needs empirical validation across different scales.
Research Context
This work builds on the growing literature of rubric-based and criterion-guided reward modeling, extending recent findings that explicit evaluation criteria improve reward model reliability. It directly addresses limitations of prior work (like ERM and similar approaches) that require expensive rubric annotation by proposing a self-supervised alternative. The paper contributes to the broader RLHF and preference learning community by tackling the scalability challenge—showing that you can achieve rubric-like benefits without human rubric writing. This opens a research direction toward fully learned, self-improving evaluation frameworks where models learn what criteria matter by observing preference signals, rather than relying on human-provided structure. The findings also connect to work on cooperative learning and adversarial robustness, suggesting that reward models benefit from learning to critically evaluate their own feedback sources.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::