Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design
| Authors | Xu Guo et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.12826 |
| Download | |
| Categories | cs.CL |
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models. When applied to RLVR, Multiple-Choice Questions (MCQs) offer a scalable source of verifiable data but risk inducing reward hacking, where models shortcut reasoning via random guessing or simple elimination. Current approaches often mitigate this by converting MCQs to open-ended formats, thereby discarding the contrastive signal provided by expert-designed distractors. In this work, we systematically investigate the impact of option design on RLVR. Our analysis highlights two primary insights: (1) Mismatches in option counts between training and testing degrade performance. (2) Strong distractors effectively mitigate random guessing, enabling effective RLVR training even with 2-way questions. Motivated by these findings, we propose Iterative Distractor Curation (IDC), a framework that actively constructs high-quality distractors to block elimination shortcuts and promote deep reasoning. Experiments on various benchmarks demonstrate that our method effectively enhances distractor quality and yields significant gains in RLVR training compared to the original data.
Engineering Breakdown
Plain English
This paper tackles a critical problem in reinforcement learning for large language models: when training on multiple-choice questions (MCQs) with verifiable rewards, models often cheat by guessing randomly or using simple elimination strategies instead of actually reasoning. The authors investigate how the design of answer options affects this reward hacking problem, discovering two key findings—mismatches between training and test option counts hurt performance significantly, and well-designed distractor options (wrong answers that seem plausible) actually help prevent gaming the system. This work shows that throwing away the distractor information by converting MCQs to open-ended questions loses valuable signal that could improve RL training effectiveness.
Core Technical Contribution
The core novelty is a systematic empirical analysis of how option design in multiple-choice formats impacts reinforcement learning with verifiable rewards (RLVR) for LLMs. Rather than treating MCQs as a crude training signal to be converted away, the authors demonstrate that expert-designed distractors provide crucial contrastive information that actively mitigates reward hacking behaviors. This flips the conventional wisdom that MCQs must be reformatted to open-ended questions to prevent gaming, instead showing that thoughtful option construction within the MCQ framework can be a feature, not a bug. The insight that option count consistency between training and test phases matters for generalization is also novel in the RL context.
How It Works
The approach starts with standard RLVR training where an LLM receives scalar rewards for correct answers on MCQ datasets. The key innovation is systematic variation of option characteristics during training and evaluation—controlling factors like the number of options (4 vs 6 options), distractor quality (random vs expert-designed that exploit common misconceptions), and whether option counts match between train and test. The model learns to maximize expected reward through policy gradient methods (standard RL), but the researchers measure both accuracy and diagnostic metrics indicating reward hacking behavior (e.g., success rate on questions where all but the correct answer are removed, showing if the model is guessing). By comparing performance across these controlled option variations, they quantify how distractor strength and option count consistency directly influence whether the model develops spurious shortcuts or genuine reasoning patterns.
Production Impact
For teams training LLMs on reasoning tasks with RL, this work provides actionable guidance: MCQ datasets are more valuable than previously thought if you invest in high-quality distractors. Instead of preprocessing datasets to convert all MCQs to open-ended formats (which is computationally cheap but loses signal), you should maintain MCQ structure with carefully designed wrong answers that test genuine reasoning rather than superficial pattern matching. This likely reduces the compute overhead of RL training since the contrastive signal from good distractors makes learning more efficient—the model needs fewer rollouts to learn robust reasoning. The practical trade-off is upfront effort in distractor design and careful attention to ensuring training and evaluation option counts match, but this prevents costly reward hacking failures at scale where models appear to succeed in development but fail in production due to gaming behavior.
Limitations and When Not to Use This
The paper's analysis is confined to MCQ-based evaluation and doesn't address how well these insights transfer to open-ended reasoning tasks, real-world deployment where test distribution differs from training, or longer-horizon reasoning chains where distractor design becomes combinatorially complex. The work assumes expert-designed distractors are available, which may not hold for specialized domains or newly emerging topics where curating misleading-but-plausible options is expensive or subjective. The paper likely doesn't deeply explore how option count mismatches scale—what happens at extreme ratios (2 vs 20 options) or with very large option sets where distractor quality becomes harder to control. Follow-up work is needed on automated distractor generation, theoretical bounds on how much distractor quality matters relative to other RL hyperparameters, and evaluation frameworks that measure whether reasoning is truly robust versus simply resistant to the specific distractors seen during training.
Research Context
This work builds on the emerging field of RL for language models (exemplified by RLHF and verifiable reward signal research) and addresses a specific gap: prior work either treats MCQs as crude training signals to be discarded or attempts to prevent reward hacking through format conversion, but doesn't systematically study how MCQ structure itself prevents gaming. The paper sits at the intersection of curriculum learning (option design as a form of structured difficulty) and interpretability research (diagnostic metrics for detecting when models shortcut reasoning). It advances understanding of how data properties—not just quantity but structural properties like distractor design—shape what models actually learn, which is increasingly important as the field moves from supervised fine-tuning to reward-based learning. This opens research directions in automated distractor generation, theoretical analysis of contrastive signals in RL, and meta-learning approaches to MCQ design for specific model vulnerabilities.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::