Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problems
| Authors | Justin Chih-Yao Chen et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2604.04767 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of LLMs, yet a fundamental limitation remains: models cannot learn from problems that are too difficult to solve under their current policy, as these yield no meaningful reward signal. We propose a simple yet effective solution based on task reformulation. We transform challenging open-ended problems into cognitively simpler variants -- such as multiple-choice and cloze formats -- that preserve the original answer while reducing the effective search space and providing denser learning signals. These reformulations span a spectrum from discriminative to generative tasks, which we exploit to bootstrap learning: models first learn from structured, easier formats, and this knowledge transfers back to improve performance on the original open-ended problems. Building on this insight, we introduce Cog-DRIFT, a framework that constructs reformulated variants and organizes them into an adaptive curriculum based on difficulty. Training progresses from easier to harder formats, enabling the model to learn from problems that previously yielded zero signal under standard RL post-training. Cog-DRIFT not only improves on the originally unsolvable hard problems (absolute +10.11% for Qwen and +8.64% for Llama) but also generalizes well to other held-out datasets. Across 2 models and 6 reasoning benchmarks, our method consistently outperforms standard GRPO and strong guided-exploration baselines. On average, Cog-DRIFT shows +4.72% (Qwen) and +3.23% (Llama) improvements over the second-best baseline. We further show that Cog-DRIFT improves pass@k at test time, and the curriculum improves sample efficiency. Overall, our results highlight task reformulation and curriculum learning as an effective paradigm for overcoming the exploration barrier in LLM post-training.
Engineering Breakdown
Plain English
This paper tackles a fundamental problem in reinforcement learning from verifiable rewards (RLVR): LLMs can't learn from problems that are too hard to solve with their current policy because there's no reward signal. The authors propose task reformulation—converting difficult open-ended problems into simpler variants like multiple-choice or cloze formats that preserve the correct answer but reduce search complexity. The key insight is that models can bootstrap learning from these easier structured formats, then transfer that knowledge back to harder generative tasks, enabling learning from problems previously considered intractable.
Core Technical Contribution
The core novelty is a task reformulation framework that transforms hard reasoning problems into a spectrum of easier cognitive variants while maintaining semantic equivalence to the original problem. Unlike prior RLVR approaches that are bottlenecked by unsolved problems yielding zero reward, this method creates intermediate learning signals through discriminative-to-generative task progressions. The authors discovered that bootstrapping on reformulated tasks (multiple-choice → cloze → open-ended) provides dense gradient signals that transfer back to the harder original task, effectively lowering the effective policy requirement needed to learn.
How It Works
The system takes a hard reasoning problem (e.g., a complex math question) and automatically creates multiple reformulated variants: (1) discriminative tasks that narrow the search space (multiple-choice with plausible distractors), (2) intermediate cloze-style tasks that fill in reasoning steps, and (3) generative open-ended tasks closer to the original. During training, the policy first learns from the easiest reformulated variant, which generates learning signals even when the original policy fails. These learned representations and reasoning patterns then transfer to progressively harder variants, culminating in the original open-ended problem. The training loop cycles through this spectrum, with earlier task successes bootstrapping policy quality for harder tasks, creating a curriculum where previous reformulation successes inform harder task learning.
Production Impact
In production RL systems for LLMs, this approach directly solves the 'reward starvation' problem where hard problems yield no training signal, effectively doubling or tripling the fraction of training data that generates meaningful gradients. Engineers would add an offline task reformulation component before the RL pipeline—given a set of hard problems, automatically generate multiple-choice and cloze variants, then interleave them in training batches. The tradeoff is moderate: task reformulation adds ~10-20% compute overhead for variant generation (typically done once via template-based or LLM-guided approaches), but this is recovered through higher sample efficiency and faster policy convergence. Integration into existing RLVR systems (like those using verifiable steps in reasoning) is straightforward—it's essentially curriculum learning with derived tasks, compatible with standard PPO or other RL algorithms.
Limitations and When Not to Use This
The approach assumes that reformulated variants genuinely preserve the reasoning required for the original task—if reformulation oversimplifies (e.g., multiple-choice with obvious answers), transfer back to open-ended reasoning may be brittle. The paper doesn't address when reformulation fails: for some problem types (creative or novel reasoning), reducing to discriminative formats may remove essential difficulty. Scalability of automatic reformulation isn't clear—the method likely works well for math/QA where format variants are natural, but generalizing to domains like code generation or scientific reasoning needs validation. The paper appears incomplete (abstract cuts off), so details on transfer mechanisms, empirical results on different problem classes, and comparison to alternative curriculum strategies are unavailable.
Research Context
This work extends the RLVR framework (which learns from problems with verifiable solutions, unlike RLHF) by addressing a known limitation: policies get stuck on hard problems with zero reward. It builds on curriculum learning research showing that easy-to-hard task progressions improve learning, and connects to work on problem reformulation in automated reasoning. The contribution fits into the broader trend of improving LLM reasoning beyond chain-of-thought by providing denser RL signals—sitting between simple verifiable rewards and complex hierarchical RL approaches. This opens research into automatic task reformulation strategies, transfer learning between task variants, and the theoretical question of what problem properties enable effective reformulation-based bootstrapping.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::