ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement
| Authors | Difan Jiao et al. |
| Year | 2026 |
| HF Upvotes | 38 |
| arXiv | 2604.01591 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.
Engineering Breakdown
Plain English
ThinkTwice is a two-phase training framework that teaches language models to both solve reasoning problems and then refine their own solutions, using a reinforcement learning approach called Group Relative Policy Optimization (GRPO). The method trains on mathematical reasoning benchmarks without requiring critique annotations or correctness signals beyond a simple right/wrong reward signal. On the Qwen3-4B model, ThinkTwice achieved a 5 percentage point improvement over baseline GRPO on the AIME benchmark before refinement, demonstrating that jointly optimizing solving and refinement capabilities produces better mathematical reasoning performance than traditional single-phase approaches.
Core Technical Contribution
The core novelty is a paired training procedure that alternates between two distinct optimization phases: first optimizing the model to solve reasoning problems, then immediately optimizing it to refine those same solutions. Unlike prior work that treats solving and refinement as separate tasks or requires explicit critique data, ThinkTwice uses the identical binary correctness reward in both phases, making the approach more practical and annotation-efficient. This joint optimization within GRPO's policy gradient framework enables models to learn complementary capabilities—initial problem-solving and error-correction—from a single simple reward signal without additional human annotation overhead.
How It Works
The training loop operates in paired steps: (1) In phase one, the model generates solutions to reasoning problems and receives policy gradient updates based on whether answers are correct or incorrect, optimizing the model's ability to solve new problems. (2) In phase two, using the same problem set, the model takes its previously generated solutions as input and generates refined versions, again receiving reward signals only from correctness without any critique or error annotation. (3) This is implemented using GRPO, which groups trajectories and computes policy gradients relative to group baselines, making the optimization more stable and sample-efficient than standard policy gradient methods. (4) The binary reward (correct/incorrect) is the sole training signal; there are no intermediate steps, reasoning scores, or human-written critiques—just end-to-end correctness labels. (5) By alternating these phases across training steps, the model learns to both generate diverse solution approaches and to identify and fix errors within those approaches, each phase reinforcing the other.
Production Impact
Adopting ThinkTwice would streamline the pipeline for building reasoning-capable LLMs by eliminating the need for expensive critique annotations or multi-stage reward modeling. Production teams currently building mathematical reasoning systems would need only binary correctness labels per problem, which are trivial to generate or verify automatically, reducing annotation cost by orders of magnitude compared to RLHF approaches requiring detailed human feedback. The two-phase training could be integrated into existing RL training loops as a drop-in replacement for single-phase GRPO, requiring minimal infrastructure changes and working with both smaller models (Qwen3-4B) and larger ones (Olmo3-7B), making it broadly applicable across model sizes. The latency impact is modest since refinement happens post-training; inference latency would only increase if you actually use the refinement phase at serving time, which is optional. Trade-offs include increased total training compute (running two optimization phases instead of one per step) and slightly longer wall-clock training time, though the improved convergence may offset this.
Limitations and When Not to Use This
The paper focuses exclusively on mathematical reasoning tasks and benchmarks; it's unclear whether the two-phase approach generalizes to other domains like code generation, summarization, or open-ended QA where correctness is harder to define or measure. The approach assumes access to automatic correctness checking, which is feasible for math (where answers are often verifiable) but breaks down for subjective tasks or domains without clear ground truth—this severely limits applicability. The experiments are conducted on relatively small models (up to 7B parameters) and mid-scale benchmarks; scaling properties to 70B+ models and production-scale data are unknown, and the computational overhead of two-phase training may not be justified at very large scales where single-phase methods are already effective. Finally, the paper provides no analysis of what kinds of errors the refinement phase actually corrects or when refinement hurts performance (error amplification), limiting understanding of failure modes that would be critical for production systems.
Research Context
This work builds directly on Group Relative Policy Optimization (GRPO), a recent reinforcement learning technique that stabilizes policy gradients by computing advantages relative to group baselines rather than running expensive value function training. It addresses a known limitation in reasoning-focused LLM training: that single-phase RL often plateaus because the model never explicitly practices error correction on its own outputs. The paper contributes to the broader research direction of sample-efficient RL for LLMs, where methods like DPO and its variants have shown that you can achieve strong results without full RL infrastructure—ThinkTwice follows this trend by using only binary rewards rather than learned reward models. Improvements are demonstrated on standard mathematical reasoning benchmarks (AIME, likely also AMC, GSM8K, MATH, Olympiad problems based on context), which are the canonical evaluation suites for measuring LLM reasoning capability in the research community.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::