Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability
| Authors | Qihan Ren et al. |
| Year | 2026 |
| HF Upvotes | 304 |
| arXiv | 2604.06628 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
A prevailing narrative in LLM post-training holds that supervised finetuning (SFT) memorizes while reinforcement learning (RL) generalizes. We revisit this claim for reasoning SFT with long chain-of-thought (CoT) supervision and find that cross-domain generalization is not absent but conditional, jointly shaped by optimization dynamics, training data, and base-model capability. Some reported failures are under-optimization artifacts: cross-domain performance first degrades before recovering and improving with extended training (a dip-and-recovery pattern), so shorttraining checkpoints can underestimate generalization. Data quality and structure both matter: low-quality solutions broadly hurt generalization,while verified long-CoT traces yield consistent cross-domain gains. Model capability is essential: stronger models internalize transferable procedural patterns (e.g., backtracking) even from a toy arithmetic game, while weaker ones imitate surface verbosity. This generalization is asymmetric, however: reasoning improves while safety degrades, reframing the question from whether reasoning SFT generalizes to under what conditions and at what cost.
Engineering Breakdown
Plain English
This paper challenges the conventional wisdom that supervised finetuning (SFT) on reasoning tasks only memorizes while reinforcement learning generalizes to new domains. The authors trained language models on long chain-of-thought (CoT) reasoning supervision and tracked cross-domain generalization carefully, finding that generalization is conditional—it depends on optimization dynamics, data quality, and base model capability. They discovered a critical dip-and-recovery pattern: performance initially degrades on out-of-domain tasks before recovering and improving with extended training, which means short training runs can falsely suggest models don't generalize. The key finding is that verified high-quality long-CoT traces consistently produce cross-domain gains, challenging the narrative that SFT memorizes while RL generalizes.
Core Technical Contribution
The core contribution is empirical evidence that SFT with reasoning supervision can achieve conditional cross-domain generalization, contrary to the prevailing narrative in post-training research. The authors identify three critical factors shaping generalization: (1) optimization dynamics exhibiting a dip-and-recovery pattern that requires careful checkpoint selection, (2) data quality where verified solutions matter significantly more than unverified ones, and (3) base model capability as a prerequisite. This reframes the SFT-vs-RL debate from a binary "memorization vs generalization" dichotomy into a more nuanced understanding where SFT can generalize when conditions align. The novelty lies in systematically measuring these conditioning factors and showing that prior conclusions about SFT were actually underestimation artifacts from premature evaluation.
How It Works
The experimental methodology involves training LLMs on long chain-of-thought reasoning tasks with different supervision quality levels, then evaluating generalization by testing on held-out domains not seen during training. The authors track the same model checkpoint across training iterations to observe the dip-and-recovery phenomenon: initially, as the model optimizes on the training domain, performance on out-of-domain tasks drops (the dip), then recovers and exceeds initial performance with continued training (the recovery phase). They systematically vary data quality by comparing verified solutions (correct reasoning paths) versus low-quality or unverified solutions, measuring how this impacts both in-domain fit and cross-domain generalization. The key metric is cross-domain performance as a function of training steps, model size, and solution quality, revealing that generalization is not absent but conditional on reaching sufficient training iterations where the model has internalized generalizable reasoning patterns rather than memorizing training specifics.
Production Impact
For production systems, this paper suggests that SFT-based reasoning fine-tuning can achieve strong cross-domain transfer without requiring RL, which has significant operational implications: SFT is simpler to implement (no reward model, no on-policy sampling, lower infrastructure complexity), more stable to train (no RL variance), and faster to iterate on. The dip-and-recovery pattern means practitioners must invest in proper checkpoint selection and validation on held-out domains rather than stopping at the first performance plateau—a shift in evaluation discipline. Data quality becomes the critical lever: investing in verified, high-quality long-CoT traces (via human annotation or synthetic verification) yields consistent generalization benefits, making data curation a primary production cost. However, this approach requires identifying the recovery phase correctly, which demands careful validation infrastructure and computation budget to run full training cycles before deployment decisions, adding engineering overhead compared to shorter fine-tuning runs.
Limitations and When Not to Use This
The paper's scope is limited to reasoning tasks with long chain-of-thought supervision—it's unclear whether these findings generalize to other SFT domains like instruction following, summarization, or creative tasks where reasoning patterns may not drive generalization. The dip-and-recovery pattern's timing and severity likely depend on specific dataset characteristics, model architecture, and task difficulty, but the paper doesn't provide clear guidance on predicting these dynamics for new tasks. The work assumes access to verified long-CoT traces or high-quality solutions, which is a significant practical constraint in domains where generating and verifying correct reasoning paths is expensive or difficult. Additionally, the paper doesn't deeply explore the mechanisms driving the dip-and-recovery pattern—it identifies the phenomenon empirically but the theoretical understanding of why this occurs remains incomplete, limiting practitioners' ability to predict or control it.
Research Context
This work directly challenges a prevailing narrative in LLM post-training research established by recent work emphasizing RL-based fine-tuning (like RLHF and variants) as the path to generalization. It builds on foundational work in chain-of-thought prompting and reasoning supervision, extending those ideas with careful empirical analysis of generalization dynamics. The paper fits into a broader research direction questioning simplistic dichotomies in post-training (SFT vs RL, memorization vs generalization) in favor of more nuanced, conditional understanding. It opens research directions into understanding optimization dynamics in reasoning tasks, predicting dip-and-recovery patterns, and potentially designing training procedures that accelerate the recovery phase or eliminate the dip altogether.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::