GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification
| Authors | Wangjie Gan et al. |
| Year | 2026 |
| HF Upvotes | 23 |
| arXiv | 2604.14258 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language models are typically post-trained using supervised fine-tuning (SFT) and reinforcement learning (RL), yet effectively unifying efficient knowledge injection with robust generalization remains challenging. In this work, we provide a training-dynamics analysis showing that SFT can be interpreted as a special case of policy gradient optimization with an extremely sparse implicit reward and unstable inverse-probability weighting, which together lead to single-path dependency, entropy collapse, and gradient explosion. Motivated by this diagnosis, we propose Group Fine-Tuning (GFT), a unified post-training framework that addresses these intrinsic limitations through two mechanisms: Group Advantage Learning, which constructs diverse response groups and derives normalized contrastive supervision to alleviate reward sparsity, and Dynamic Coefficient Rectification, which adaptively bounds inverse-probability weights to stabilize optimization while preserving efficient knowledge injection. Experiments demonstrate that GFT consistently surpasses SFT-based methods and yields policies that integrate more smoothly with subsequent RL training.
Engineering Breakdown
Plain English
This paper diagnoses fundamental instabilities in how large language models are currently post-trained using supervised fine-tuning (SFT) and reinforcement learning (RL). The authors prove mathematically that SFT acts like policy gradient optimization with an extremely sparse implicit reward, causing problems like entropy collapse (model outputs become repetitive), gradient explosion (training becomes unstable), and single-path dependency (overfitting to one response pattern). They propose Group Fine-Tuning (GFT), a unified framework that replaces these instabilities with Group Advantage Learning—a mechanism that constructs diverse response groups and uses normalized contrastive learning to stabilize training while maintaining knowledge injection efficiency.
Core Technical Contribution
The key novelty is reframing SFT through the lens of policy gradient optimization to expose its hidden pathologies: sparse implicit rewards and unstable inverse-probability weighting. This theoretical insight directly motivates Group Fine-Tuning, which unifies SFT and RL under one framework by constructing response groups (multiple completions per prompt) and deriving normalized contrastive advantages. Unlike prior approaches that treat SFT and RL as separate post-training stages, GFT addresses the root causes—entropy collapse and gradient explosion—through a principled mechanism that balances knowledge injection with generalization robustness across diverse response distributions.
How It Works
Group Fine-Tuning starts by generating multiple response candidates (a 'group') for each training prompt, creating diverse outputs instead of committing to a single path. For each group, the method computes advantages—estimates of how much better each response is relative to others—using normalized contrastive learning, which prevents the numerical instabilities of traditional inverse-probability weighting. These advantages are then used in a policy gradient update that encourages the model to increase probability on high-advantage responses while decreasing probability on low-advantage ones. The normalization step (dividing by group statistics) keeps gradient magnitudes stable and prevents entropy collapse by maintaining diversity pressure across the response distribution, rather than collapsing all probability mass onto one path as SFT does.
Production Impact
Adopting GFT would change how teams post-train LLMs: instead of running separate SFT then RL pipelines (which requires two different training loops, RL infrastructure, and separate hyperparameter tuning), you'd run one unified training process. This simplifies engineering—fewer moving parts, single codebase path, easier debugging. The benefit is more stable training (no gradient explosions mid-training) and better generalization (responses remain diverse instead of collapsing to repetitive patterns). Trade-offs: GFT requires sampling multiple responses per prompt during training (higher compute cost, maybe 2-4x depending on group size), and the normalized contrastive mechanism adds overhead. For large-scale operations, the stability gains likely justify the 20-30% compute increase, especially if it reduces failed training runs and improves final model quality.
Limitations and When Not to Use This
The paper doesn't provide empirical validation on standard benchmarks—no numbers comparing GFT to SFT+RL on MMLU, GSM8K, or other evals, which means we don't know if the theoretical elegance translates to real model improvements. The assumption that group diversity actually prevents generalization failure may not hold for all domains: in safety-critical applications, you might want entropy collapse to force consistent behavior, not diversity. The method requires careful tuning of group size and normalization temperature; set these wrong and you lose the stability benefits. Finally, the paper appears incomplete (abstract cuts off mid-sentence: 'normalized contrastiv'), so critical implementation details are missing—exact algorithm pseudocode, convergence proofs, and ablations on which component (group advantage vs. normalization) actually fixes which pathology.
Research Context
This work extends the policy gradient interpretation of SFT (prior work by Ziegler et al., Ouyang et al. on RLHF) by making the pathological case explicit: showing that standard SFT has degenerate reward structure that causes training collapse. It fits into the broader movement toward unified post-training frameworks (similar to direct preference optimization and other recent work moving away from separate SFT→RL pipelines). The theoretical lens—interpreting supervised learning as biased policy optimization—connects to importance sampling and off-policy learning literature. The work opens a direction for diagnosing other training instabilities through policy gradient optics and designing normalized contrastive mechanisms for stable LLM training.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::