How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum
| Authors | Chu-Cheng Lin & Eugene Ie |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.25907 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Adapting reasoning models to new tasks during post-training with only output-level supervision stalls under reinforcement learning from verifiable rewards (RLVR) when the initial success probability p_0 is small. Using the Tsallis q-logarithm, we define a loss family J_Q that interpolates between RLVR (at q{=}0, the exploitation pole) and the log-marginal-likelihood over latent trajectories (at q{=}1, the density-estimation pole). All members share the same per-example gradient direction, differing only by a scalar amplification P_{θ^{-q}} that reweights each instance independently of the learning rate. This amplification is the mechanism that addresses cold-start stalling: under gradient flow, the exploitation pole requires Ω(1{p_0}) time to escape cold start, while the density-estimation pole escapes in Θbig(log(1{p_0})big); intermediate q trades escape speed against noise memorization. Because P_θ is intractable, we derive two Monte Carlo estimators from the two factorizations of the gradient: Gradient-Amplified RL (GARL) samples from the prior and amplifies the RL gradient, and Posterior-Attenuated Fine-Tuning (PAFT) importance-resamples from the posterior and runs standard SFT. Both have bias Obig(q{M P_θ^{q+1}}big); GARL has lower variance, PAFT has semantically coherent gradients. On FinQA, HotPotQA, and MuSiQue, GARL at q{=}0.75 substantially mitigates cold-start stalling, escaping cold start where GRPO fails entirely. In warm start, GARL at low q dominates FinQA where training is stable; on HotPotQA and MuSiQue, GARL destabilizes during training, and PAFT at q{=}0.75 provides stable gradients (best overall on HotPotQA at 47.9 maj@16, +14.4 over GRPO).
Engineering Breakdown
Plain English
This paper addresses a critical failure mode in reinforcement learning for reasoning models: when initial task success rates are very low (small p_0), standard RL approaches stall during post-training. The authors propose a new loss family J_Q based on the Tsallis q-logarithm that smoothly interpolates between pure exploitation (q=0, standard RLVR) and density estimation over latent reasoning trajectories (q=1). The key innovation is a per-example reweighting factor P_θ^{-q} that amplifies gradients for harder examples without requiring learning rate changes. Under theoretical analysis, this approach reduces cold-start convergence time from Ω(1/p_0)—which becomes prohibitive when success is rare—to a tractable regime.
Core Technical Contribution
The core novelty is the Tsallis q-logarithm loss family that unifies two seemingly incompatible objectives: reward maximization (RLVR) and marginalized trajectory likelihood (standard supervised learning). Unlike prior interpolation schemes that require tuning multiple hyperparameters, all members of J_Q share identical per-example gradient directions, differing only by an adaptive scalar amplification that reweights instances based on model confidence (θ^{-q}). This mechanism directly addresses the cold-start problem in reasoning model adaptation by providing automatic curriculum-like behavior—harder examples get amplified gradients—without explicit curriculum engineering. The theoretical analysis shows this reweighting reduces the fundamental convergence bottleneck from inverse-probability dependence to a manageable regime.
How It Works
The method takes a base reasoning model and a verifiable reward signal (e.g., correctness of final answer) as input. During post-training, the loss J_Q(θ; q) is computed as a weighted expectation over both successful and failed rollouts, where the weighting depends on the Tsallis q-logarithm parameter. The gradient computation ∇J_Q has a special property: the direction (which examples matter most) stays constant across all q values, but each gradient is multiplied by P_θ^{-q}, where P_θ is the model's predicted success probability and q controls the amplification strength. Lower q values (toward 0) emphasize exploitation of known good trajectories; higher q values (toward 1) encourage broader exploration of the latent trajectory space. The model learns via gradient flow on this composite loss, with the reweighting mechanism automatically increasing gradient signal for low-probability examples—exactly where learning matters most during cold-start.
Production Impact
For teams deploying reasoning models (code generation, math solving, planning) on new specialized tasks, this approach directly solves the problem of adaptation when task success starts near zero. Instead of waiting through exponentially slow initial learning or needing hand-curated seed solutions, you can apply this loss and see convergence in realistic timescales. The implementation is straightforward—swap your RL loss function to J_Q and tune a single scalar q value, no architectural changes needed. Trade-offs are modest: you'll compute model confidence scores P_θ during training (negligible overhead), and you gain the benefit of automatic curriculum effects without explicit negative mining or example reweighting schemes. For production systems, this is particularly valuable when bootstrapping reasoning models on enterprise-specific tasks (legal document analysis, domain-specific theorem proving) where you can verify correctness but initial success is rare.
Limitations and When Not to Use This
The paper's theoretical results assume gradient flow (continuous updates), which doesn't perfectly match SGD with finite batch sizes and learning rates used in production—the convergence guarantees may be looser in practice. The approach requires a well-defined verifiable reward (ground truth answers), limiting applicability to tasks without clear correctness criteria or where verification is expensive. The Tsallis q-logarithm formulation, while elegant, introduces a hyperparameter q that still needs tuning; the paper doesn't provide principled selection criteria for different task regimes. Importantly, the method addresses cold-start stalling but doesn't fundamentally change sample complexity—you still need sufficient diverse trajectories to avoid distribution shift, and the theory doesn't account for model capacity limitations or exploration-exploitation tradeoffs in the intermediate regime where success probability is neither negligible nor close to one.
Research Context
This work builds directly on recent progress in reinforcement learning from verifiable rewards (RLVR) for reasoning models—specifically addressing the identified failure mode when initial success probability is low. It connects to the broader literature on interpolation between RL objectives (e.g., KL-regularized RL, offline RL) but achieves it through information-geometric properties of the Tsallis entropy rather than explicit regularization. The contribution is particularly relevant to recent advances in long-chain-of-thought models and their adaptation to downstream tasks, where cold-start failure is a known bottleneck. The Tsallis q-logarithm perspective opens a new direction for understanding curriculum effects and confidence-based reweighting from first principles, potentially applicable to other low-data regime problems in reinforcement learning beyond reasoning model post-training.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::