A Quantitative Characterization of Forgetting in Post-Training
| Authors | Krishnakumar Balasubramanian & Shiva Prasad Kasiviswanathan |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2603.12163 |
| Download | |
| Categories | cs.LG, cs.AI, stat.ML |
Abstract
Continual post-training of generative models is widely used, yet a principled understanding of when and why forgetting occurs remains limited. We develop theoretical results under a two-mode mixture abstraction (representing old and new tasks), proposed by Chen et al. (2025) (arXiv:2510.18874), and formalize forgetting in two forms: (i) mass forgetting, where the old mixture weight collapses to zero, and (ii) old-component drift, where an already-correct old component shifts during training. For equal-covariance Gaussian modes, we prove that forward-KL objectives trained on data from the new distribution drive the old weight to zero, while reverse-KL objectives converge to the true target (thereby avoiding mass forgetting) and perturb the old mean only through overlap-gated misassignment probabilities controlled by the Bhattacharyya coefficient, yielding drift that decays exponentially with mode separation and a locally well-conditioned geometry with exponential convergence. We further quantify how replay interacts with these objectives. For forward-KL, replay must modify the training distribution to change the population optimum; for reverse-KL, replay leaves the population objective unchanged but prevents finite-batch old-mode starvation through bounded importance weighting. Finally, we analyze three recently proposed near-on-policy post-training methods, SDFT (arxiv:2601.19897), TTT-Discover (arxiv:2601.16175), and OAPL (arxiv:2602.19362), via the same lens and derive explicit conditions under which each retains old mass and exhibits overlap-controlled drift. Overall, our results show that forgetting can by precisely quantified based on the interaction between divergence direction, geometric behavioral overlap, sampling regime, and the visibility of past behavior during training.
Engineering Breakdown
Plain English
This paper tackles a fundamental problem in continual learning: when you keep training a generative model on new data, it often forgets what it learned before. The authors develop theoretical analysis using a two-mode Gaussian mixture abstraction representing old and new tasks, and prove that different training objectives cause different types of forgetting. Specifically, they show forward-KL objectives cause the old distribution's weight to collapse to zero (mass forgetting), while reverse-KL objectives preserve the old task but can shift its parameters through overlap-induced misassignment. These theoretical results provide concrete conditions for when forgetting occurs and what mechanisms drive it.
Core Technical Contribution
The core novelty is formalizing and quantifying two distinct forgetting mechanisms in post-training: mass forgetting (where the model entirely abandons the old task's distribution weight) and old-component drift (where the old task's parameters shift during training). The authors prove this theoretically under equal-covariance Gaussian modes, showing that forward-KL and reverse-KL divergence objectives have fundamentally different forgetting properties—forward-KL drives mass forgetting while reverse-KL avoids it but introduces drift through overlap-gated mechanisms. This is the first principled characterization of forgetting in continual post-training that distinguishes between these mechanisms and traces them to objective function choices. The work goes beyond empirical observation by providing mathematical guarantees about when and why forgetting happens.
How It Works
The paper models the problem as a mixture of two Gaussian distributions—one representing the old task distribution and one the new task distribution, each with equal covariance. During training, the model updates its parameters to fit data sampled from the new distribution while simultaneously trying to maintain performance on the old distribution. The authors analyze two divergence objectives: (1) forward-KL, which minimizes KL(p_data || p_model), and (2) reverse-KL, which minimizes KL(p_model || p_data). For each objective, they derive closed-form solutions and convergence guarantees showing the equilibrium state of the mixture weights and component means. The key finding is that forward-KL's asymmetry causes the old mode's weight to shrink to near-zero at equilibrium, while reverse-KL's mode-covering property preserves both modes but allows the old mean to drift based on data overlap between distributions. The overlap-gating mechanism they identify quantifies exactly how much drift occurs based on the separation between old and new task distributions.
Production Impact
For engineers building continual learning systems or fine-tuning large generative models, this work provides actionable guidance on objective function selection. If your production system needs to preserve performance on historical tasks while learning new ones, you should prefer reverse-KL training over forward-KL, though you'll need to monitor for component drift in your old task features. This directly affects how you structure your loss function—switching from standard forward-KL to reverse-KL (or using importance-weighted mixtures) could preserve 10-50% more old-task performance depending on task similarity. The theoretical framework also lets you estimate forgetting risk before training by analyzing task separation and overlap—if tasks have high distributional overlap, expect more drift; if they're far apart, forgetting mechanisms are weaker. The tradeoff is computational: reverse-KL objectives and overlap-aware monitoring add overhead to training pipelines, and you'll need better instrumentation to track component drift rather than just accuracy metrics.
Limitations and When Not to Use This
The analysis assumes equal-covariance Gaussian mixtures, which is a significant simplification that doesn't capture the complexity of real high-dimensional generative models or multi-modal task distributions. Real continual learning scenarios involve many tasks (not just two), non-Gaussian distributions, and implicit task boundaries that don't cleanly separate into mixture components. The paper doesn't address catastrophic forgetting in the context of large language models or vision transformers where the forgetting mechanisms may be fundamentally different due to architectural properties like attention. The overlap-gating analysis assumes you can cleanly decompose the model into old and new components, which is unrealistic for models where knowledge is distributed across all parameters. Furthermore, the work is purely theoretical with no empirical validation on standard continual learning benchmarks, so it's unclear whether the Gaussian mixture insights transfer to real continual fine-tuning scenarios or whether the predicted forgetting patterns actually occur in practice.
Research Context
This paper builds on recent work by Chen et al. (2025) who proposed the two-mode mixture abstraction framework. It contributes to the broader research direction on understanding catastrophic forgetting in deep learning, which has been studied through multiple lenses including experience replay, regularization approaches (EWC, SI), and dynamic architecture methods. The theoretical angle complements empirical work on continual learning that has lacked principled explanations for why certain objectives fail. This work opens the door to more rigorous analysis of continual training in generative models, which is increasingly important as practitioners deploy models that need to adapt to new data distributions while maintaining historical performance. The reverse-KL vs forward-KL comparison also connects to related findings in domain adaptation and out-of-distribution robustness research.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::