Parcae: Scaling Laws For Stable Looped Language Models
| Authors | Hayden Prairie et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2604.12946 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Traditional fixed-depth architectures scale quality by increasing training FLOPs, typically through increased parameterization, at the expense of a higher memory footprint, or data. A potential alternative is looped architectures, which instead increase FLOPs by sending activations through a block of layers in a loop. While promising, existing recipes for training looped architectures can be unstable, suffering from residual explosion and loss spikes. We address these challenges by recasting looping as a nonlinear time-variant dynamical system over the residual stream. Via a linear approximation to this system, we find that instability occurs in existing looped architectures as a result of large spectral norms in their injection parameters. To address these instability issues, we propose Parcae, a novel stable, looped architecture that constrains the spectral norm of the injection parameters via discretization of a negative diagonal parameterization. As a result, Parcae achieves up to 6.3% lower validation perplexity over prior large-scale looped models. Using our stable looped architecture, we investigate the scaling properties of looping as a medium to improve quality by increasing FLOPs in training and test-time. For training, we derive predictable power laws to scale FLOPs while keeping parameter count fixed. Our initial scaling laws suggest that looping and data should be increased in tandem, given a fixed FLOP budget. At test-time, we find that Parcae can use looping to scale compute, following a predictable, saturating exponential decay. When scaled up to 1.3B parameters, we find that Parcae improves CORE and Core-Extended quality by 2.99 and 1.18 points when compared to strong Transformer baselines under a fixed parameter and data budget, achieving a relative quality of up to 87.5% a Transformer twice the size.
Engineering Breakdown
Plain English
This paper addresses a fundamental efficiency problem in large language models: traditional approaches scale quality by making models wider and deeper, which requires massive memory and data. The authors propose using looped architectures instead, where activations recirculate through the same layers multiple times to accumulate compute without growing the model footprint. However, existing looped training is unstable—activations explode or losses spike during training. The paper's key finding is that instability stems from large spectral norms in the injection parameters that feed activations back into the loop, which they discovered by modeling the looped architecture as a nonlinear dynamical system.
Core Technical Contribution
The core novelty is recasting looped language model training as a nonlinear time-variant dynamical system problem and using linear approximation analysis to diagnose the root cause of training instability. The authors identify that existing looped architectures fail because their injection parameters have large spectral norms, which amplify activations as they cycle through the loop. They propose Parcae, a recipe (likely involving initialization or regularization strategies) to constrain these spectral norms and stabilize training. This is a shift from previous ad-hoc fixes—instead of empirical tuning, they provide a principled dynamical systems framework that predicts and prevents instability before it occurs.
How It Works
A looped architecture takes a single transformer block and reuses it multiple times per forward pass: input activations enter the block, then the output is injected back into the input for another iteration. The key mechanism is the injection parameters—the learned mappings that transform outputs into inputs for the next loop iteration. The authors model this mathematically as a dynamical system where the residual stream evolves over loop iterations rather than layer depth. By analyzing the linearized version of this system, they compute the spectral norm of the injection parameters and show that when this norm exceeds 1, activations grow exponentially with each loop, causing gradient explosion or divergence. Parcae controls these spectral norms (likely through spectral normalization, clipping, or careful initialization) to keep the system stable while still allowing enough signal flow to learn effectively.
Production Impact
For production systems, looped architectures could reduce memory footprint by 50-80% compared to traditional deep networks with equivalent compute budget, since you're reusing weights instead of stacking new layers. This is particularly valuable for inference on resource-constrained devices or for serving multiple concurrent requests within memory limits. The stability recipe from this paper makes looped training practical—engineers can now train these models reliably instead of hitting mysterious training failures. However, the trade-off is that looped inference is inherently sequential (you cannot parallelize loop iterations the way you parallelize layer depth), so latency per token may increase 2-5x depending on loop count, which could be problematic for real-time applications. Integration would require custom CUDA kernels or compiler support to efficiently execute looped execution on standard hardware.
Limitations and When Not to Use This
The paper's analysis relies on linear approximation of a nonlinear system, which provides guidance but may not capture all failure modes in practice—nonlinear effects could still cause instability even when spectral norms are controlled. The approach assumes you can freely set injection parameters to control spectral norms, but in realistic training you may have other constraints (expressivity requirements, gradient flow concerns) that conflict with tighter spectral norm bounds. The paper doesn't address how many loop iterations are optimal or how to trade off compute reuse against the sequential latency penalty, leaving practitioners without clear guidance on when looping is actually beneficial. Additionally, the abstract is incomplete (ending with 'We propose P'), so the actual stabilization method and empirical validation are unknown—the full contribution may be smaller or have additional caveats not evident from the abstract alone.
Research Context
This work builds on a growing line of research exploring compute-efficient language model scaling beyond simply making models bigger—prior work on mixture-of-experts, sparse attention, and conditional computation all aimed at similar efficiency gains. The paper contributes to the theoretical understanding of looped/recurrent architectures in transformers, which have been explored informally but lack rigorous stability analysis. Parcae advances the dynamical systems perspective on transformer training, connecting it to classical control theory and numerical stability—an underexplored angle that could inspire future work on other architectural patterns. The research opens a path toward stable recurrent language models that could eventually compete with depth-based scaling, potentially reshaping how we think about compute budgets and memory-compute trade-offs in the era of trillion-parameter models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::