Prescriptive Scaling Laws for Data Constrained Training
| Authors | Justin Lovelace et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2605.01640 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay (λ=1.0) reduces this coefficient by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.
Engineering Breakdown
Plain English
This paper addresses a critical shift in modern pretraining: we now have more compute than high-quality unique training data. The authors challenge the widely-used Chinchilla scaling law, which assumes every token is unique and doesn't account for data repetition. They propose a new scaling law that models overfitting from repeated data as a simple additive penalty, and show this accurately predicts model behavior when training data is limited. Their key finding is that beyond a certain point, repeating data and increasing model size leads to worse performance than smaller models trained on less repetition—a counterintuitive result that changes compute allocation strategy.
Core Technical Contribution
The core novelty is a data-aware scaling law that explicitly models the cost of token repetition through an additive overfitting penalty term. Unlike Chinchilla, which treats all tokens as independent, this work captures how repeated data progressively hurts model performance and provides closed-form guidance on compute-optimal allocation under data constraints. The authors derive qualitatively different recommendations: instead of the Chinchilla ratio of model size to data, they show a regime where additional model capacity is wasteful if you must repeat data. This shifts the optimization frontier from a single scaling law to a family of laws parameterized by data budget, enabling prescriptive (rather than descriptive) guidance for practitioners facing real data scarcity.
How It Works
The method starts by observing that when training data is repeated, the loss doesn't scale smoothly as it does with unique tokens—there's additional degradation (overfitting). The authors model total loss as: base loss + overfitting penalty, where the penalty grows as data repetition increases. They fit this model empirically across experiments with varying model sizes, token counts, and repetition factors to extract the functional form of the overfitting penalty (typically a power law). Once calibrated, this loss model becomes a constraint in a compute optimization problem: given a fixed compute budget C and a fixed data quantity D, what combination of model size N and training tokens T minimizes loss, accounting for the fact that tokens beyond D must be repetitions? The solution yields a compute-optimal point that differs from Chinchilla's recommendation, often suggesting smaller models and less aggressive scaling when data is constrained.
Production Impact
For teams pretraining large language or multimodal models, this directly changes resource allocation decisions. If you have 1M high-quality tokens but 10x the compute, the old wisdom (Chinchilla) suggested building a larger model and repeating data; this paper says you should either acquire more data, build a smaller model, or accept that you've hit your performance ceiling without more unique data. In production pipelines, this means revisiting your model size vs. data collection trade-off: investing in data diversity and scale may have higher ROI than scaling compute when you're data-constrained. The paper also enables better budget forecasting—teams can now estimate the diminishing returns of data repetition and plan data collection or synthetic data generation accordingly. The main trade-off is that applying this law requires empirically calibrating the overfitting penalty for your specific data distribution and model class, which adds a preliminary experimental phase before large-scale training.
Limitations and When Not to Use This
The paper assumes that the additive overfitting penalty model holds across different data domains and model architectures, which may not generalize to all scenarios (e.g., synthetic data may have different repetition characteristics than natural text). The approach also doesn't account for curriculum learning, data augmentation, or other techniques that might reduce overfitting from repetition, potentially understating the utility of repeated data in those cases. The paper is fundamentally limited to the pretraining phase and doesn't address how data scarcity in pretraining affects downstream task performance or fine-tuning efficiency. Finally, while the scaling law provides compute-optimal guidance, it doesn't solve the harder problem of actually obtaining high-quality unique data or predicting generalization to out-of-distribution tasks—it only optimizes within your existing data budget.
Research Context
This work builds directly on the Chinchilla scaling law (Hoffmann et al., 2022) and the broader literature on compute-optimal training (Kaplan et al., Hoffmann et al.). It extends scaling laws from the abundance regime (where data is plentiful) to the scarcity regime, a practically important frontier given recent data bottlenecks in large model training. The paper connects to concurrent work on data quality, synthetic data, and the limits of scale in pretraining. It opens a research direction toward dynamic, constraint-aware scaling laws that can be tailored to different budget scenarios, and likely motivates follow-up work on optimal data collection strategies and the interplay between pretraining data scarcity and downstream performance.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::