Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder
| Authors | Götz-Henrik Wiegand et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.09389 |
| Download | |
| Categories | cs.LG, cs.CL |
Abstract
Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30% of the training data is sufficient to reach approximately 90% of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.
Engineering Breakdown
Plain English
This paper investigates how Transformer language models perform at smaller scales and with limited training data, a practical question that's less explored than large-scale scaling laws. The authors built a reduced attention-only decoder architecture and trained it on progressively larger power-of-two subsets of data to isolate dataset-size effects. They found smooth, predictable performance improvements with clear diminishing returns following established scaling-law patterns. A key finding: you can reach 90% of full-model validation accuracy using only 30% of the training data, providing actionable guidance for resource-constrained training scenarios.
Core Technical Contribution
The core contribution is a controlled experimental framework that isolates dataset scaling effects in smaller-scale Transformer training by using a simplified attention-only decoder. Unlike prior scaling law research that focuses on massive models and datasets, this work systematically varies only dataset size while holding other variables constant, providing empirical evidence that scaling laws hold predictably even at smaller scales. The key novelty is demonstrating quantitatively that the 30%-data-to-90%-performance relationship is reproducible and consistent, enabling practitioners to make data-budgeting decisions with confidence. This bridges the gap between theoretical scaling laws and practical, resource-constrained training regimes.
How It Works
The authors design an attention-only decoder architecture (a simplified Transformer variant without feed-forward layers and with minimal parameters) to reduce confounding factors during training. They create power-of-two subsets of their training dataset—for example, 2^10, 2^11, 2^12 tokens—and train independent models on each subset using identical hyperparameters and random seeds. After each training run, they measure validation token-level accuracy (the fraction of tokens correctly predicted on a held-out validation set). By plotting accuracy against dataset size, they observe a smooth, predictable curve showing diminishing returns—early datasets yield steep accuracy gains, while later datasets yield smaller incremental improvements. This curve follows the mathematical form predicted by Chinchilla and similar scaling law papers, confirming that the underlying power-law relationship is robust across scales.
Production Impact
For teams training language models on constrained budgets (smaller companies, on-device models, fine-tuning scenarios), this result provides concrete guidance: you can plan to use ~30% of available data and still hit 90% of peak performance, freeing 70% of training compute for other tasks like hyperparameter tuning, ensemble building, or other models. This directly reduces training time and infrastructure cost without sacrificing model quality meaningfully. In production pipelines, this enables data-efficient strategies: you could train a 30% baseline model quickly for early validation, then scale to full data only after confirming the research direction. The trade-off is explicit and quantified—the remaining 10% performance gap may or may not matter depending on downstream task sensitivity, so teams need to measure task-specific accuracy loss, not just token-level accuracy.
Limitations and When Not to Use This
The paper studies only token-level validation accuracy, which doesn't capture downstream task performance (e.g., question-answering, code generation, summarization), so the 90% number may not translate directly to real-world metrics. The attention-only decoder is a significant simplification of modern production Transformers (no feed-forward layers, reduced depth), so scaling-law exponents and dataset-efficiency curves may differ with larger, denser models. The paper doesn't address data quality, distribution shift, or domain-specific effects—these results assume the dataset is homogeneous and well-shuffled, which rarely holds for real-world text. Open questions remain: Does the 30%-to-90% ratio hold for multi-task learning, long-context models, or fine-tuning from pretrained checkpoints? What happens when data quality varies (clean vs. web-scraped vs. synthetic)?
Research Context
This work builds directly on scaling-law research pioneered by Kaplan et al. (GPT-3 scaling laws) and Hoffmann et al. (Chinchilla compute-optimal training), extending their insights to the smaller-scale regime where most practitioners actually operate. It addresses a practical gap: prior papers derive scaling laws on 10B+ parameter models with trillions of tokens, but most real-world teams train 100M–1B models on billions of tokens. By validating that power-law scaling holds even at reduced scale, this paper strengthens the empirical foundation for scaling-law theory. The work opens a research direction toward dataset-efficient training strategies and could inspire follow-up studies on quality-vs-quantity trade-offs, curriculum learning with scaling laws, or optimal stopping rules.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::