What do Language Models Learn and When? The Implicit Curriculum Hypothesis
| Authors | Emmy Liu et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.08510 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent (ρ= .81 across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining (R^2 = .68-.84 across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.
Engineering Breakdown
Plain English
This paper investigates how large language models acquire different skills during pretraining by proposing the Implicit Curriculum Hypothesis—the idea that models learn capabilities in a predictable, compositional order regardless of scale or data mixture. The authors designed a suite of composable benchmark tasks spanning retrieval, morphology, coreference resolution, logical reasoning, and math, then tracked when four model families (ranging from 410M to 13B parameters) reached performance thresholds on each task. They discovered that emergence orderings are consistent across models and data configurations, suggesting that pretraining follows an implicit curriculum rather than learning randomly. This finding challenges the black-box nature of LLM training and provides a framework for predicting what capabilities emerge when, which has direct implications for model selection, data curation, and compute allocation.
Core Technical Contribution
The core contribution is the Implicit Curriculum Hypothesis itself—a testable, empirical claim that large language models follow a compositional and predictable learning trajectory during pretraining, separate from and more granular than aggregate scaling laws. Rather than measuring only aggregate validation loss (which obscures what specific skills improve), the authors systematically decompose learning into fine-grained task emergence points and show these orderings are reproducible across different model sizes and data mixtures. This shifts the focus from 'how much does performance improve' (scaling laws) to 'what order do capabilities emerge' (skill curriculum), enabling mechanistic understanding of capability emergence. The technical novelty lies in designing composable, interpretable tasks that isolate specific linguistic and reasoning phenomena and then using them as diagnostic probes to create a temporal map of learning.
How It Works
The methodology builds task emergence maps by training model families of different sizes on the same pretraining data and periodically evaluating checkpoints against a suite of carefully designed diagnostic tasks. Each task is composed to test a specific capability: retrieval tests whether models store and access factual information, morphological transformations test grammatical understanding, coreference resolution tests entity tracking across sentences, logical reasoning tests compositional inference, and mathematics tests symbolic manipulation. For each task, the researchers record the training step or compute budget at which the model crosses a performance threshold (e.g., 90% accuracy), treating this as the 'emergence point.' By aggregating emergence points across model sizes (410M, 1B, 7B, 13B parameters) and data mixtures, they identify consistent ordering patterns—for instance, whether models reliably master retrieval before coreference or logical reasoning before arithmetic. The output is a curriculum map showing dependencies and temporal ordering of skill acquisition, essentially creating a directed graph of capability emergence.
Production Impact
This research directly impacts model development decisions: engineers can use the curriculum map to predict approximately when a model will acquire critical capabilities, enabling better compute budgeting and model selection for specific downstream tasks. If a production application requires logical reasoning but not complex mathematics, you can estimate the minimum training compute needed rather than scaling to full pretraining, potentially saving 30-40% of training costs. The framework also informs data curation strategies—if you want to accelerate emergence of specific skills, understanding the implicit curriculum reveals which foundational capabilities must be learned first, allowing you to structure training data or use curricula that follow the model's natural learning order. However, the practical trade-off is that the diagnostic task suite itself requires careful engineering and domain expertise to design well; hasty or poorly-composed probes may give misleading emergence signals. Additionally, this approach requires dense checkpoint evaluation (tracking models at many training steps), which adds storage and computational overhead compared to standard training pipelines.
Limitations and When Not to Use This
The paper's task suite, while carefully designed, may not capture all emergent capabilities—particularly meta-reasoning, instruction-following, and in-context learning behaviors that are critical for modern LLM applications but harder to isolate in simple composable benchmarks. The curriculum emergence orderings are identified post-hoc through empirical observation rather than derived from first principles, so they may be artifacts of the specific data mixture, tokenization, or optimization algorithms used rather than fundamental properties of language learning. The findings are limited to the four model families tested (likely within a similar architectural family, likely Transformer-based); it's unclear whether the same curriculum holds for fundamentally different architectures (mixture-of-experts, recurrent models) or training regimes (reinforcement learning from human feedback, continual learning). Finally, the paper does not explain why this curriculum emerges or provide mechanistic insights into the underlying learning dynamics, leaving open the question of whether the curriculum is optimal or if alternative training approaches could reorder emergence in beneficial ways.
Research Context
This work builds on scaling law research (particularly Kaplan et al., Hoffmann et al.) by asking a finer-grained question: while scaling laws tell us that performance improves predictably with compute, this paper reveals the compositional structure hidden beneath aggregate metrics. It relates to curriculum learning literature (Bengio et al., 2009; more recent work on self-paced and hard-example mining) but inverts the framing—instead of designing an explicit curriculum to speed training, it identifies an implicit curriculum that emerges naturally in language model pretraining. The diagnostic task evaluation approach parallels recent probing work (Hewitt & Liang, Conneau et al.) that interprets what representations models learn, but applies it temporally to track when capabilities emerge. This opens a research direction toward predictive pretraining—building models that forecast capability emergence based on early training checkpoints, enabling dynamic model selection and early stopping for specific downstream tasks without full pretraining.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::