Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
| Authors | Ansar Aynetdinov et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.28075 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency. However, for high-resource non-English languages like German, French, or Japanese, aggressive filtering creates a strategic dilemma: should practitioners prioritize diversity by training once on large amounts of lightly filtered web data, or prioritize quality by strictly filtering for a high-quality core and repeating it over multiple epochs? We investigate this trade-off for German by constructing hierarchical quality filters applied to 500M web documents, comparing multi-epoch training on the filtered subsets against single-pass training on a diverse corpus. Our experiments across multiple model scales and token budgets show that repeating high-quality data consistently outperforms single-pass training on larger, less filtered sets. Notably, the performance gap persists even after 7 epochs. Our findings suggest that for non-English LLMs, semantic concentration through quality filtering offers a more viable path to efficient language modeling than simply maximizing unique data volume. We release our German language models (called Boldt), as well as our cleaned evaluation benchmarks to the research community. Our experiments indicate that they achieve state-of-the-art results despite training on 10-360x fewer tokens than comparable models.
Engineering Breakdown
Plain English
This paper investigates a fundamental trade-off in training language models on high-resource non-English languages like German: should you train once on a large diverse corpus with light filtering, or train multiple epochs on a smaller high-quality filtered subset? The authors built hierarchical quality filters for 500M German web documents and compared single-pass diverse training against multi-epoch training on filtered data across various model scales and token budgets. Their experiments reveal that repeating high-quality data outperforms training once on larger diverse corpora, challenging the assumption that diversity always beats quality for non-English languages.
Core Technical Contribution
The core contribution is a systematic empirical framework for quantifying the quality-versus-diversity trade-off in language model training for high-resource non-English languages. Rather than assuming that more diverse data is always better (the intuition from English web corpus research), the authors demonstrate through controlled experiments that for German, aggressive quality filtering combined with repeated epochs actually outperforms single-pass training on much larger unfiltered data. This inverts the typical scaling assumption and provides a practical decision framework with hierarchical filtering as the key technical mechanism—allowing practitioners to construct quality tiers and measure the efficiency gains from repetition versus diversity.
How It Works
The approach starts with 500M raw German web documents as input and applies a hierarchical quality filtering pipeline that progressively narrows the corpus into higher-quality subsets (likely using criteria like language detection, duplicate removal, toxicity filtering, and linguistic quality metrics). The authors then conduct parallel experiments: one track trains a language model once on the full lightly-filtered diverse corpus, while another track trains the same model architecture multiple times (repeated epochs) on progressively smaller but higher-quality filtered subsets. Both tracks operate under the same token budget (total training compute), so repetition trades width for depth. The experiments sweep across multiple model sizes and token budgets to isolate the effect of quality filtering and epoch repetition, measuring final downstream task performance to determine which strategy wins.
Production Impact
For teams training non-English language models, this paper directly changes data engineering decisions: instead of aggressively pursuing scale and diversity (which requires expensive crawling, deduplication, and storage), you can instead invest in quality filtering and plan to repeat epochs—reducing data infrastructure complexity and storage costs while potentially improving convergence. In production pipelines, this means you'd implement tier-based filtering (building quality classifier models and heuristics) and tune repetition schedules per language and model size, rather than building increasingly massive data warehouses. The trade-off is that you need predictable compute budgets and must commit to repeated training runs rather than one-shot training, which affects iteration speed and hardware utilization planning. For multilingual systems, this suggests allocating data budgets differently per language based on corpus size and quality availability.
Limitations and When Not to Use This
The paper focuses only on German (one language family, one language context), so the generalizability to other high-resource languages like Japanese, Chinese, or Arabic remains unclear—filtering quality metrics are often language-specific, and the trade-off might differ substantially. The approach assumes you have reliable hierarchical quality filters that work for your language, which is a strong assumption for truly high-quality classification; if your filters are noisy or biased, repetition could amplify those errors more than diverse single-pass training would. The paper doesn't address the computational cost of building those quality filters or the effort required to validate filter quality across different language domains. Additionally, the comparison doesn't include more recent techniques like instruction tuning, reinforcement learning feedback, or mixture-of-experts architectures that might interact differently with diversity and repetition.
Research Context
This work extends recent findings from English web corpus research (where aggressive filtering improves efficiency) by asking whether those conclusions hold for non-English languages with different corpus characteristics and quality distributions. It builds on the scaling law literature (Chinchilla, Kaplan et al.) that established token budgets as the key constraint, but questions the implicit assumption that diverse tokens are always preferred. The paper sits at the intersection of data curation work (like C4, Common Crawl quality studies) and curriculum learning research, suggesting that repetition with quality filtering is a form of curriculum. This opens a new research direction: language-specific data efficiency, where the optimal training strategy depends on language properties and corpus characteristics, rather than a one-size-fits-all approach.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::