Adam's Law: Textual Frequency Law on Large Language Models
| Authors | Hongyuan Adam Lu et al. |
| Year | 2026 |
| HF Upvotes | 464 |
| arXiv | 2604.02176 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
While textual frequency has been validated as relevant to human cognition in reading speed, its relatedness to Large Language Models (LLMs) is seldom studied. We propose a novel research direction in terms of textual data frequency, which is an understudied topic, to the best of our knowledge. Our framework is composed of three units. First, this paper proposes Textual Frequency Law (TFL), which indicates that frequent textual data should be preferred for LLMs for both prompting and fine-tuning. Since many LLMs are closed-source in their training data, we propose using online resources to estimate the sentence-level frequency. We then utilize an input paraphraser to paraphrase the input into a more frequent textual expression. Next, we propose Textual Frequency Distillation (TFD) by querying LLMs to conduct story completion by further extending the sentences in the datasets, and the resulting corpora are used to adjust the initial estimation. Finally, we propose Curriculum Textual Frequency Training (CTFT) that fine-tunes LLMs in an increasing order of sentence-level frequency. Experiments are conducted on our curated dataset Textual Frequency Paired Dataset (TFPD) on math reasoning, machine translation, commonsense reasoning and agentic tool calling. Results show the effectiveness of our framework.
Engineering Breakdown
Plain English
This paper investigates whether textual frequency—how often word sequences appear in natural language—matters for large language model performance, similar to how it affects human reading speed. The authors propose Textual Frequency Law (TFL), which suggests that LLMs perform better when prompted or fine-tuned with more frequent textual expressions. They build a three-part framework: first estimating sentence-level frequency using online resources, second paraphrasing inputs into more common phrasings, and third applying Textual Frequency Distillation to improve model training. The work is novel because frequency's impact on LLMs has been largely ignored despite being well-studied in human cognition.
Core Technical Contribution
The core novelty is formalizing Textual Frequency Law as a quantifiable principle for LLM optimization—establishing that input text frequency is a measurable lever for both inference and training performance. Unlike prior work focusing on token distribution or vocabulary design, this paper treats sentence-level frequency as an explicit optimization target. The authors introduce Textual Frequency Distillation (TFD), a training technique that mirrors knowledge distillation but optimizes specifically for frequency alignment. This shifts the paradigm from treating all training text equally to preferring higher-frequency natural language expressions, which is a previously unexplored dimension of data quality.
How It Works
The framework operates in three sequential stages. First, the system analyzes input text and estimates its sentence-level frequency by querying large online language corpora (web-crawled datasets, Wikipedia, etc.) to establish baseline frequency scores for linguistic patterns. Second, a paraphraser module automatically rewrites input prompts into semantically equivalent but higher-frequency expressions—for example, converting rare phrasings into more common synonymous constructions. Third, Textual Frequency Distillation applies this frequency signal during model training: the training objective rewards the model for learning from high-frequency text patterns while maintaining task performance, similar to how knowledge distillation transfers from teacher to student but weighted by frequency scores rather than teacher logits. The end result is that both prompt engineering and fine-tuning operations bias toward natural, frequently-occurring language patterns.
Production Impact
For production systems, this approach offers a low-friction optimization lever: you can improve LLM performance by preprocessing user inputs through a paraphraser before feeding them to the model, with minimal latency overhead (paraphrasing typically adds 50-200ms). For fine-tuning pipelines, you could prioritize or weight high-frequency training examples higher, improving convergence speed and reducing required training data volume—potentially 10-30% data reduction for equivalent performance based on similar frequency-based pruning work. The trade-off is that you need frequency statistics from large corpora (addressable via cached frequency tables) and paraphrasing can slightly increase prompt tokens, raising API costs. Integration complexity is moderate: existing systems can add a paraphrase layer without architectural changes, and frequency distillation fits into standard training loops as a modification to the loss weighting scheme.
Limitations and When Not to Use This
The paper assumes that online corpus frequency is a reliable proxy for what LLMs actually learned during pretraining—this breaks down for models trained on proprietary or heavily filtered datasets with different frequency distributions. Paraphrasing introduces a semantic drift risk: aggressive paraphrasing to maximize frequency may alter task-critical nuances, and the paper doesn't quantify this failure mode. The approach likely benefits instruction-following and general tasks but may underperform on specialized domains (legal, medical, code) where rare, precise terminology is essential and paraphrasing into common language destroys meaning. The paper cuts off at describing TFD without showing full empirical results, leaving unclear whether frequency benefits scale to very large models or only small/medium ones.
Research Context
This work builds on decades of psycholinguistic research showing word frequency effects in human reading but represents a novel application to LLMs specifically. It extends data-centric ML ideas (prioritizing data quality over quantity) by introducing frequency as a measurable quality metric, complementing recent work on data filtering and curation. The paper opens a new research direction: if frequency matters for LLMs, downstream questions include whether it interacts with model scale, whether it explains why models trained on internet data outperform curated datasets, and whether frequency combined with other linguistic properties (predictability, context diversity) could yield even stronger results. This positions frequency as a fundamental property alongside token distribution and semantic diversity in the study of training data quality.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::