On the Proper Treatment of Units in Surprisal Theory
| Authors | Samuel Kiegeland et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.28147 |
| Download | |
| Categories | cs.CL |
Abstract
Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive.
Engineering Breakdown
Plain English
This paper addresses a fundamental mismatch in how surprisal theory—a cognitive model linking processing effort to linguistic predictability—is empirically tested. Researchers typically measure surprisal by segmenting experiments into linguistically meaningful units (like words), but pretrained language models assign probabilities to fixed token alphabets that don't align with those units. The authors disentangle these two conflated choices and provide a unified mathematical framework for computing surprisal over any arbitrary unit inventory. Their contribution clarifies how to properly map between model predictions and behavioral measurements, fixing a widespread methodological inconsistency in psycholinguistic research.
Core Technical Contribution
The paper's core innovation is formalizing the distinction between two independent modeling choices: (1) the definition of the analytical unit (word, character, morpheme, etc.) and (2) the regions of interest over which predictions are aggregated. Prior work implicitly conflated these choices through ad hoc procedures that made comparisons across studies unreliable. The authors derive a unified mathematical framework that allows surprisal computation over arbitrary unit inventories by properly marginalizing token-level probabilities from language models. This enables consistent, theoretically grounded surprisal estimation regardless of unit choice, making results reproducible and comparable across different experimental paradigms.
How It Works
The framework begins with token-level probability distributions from a pretrained language model, which assign mass to a fixed vocabulary (typically 50K–250K tokens). Rather than directly using these token probabilities as surprisal, the authors define a principled mapping to higher-level units through marginalization. Given a target unit (e.g., a word composed of multiple subword tokens), they sum or integrate the probability mass assigned by the model to all token sequences that realize that unit. The surprisal of the unit is then −log of this marginalized probability. The framework handles ambiguous unit boundaries (common in tokenization) by either summing over all valid token segmentations or selecting the highest-probability segmentation depending on the theoretical choice. This produces unit-level surprisal values that are theoretically grounded in the model's actual probability distribution, not post-hoc approximations.
Production Impact
For engineers building psycholinguistic evaluation pipelines or cognitive modeling systems, this framework eliminates a significant source of subtle errors in how surprisal metrics are computed. Instead of relying on heuristic procedures (e.g., summing log-probabilities of subword tokens), you can now implement a standard, mathematically principled computation that works consistently across different model tokenizers and unit definitions. This is particularly valuable when comparing pretrained models with different tokenization schemes—BERT-style WordPiece, GPT-style BPE, SentencePiece—since the framework makes such comparisons reliable. Integration cost is moderate: you need a clean implementation of marginalization over token sequences, which is straightforward given a model's token probabilities and a tokenizer. The latency impact is minimal for single-pass inference, though computing marginalizations over large alternative segmentations could add overhead in high-throughput scenarios; this can be mitigated through caching.
Limitations and When Not to Use This
The paper assumes access to a tractable token-level probability distribution, which may be expensive or impossible to obtain for very large models running on distributed inference systems where only beam-search outputs are cached. The framework also requires specifying a principled unit inventory in advance; it does not address how to select the 'correct' linguistic unit of analysis, which remains a theoretical question in cognitive science. Computing marginalizations over all possible token segmentations can be exponential in unit length if not carefully implemented, and the paper does not provide detailed algorithmic guidance on efficient marginalization for long sequences or ambiguous tokenizations. Finally, the work is evaluated primarily in the psycholinguistic domain; its applicability to other NLP evaluation tasks (machine translation, parsing, etc.) where surprisal matters is not explored.
Research Context
This work builds directly on decades of surprisal-theory research in psycholinguistics, starting with Hale (2001) and Levy (2008), which established that reading times and processing difficulty correlate with model-predicted surprisal. Recent work (e.g., Wilcox et al. 2023) has shown that pretrained neural language models' surprisal values predict human reading times, but these studies glossed over the unit-alignment problem. The paper formalizes what practitioners had been doing informally, filling a gap between theoretical cognitive modeling and practical implementation with modern tokenized models. This opens avenues for more rigorous benchmarking of language models against human processing data and for developing better theories of how neural architectures align with human linguistic cognition.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::