World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings
| Authors | Elan Barenholtz |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.04317 |
| Download | |
| Categories | cs.CL, cs.AI, cs.LG |
Abstract
Recent work interprets the linear recoverability of geographic and temporal variables from large language model (LLM) hidden states as evidence for world-like internal representations. We test a simpler possibility: that much of the relevant structure is already latent in text itself. Applying the same class of ridge regression probes to static co-occurrence-based embeddings (GloVe and Word2Vec), we find substantial recoverable geographic signal and weaker but reliable temporal signal, with held-out R^2 values of 0.71-0.87 for city coordinates and 0.48-0.52 for historical birth years. Semantic-neighbor analyses and targeted subspace ablations show that these signals depend strongly on interpretable lexical gradients, especially country names and climate-related vocabulary. These findings suggest that ordinary word co-occurrence preserves richer spatial, temporal, and environmental structure than is often assumed, revealing a remarkable and underappreciated capacity of simple static embeddings to preserve world-shaped structure from text alone. Linear probe recoverability alone therefore does not establish a representational move beyond text.
Engineering Breakdown
Plain English
This paper challenges the assumption that large language models develop world-like internal representations by showing that much of the geographic and temporal structure recovered from LLM hidden states already exists in simpler word embeddings. The authors applied ridge regression probes to static embeddings like GloVe and Word2Vec, finding they could recover city coordinates with R² values of 0.71-0.87 and historical birth years with R² of 0.48-0.52. By analyzing semantic neighborhoods and ablating specific subspaces, they discovered these signals depend heavily on interpretable lexical patterns—particularly country names and climate-related vocabulary—rather than complex learned representations. This suggests that LLMs may not be developing novel geographic reasoning so much as leveraging structured information already present in the text they're trained on.
Core Technical Contribution
The core novelty is showing that linear probes applied to static, non-contextual word embeddings recover comparable levels of geographic and temporal structure as probes applied to LLM hidden states, contradicting the narrative that LLMs build sophisticated world models. The authors' specific contribution is a systematic comparison across embedding types (GloVe, Word2Vec) with quantitative held-out evaluation and mechanistic analysis via semantic-neighbor inspection and targeted subspace ablations. Rather than proposing a new algorithm, the paper makes an interpretability claim: the relevant signal depends on lexical gradients (country names, climate vocabulary) that are already encoded in text-level co-occurrence statistics. This reframes what probing success actually demonstrates—not model sophistication, but information already available in raw linguistic patterns.
How It Works
The approach starts with static word embeddings (GloVe or Word2Vec trained on standard corpora) and treats each word as a data point with geographic (city coordinates) or temporal (birth years) labels extracted from text context or external knowledge bases. Ridge regression probes are fit to predict these target variables from embedding vectors, using standard train-test splits with held-out evaluation to measure generalization (R² scores). To understand what drives the recovered signal, the authors perform semantic-neighbor analysis—examining which words have the strongest predictive features—and discover that top predictors are interpretable country names and climate-related terms. They then conduct targeted subspace ablations, removing specific dimensions or word groups (e.g., all country names) and remeasuring probe performance to quantify how much each lexical gradient contributes to the overall signal. This mechanistic breakdown reveals that the model isn't performing sophisticated geographic inference but rather exploiting surface-level text patterns.
Production Impact
For engineers building LLM-based systems, this work suggests that probing results should not be over-interpreted as evidence of deep world knowledge—much apparent structure may derive from textual patterns rather than learned reasoning. If your system relies on downstream tasks requiring geographic or temporal reasoning, you should validate whether the LLM is actually generalizing to novel contexts or merely interpolating from text statistics; the paper implies the latter is more likely. On the positive side, this finding could simplify certain pipelines: for lightweight geolocation or temporal tagging tasks where text contains surface-level cues (country names, climate descriptors), you might achieve comparable performance with smaller, non-contextual embeddings (GloVe/Word2Vec) at a fraction of the compute cost. However, the trade-off is that static embeddings won't handle context-dependent or out-of-distribution cases; you'd need to measure task-specific performance before committing to this simplification in production.
Limitations and When Not to Use This
The paper focuses narrowly on geographic and temporal variables, which are among the most text-observable properties; it's unclear whether the findings generalize to more abstract world knowledge (e.g., causal relationships, counterfactuals, or physical laws) that truly require model reasoning. The analysis relies on manual inspection of semantic neighbors and heuristic subspace ablations, which may miss more subtle interactions or distributed representations that don't align neatly with interpretable word groups. The study does not directly compare the downstream task performance of LLM probes versus static embedding probes on real applications, so it remains an open question whether this interpretability insight translates to practical performance differences. Additionally, the paper doesn't address modern large-scale embeddings (e.g., BERT, contextual embeddings) in detail, so it's unclear how the findings scale to contextualized representations, which may show different dependencies on lexical gradients.
Research Context
This work builds on a recent trend in mechanistic interpretability of LLMs, particularly efforts to understand what structure models learn about the world by applying linear probes to hidden states. It directly engages with prior claims that LLMs develop world-like representations—a belief popular in the interpretability community but potentially overstated. The paper is part of a broader conversation about what probing success actually measures: genuine model knowledge versus superficial pattern matching. This research direction connects to work on disentanglement, probing reliability, and the assumptions underlying mechanistic interpretability studies, encouraging skepticism about strong claims of world knowledge based purely on linear recoverability.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::