The Geometric Alignment Tax: Tokenization vs. Continuous Geometry in Scientific Foundation Models
| Authors | Prashant C. Raju |
| Year | 2026 |
| HF Upvotes | 10 |
| arXiv | 2604.04155 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Foundation models for biology and physics optimize predictive accuracy, but their internal representations systematically fail to preserve the continuous geometry of the systems they model. We identify the root cause: the Geometric Alignment Tax, an intrinsic cost of forcing continuous manifolds through discrete categorical bottlenecks. Controlled ablations on synthetic dynamical systems demonstrate that replacing cross-entropy with a continuous head on an identical encoder reduces geometric distortion by up to 8.5x, while learned codebooks exhibit a non-monotonic double bind where finer quantization worsens geometry despite improving reconstruction. Under continuous objectives, three architectures differ by 1.3x; under discrete tokenization, they diverge by 3,000x. Evaluating 14 biological foundation models with rate-distortion theory and MINE, we identify three failure regimes: Local-Global Decoupling, Representational Compression, and Geometric Vacuity. A controlled experiment confirms that Evo 2's reverse-complement robustness on real DNA reflects conserved sequence composition, not learned symmetry. No model achieves simultaneously low distortion, high mutual information, and global coherence.
Engineering Breakdown
Plain English
This paper identifies a fundamental problem in foundation models for biology and physics: their learned representations don't preserve the continuous geometric structure of the systems they're modeling. The authors show that this failure stems from the 'Geometric Alignment Tax'—the inherent cost of forcing smooth, continuous manifolds through discrete categorical bottlenecks like tokenization. Through controlled experiments on synthetic dynamical systems, they demonstrate that switching from cross-entropy loss to a continuous prediction head reduces geometric distortion by 8.5x on identical encoders, while learned codebooks create a paradoxical trade-off where finer quantization actually worsens geometric fidelity despite improving reconstruction accuracy. Under continuous objectives, different architectures diverge by only 1.3x, but under discrete tokenization they diverge catastrophically by 3,000x—suggesting tokenization is the primary culprit destroying geometric information.
Core Technical Contribution
The paper's core novelty is formally identifying and quantifying the Geometric Alignment Tax—a previously implicit cost function that governs how much continuous geometric information is lost when discrete bottlenecks are introduced in neural architectures. Rather than proposing a new architecture, the authors provide a diagnostic framework showing that the choice between continuous and discrete objectives (not encoder design) is the dominant factor determining whether learned representations preserve the manifold structure of physics and biology systems. They empirically demonstrate a critical finding: learned vector quantization creates a non-monotonic failure mode where increasing codebook resolution paradoxically worsens geometric preservation, revealing a fundamental tension between reconstruction fidelity and geometric alignment that prior work ignored.
How It Works
The mechanism works through controlled ablation studies on synthetic dynamical systems with known ground-truth continuous geometry. For a baseline model, the authors train an encoder-based architecture with identical weights and architecture, but swap the output head: one uses cross-entropy loss (discrete, categorical) while the other uses L2 or continuous regression loss. They then measure geometric distortion by computing how well the learned representations preserve the original system's manifold structure (likely using metrics like tangent space alignment or geodesic distance preservation). For the codebook experiments, they train discrete vector quantization (VQ) variants with different codebook sizes and measure the double bind: smaller codebooks are easier to learn but destroy geometry worse, while larger codebooks improve reconstruction but still degrade geometry compared to continuous heads. The key finding emerges by comparing three architectures under both continuous and discrete training regimes—the architecture choice matters far less (1.3x variance) than the objective choice (3,000x variance).
Production Impact
For engineers building physics simulators, protein structure predictors, or molecular dynamics models, this directly changes how you'd design the output layer: use continuous prediction heads instead of tokenization when geometric fidelity matters. If you're currently using discrete tokens (common in transformer-based bio/physics models), switching to continuous objectives could reduce representation distortion by 8.5x, meaning downstream tasks like trajectory prediction or ODE solving would be substantially more accurate without any encoder redesign. The trade-off is clear: discrete tokenization provides compression and vocabulary-based interpretability, but at massive cost to geometric fidelity—so only use it if you genuinely need compression or discrete interpretability for your use case. In production pipelines, this suggests a two-stage approach: train continuous representations internally for fidelity, optionally add a discrete bottleneck only at the final output layer if downstream systems require it, rather than tokenizing early as is standard practice.
Limitations and When Not to Use This
The paper evaluates primarily on synthetic dynamical systems with known continuous geometry, so the findings may not transfer cleanly to real biological data where ground-truth geometry is unknown or multimodal. The evaluation section is incomplete in the abstract (cuts off at '14 biologica...'), so critical details about which biological and physics benchmarks were tested, what baseline methods were compared against, and actual end-to-end task performance are missing—geometric fidelity doesn't always correlate with downstream task accuracy. The approach assumes that preserving continuous manifold structure is the right objective, but some applications may benefit from discrete abstraction and may not need geometric alignment at all. The paper doesn't address how to scale this insight to very large models (the synthetic experiments may use small networks), and it's unclear whether the geometric alignment tax remains constant or changes with model capacity, data scale, or domain complexity.
Research Context
This work builds directly on the long-standing observation that transformer-based and discrete tokenization approaches (like VQ-VAE, used widely in video and biology models) lose information, but reframes the loss as a fundamental geometry problem rather than just a reconstruction accuracy problem. It connects to broader research on geometric deep learning and manifold preservation in neural networks, and provides formal evidence for why continuous representations (like those used in neural ODE solvers and implicit neural representations) outperform discrete ones for physics and biology. The paper challenges the tokenization-first paradigm that has dominated recent foundation model design, suggesting instead that the physics/biology communities should learn from continuous-domain deep learning rather than copying the discrete architecture patterns from NLP. This opens a research direction: how to design discrete bottlenecks that preserve geometric structure (hybrid approaches), and whether other domains like vision benefit from similar continuous-first design principles when the underlying data has manifold structure.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::