Influence Malleability in Linearized Attention: Dual Implications of Non-Convergent NTK Dynamics
| Authors | Jose Marie Antonio Miñoza et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2603.13085 |
| Download | |
| Categories | cs.LG, cs.CV, stat.ML |
Abstract
Understanding the theoretical foundations of attention mechanisms remains challenging due to their complex, non-linear dynamics. This work reveals a fundamental trade-off in the learning dynamics of linearized attention. Using a linearized attention mechanism with exact correspondence to a data-dependent Gram-induced kernel, both empirical and theoretical analysis through the Neural Tangent Kernel (NTK) framework shows that linearized attention does not converge to its infinite-width NTK limit, even at large widths. A spectral amplification result establishes this formally: the attention transformation cubes the Gram matrix's condition number, requiring width for convergence, a threshold that exceeds any practical width for natural image datasets. This non-convergence is characterized through influence malleability, the capacity to dynamically alter reliance on training examples. Attention exhibits 6--9 higher malleability than ReLU networks, with dual implications: its data-dependent kernel can reduce approximation error by aligning with task structure, but this same sensitivity increases susceptibility to adversarial manipulation of training data. These findings suggest that attention's power and vulnerability share a common origin in its departure from the kernel regime.
Engineering Breakdown
Plain English
This paper investigates why linearized attention mechanisms fail to converge to their theoretical infinite-width limits, even when networks are quite large. The authors prove that linearized attention transforms the Gram matrix in a way that cubes its condition number, meaning you'd need width m = Ω(κ^6) for convergence—a threshold that's completely impractical for real image datasets. Using the Neural Tangent Kernel framework, they show both theoretically and empirically that this non-convergence is a fundamental trade-off inherent to how linearized attention works. This reveals that understanding attention mechanisms requires grappling with complex non-linear dynamics that simple linear approximations can't capture.
Core Technical Contribution
The paper's core contribution is formally proving a fundamental convergence barrier in linearized attention mechanisms through a spectral amplification argument. Rather than proposing a new method, the authors provide theoretical evidence that linearized attention—despite having a clean data-dependent kernel formulation—provably does not achieve the infinite-width NTK limit at practical model scales. The key insight is quantifying exactly how badly the condition number blows up (by a factor of κ³) and translating that into a concrete width requirement (m = Ω(κ^6)) that exceeds any practical threshold for natural images. This is a negative result that reframes how researchers should think about attention mechanism approximations and their theoretical properties.
How It Works
The paper starts with a linearized attention mechanism that has an exact correspondence to a data-dependent Gram-induced kernel. This kernel is derived from the attention weight matrices and allows analysis through the Neural Tangent Kernel framework, which studies how neural networks behave in the infinite-width limit. The authors then apply spectral analysis to show that the attention transformation amplifies the condition number of the Gram matrix—multiplying it by κ³, where κ is the original condition number. This spectral amplification directly translates to a width requirement: to ensure convergence within practical error bounds, the network must have width m = Ω(κ^6), which becomes astronomical for high-dimensional data. They validate this theoretical prediction empirically on natural image datasets, confirming that even large networks fail to converge as predicted by NTK theory.
Production Impact
For engineers building attention-based models, this paper suggests that linearized attention approximations—sometimes used for efficiency gains—come with a hidden theoretical cost that standard scaling doesn't overcome. If you've deployed linearized attention expecting it to behave like the infinite-width limit (as NTK theory would suggest for other mechanisms), this work shows that assumption breaks down even at large model sizes, meaning approximation error remains significant. This has implications for model compression and efficient attention variants: you cannot simply use the NTK framework to predict convergence properties of linearized attention, and empirical validation on your specific dataset becomes essential rather than optional. The practical takeaway is that switching from full attention to linearized attention for efficiency gains requires explicit testing of convergence behavior—you cannot rely on asymptotic theory. For practitioners, this argues for keeping full attention or exploring other efficiency approaches (like sparse attention or learned sparsity) rather than assuming linearized variants scale predictably.
Limitations and When Not to Use This
The paper focuses narrowly on linearized attention and the NTK framework; it doesn't propose solutions to overcome the convergence barrier or suggest practical alternatives that might avoid the κ^6 scaling. The analysis assumes the Gram matrix structure holds exactly, which may not capture all aspects of how attention behaves in realistic training (e.g., with batch normalization, skip connections, or other architectural components). The width requirement of m = Ω(κ^6) is a worst-case bound—practical datasets might converge with looser condition numbers, but the paper doesn't provide guidance on when this might happen. Additionally, the work is limited to the infinite-width regime and doesn't directly address what happens in the practical finite-width training dynamics that practitioners actually care about, meaning the theoretical results may not fully explain the empirical behavior of deployed models.
Research Context
This work builds on the Neural Tangent Kernel literature, which emerged around 2018-2019 as a way to understand deep learning through kernel methods. It extends that framework to attention mechanisms specifically, following recent work attempting to linearize or approximate attention for efficiency. The paper contributes to a growing body of negative results showing that standard scaling laws and convergence guarantees (derived from NTK theory) break down for certain architectures, challenging the universality of approaches like transfer learning and model scaling. This opens a research direction toward understanding which architectural choices are 'NTK-friendly' (converge as theory predicts) versus which ones require fundamentally different analysis, potentially motivating new theoretical tools beyond the kernel regime.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::