Linear-Time Global Visual Modeling without Explicit Attention
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-03 with 4 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Ruize He et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2605.01711 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Existing research largely attributes the global sequence modeling capability of Transformers to the explicit computation of attention weights, a process that inherently incurs quadratic computational complexity. In this work, we offer a novel perspective: we demonstrate that attention can be mathematically reframed as a Multi-Layer Perceptron (MLP) equipped with dynamically predicted parameters. Through this lens, we explain attention's global modeling power not as explicit token-wise aggregation, but as an implicit process where dynamically generated parameters act as a compressed representation of the global context. Inspired by this insight, we investigate a fundamental question: can we achieve Transformer-level sequence global modeling entirely through dynamic parameterization while maintaining linear complexity, effectively replacing explicit attention? To explore this, we design various dynamic parameter prediction strategies and integrate them into standard network layers. Extensive empirical studies on vision models demonstrate that dynamic parameterization can indeed serve as a highly effective, linear-complexity alternative to explicit attention, opening new pathways for efficient sequence modeling. Code is available at https://github.com/LeapLabTHU/WeightFormer.
Engineering Breakdown
Plain English
This paper reframes attention mechanisms as MLPs with dynamically predicted parameters instead of explicit token-wise weight computation, then demonstrates you can achieve Transformer-level global sequence modeling without quadratic attention operations. The core finding is that attention's global modeling power comes from implicit context compression through dynamic parameters, not from explicitly computing and aggregating attention weights.
Key Engineering Insight
Attention can be mathematically decomposed as a parameter-generation function rather than a weighted aggregation function — this means you can potentially replace O(n²) attention computation with O(n) MLP-based dynamic parameterization without losing the model's ability to capture global dependencies.
Why It Matters for Engineers
For production systems, this directly addresses the memory and latency bottleneck of Transformers on long sequences. If validated at scale, this approach could enable efficient inference on longer contexts (document processing, video understanding, time series) without the engineering complexity of sparse attention approximations or KV-cache management overhead.
Research Context
Prior work established that Transformers achieve global modeling through explicit attention, leading to unavoidable O(n²) scaling. This paper challenges that assumption by proposing attention is fundamentally about dynamic parameter generation, positioning implicit parameterization as a new direction for linear-time global sequence models that could compete with or replace explicit attention in future architectures.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::