Sessa: Selective State Space Attention
| Authors | Liubomyr Horbatko |
| Year | 2026 |
| HF Upvotes | 0 |
| arXiv | 2604.18580 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Modern sequence modeling is dominated by two families: Transformers, whose self-attention can access arbitrary elements of the visible sequence, and structured state-space models, which propagate information through an explicit recurrent state. These mechanisms face different limitations on long contexts: when attention is diffuse, the influence of individual tokens is diluted across the effective support, while recurrent state propagation can lose long-range sensitivity unless information is actively preserved. As a result, both mechanisms face challenges in preserving and selectively retrieving information over long contexts. We propose Sessa, a decoder that places attention inside a recurrent feedback path. This creates many attention-based paths through which past tokens can influence future states, rather than relying on a single attention read or a single recurrent chain. We prove that, under explicit assumptions and matched regimes, Sessa admits power-law memory tails O(ell^{-β}) for 0 < β< 1, with slower decay than in the corresponding Transformer and Mamba-style baselines. We further give an explicit construction that achieves this power-law rate. Under the same assumptions, Sessa is the only model class among those considered that realizes flexible selective retrieval, including profiles whose influence does not decay with distance. Consistent with this theoretical advantage, across matched experiments, Sessa achieves the strongest performance on long-context benchmarks while remaining competitive with Transformer and Mamba-style baselines on short-context language modeling.
Engineering Breakdown
Plain English
Sessa proposes a novel decoder architecture that addresses a fundamental limitation in modern sequence models: both Transformers and state-space models struggle to preserve and retrieve information over long contexts. Transformers suffer from diffuse attention where individual token influence gets diluted across many positions, while recurrent state-space models lose long-range sensitivity because they compress information through a bottleneck state. The core innovation is placing attention mechanisms inside a recurrent feedback loop, creating multiple attention-based pathways that allow past tokens to influence future computation in a structured way. This hybrid approach aims to combine the flexible information access of attention with the efficient recurrent structure of state-space models.
Core Technical Contribution
The technical novelty is the architectural innovation of embedding attention operations within a recurrent feedback path rather than using attention and recurrence as mutually exclusive mechanisms. Unlike standard Transformers that apply attention once at each layer or RNNs that use a single compressed state, Sessa creates a multiplicative effect where attention can be selectively applied at each recurrence step, enabling tokens to maintain influence through multiple pathways simultaneously. This design allows the model to learn which attention patterns are useful for long-range dependencies while maintaining the computational efficiency advantages of recurrent processing. The key insight is that attention within recurrence creates exponentially more effective paths for information flow compared to either mechanism alone.
How It Works
The Sessa decoder operates by taking a sequence input and processing it through a recurrent loop where each timestep combines traditional recurrent state propagation with selective attention. At each recurrence step, the model maintains a hidden state that carries information from previous tokens, then applies a learned attention mechanism over the visible context to selectively retrieve relevant information before updating the state for the next step. The attention is conditioned on the current hidden state, allowing the model to adaptively choose which past tokens are most relevant given what has been processed so far. The recurrent loop iterates, and after processing the full sequence, the final hidden state and the learned attention weights enable the model to decode output tokens while maintaining access to long-range context. This is fundamentally different from standard attention-only models because the recurrence creates a gating mechanism that controls information flow, and different from RNNs because attention provides explicit, learned access to arbitrary past positions rather than implicit access through a compressed state.
Production Impact
Engineers adopting Sessa would gain improved handling of long-context tasks like document summarization, code understanding, and long-form generation where both Transformers and existing state-space models struggle. The hybrid architecture could reduce memory consumption compared to full Transformer attention (which scales quadratically with sequence length) while maintaining better long-range sensitivity than pure RNNs, making it attractive for deployment on resource-constrained devices or handling documents exceeding typical context windows. Integration would require replacing standard decoder blocks with Sessa blocks, retraining from scratch or fine-tuning existing models, and benchmarking against both Transformer and Mamba-style baselines on your specific tasks. Trade-offs include potential latency overhead from iterative recurrence (cannot parallelize across timesteps like Transformers) and increased model complexity requiring careful hyperparameter tuning for attention mechanisms within the loop; the actual speedup depends on your inference framework's optimization of recurrent operations.
Limitations and When Not to Use This
The paper does not address how Sessa handles variable-length sequences during batching, which is a practical concern in production systems where sequences have different lengths. There is no discussion of training stability or gradient flow through deeply recurrent paths, which historically plagues RNN-based architectures and could limit scaling to very long sequences. The approach assumes attention computed at each recurrence step provides net benefit over the computational cost, but this may not hold for all sequence types or domains—particularly short sequences where the overhead of recurrence outweighs benefits of selective attention. Additionally, the incomplete abstract prevents assessment of whether empirical results justify the added architectural complexity, and whether Sessa actually outperforms recent efficient Transformer variants (like sparse attention or linear attention mechanisms) or modern state-space models on standard benchmarks.
Research Context
Sessa builds on the long line of work comparing and combining attention mechanisms with recurrent processing, drawing from Transformer research (Vaswani et al., 2017) and recent state-space models like S4 and Mamba that challenge Transformer dominance on long sequences. The work addresses an acknowledged research gap: while Transformers and state-space models represent two distinct paradigms with different trade-offs, few papers have attempted to genuinely fuse them by placing attention inside recurrence rather than simply using them sequentially or in parallel. This fits a broader research direction of moving beyond pure attention-only architectures toward hybrid models that combine the strengths of different inductive biases, similar to work on recurrent Transformers and attention-augmented RNNs. The contribution opens potential follow-up research on understanding when recurrent attention is preferable, how to optimize its implementation, and whether this pattern generalizes to encoder-decoder architectures or other sequence modeling domains.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::