MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-08 with 7 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Ruijie Zhou et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2605.07363 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
DeepSeek Sparse Attention (DSA) sets the state of the art for fine-grained inference-time sparse attention by introducing a learned token-wise indexer that scores every prefix token and selects the most relevant ones for the main attention. To remain expressive, the indexer uses many query heads (for example, 64 on DeepSeek-V3.2) that share the same selected token set; this multi-head design is precisely what makes the indexer the dominant cost on long contexts. We propose MISA (Mixture of Indexer Sparse Attention), a drop-in replacement for the DSA indexer that treats its indexer heads as a pool of mixture-of-experts. A lightweight router uses cheap block-level statistics to pick a query-dependent subset of only a few active heads, and only those heads run the heavy token-level scoring. This preserves the diversity of the original indexer pool while reducing the per-query cost from scoring every prefix token with every head to scoring it with only a handful of routed heads, plus a negligible router term computed on a small set of pooled keys. We further introduce a hierarchical variant of MISA that uses the routed pass to keep an enlarged candidate set and then re-ranks it with the original DSA indexer to recover the final selected tokens almost exactly. With only eight active heads and no additional training, MISA matches the dense DSA indexer on LongBench across DeepSeek-V3.2 and GLM-5 while running with eight and four times fewer indexer heads respectively, and outperforms HISA on average. It also preserves fully green Needle-in-a-Haystack heatmaps up to a 128K-token context and recovers more than 92% of the tokens selected by the DSA indexer per layer. Our TileLang kernel delivers roughly a 3.82 times speedup over DSA's original indexer kernel on a single NVIDIA H200 GPU.
Engineering Breakdown
Plain English
This paper optimizes the inference speed of DeepSeek's sparse attention mechanism by replacing its expensive indexer component with a mixture-of-experts design. Instead of running 64+ indexer heads on every token to decide which prefix tokens to attend to, MISA uses a cheap router that selects only a few active heads per query, dramatically reducing compute while maintaining the quality of token selection.
Key Engineering Insight
The breakthrough is treating redundant indexer heads as a mixture-of-experts pool — most queries don't need all 64 heads to make good attention decisions, so a lightweight block-level router can pick a query-dependent subset that cuts indexer cost without hurting expressiveness.
Why It Matters for Engineers
For engineers shipping long-context LLM services, inference latency on context scales of 100K+ tokens is a critical bottleneck. Sparse attention helps, but the indexer itself becomes the bottleneck on long sequences. This work directly attacks that bottleneck, offering a straightforward drop-in replacement that should improve throughput and reduce serving costs without retraining.
Research Context
DeepSeek's sparse attention pushed inference-time sparsity forward by learning which tokens matter per query, but the design scaled poorly because the indexer was built monolithically. MISA advances the field by showing that sparse attention indexers themselves benefit from sparsity via routing — enabling practical long-context inference at scales where the previous approach became prohibitively expensive.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::