Long Context Pre-Training with Lighthouse Attention
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 21 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Bowen Peng et al. |
| Year | 2026 |
| HF Upvotes | 21 |
| arXiv | 2605.06554 |
| Download | |
| Code | https://github.com/ighoshsubho/lighthouse-attention |
Abstract
Training causal transformers at extreme sequence lengths is bottlenecked by the quadratic time and memory of scaled dot-product attention (SDPA). In this work, we propose Lighthouse Attention, a training-only symmetrical selection-based hierarchical attention algorithm that wraps around ordinary SDPA and can be easily removed towards the end of the training. Our hierarchical selection is also gradient-free, which exempts us from dealing with a complicated and potentially inefficient backward pass kernel. Our contribution is three-fold: (i) A subquadratic hierarchical pre- and post-processing step that does adaptive compression and decompression of the sequence. (ii) A symmetrical compression strategy that pools queries, keys and values at the same time, while preserving left-to-right causality, which greatly improves parallelism. (iii) A two stage training approach which we pre-train for the majority of the time with Lighthouse Attention and recover a full attention model at the end with a short training. We run preliminary small scale LLM pre-training experiments that show the effectiveness of our method compared to full attention training with all other settings matched, where we achieve a faster total training time and lower final loss after the recovery phase. Full code is available at: https://github.com/ighoshsubho/lighthouse-attention
Engineering Breakdown
Plain English
This paper introduces Lighthouse Attention, a training-only attention mechanism that reduces the quadratic memory and compute overhead of standard transformer attention during pre-training on very long sequences. The method uses a hierarchical compression strategy that adaptively pools queries, keys, and values together while maintaining causal (left-to-right) ordering, and can be completely removed at inference time without retraining.
Key Engineering Insight
The core innovation is a gradient-free hierarchical selection process that compresses sequences during training without requiring custom backward pass kernels—meaning it wraps standard attention implementations and avoids the engineering complexity of writing fused CUDA kernels for backprop through learned selection masks.
Why It Matters for Engineers
Long-context pre-training is bottlenecked by attention's O(n²) cost, making 100K+ token training prohibitively expensive on current hardware. This approach lets engineers train transformers on extreme sequence lengths using standard SDPA implementations (easier to integrate, less maintenance burden), then deploy them at full context length at inference with no architectural changes.
Research Context
Prior work on efficient attention (sparse patterns, low-rank approximations, retrieval augmentation) faces trade-offs between training cost and inference capability. Lighthouse Attention advances the state by being a pure training optimization that doesn't constrain model architecture or inference—it's a temporary scaffold you remove after training, sidestepping the usual efficiency-versus-capability tension in long-context research.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::