Lightning Unified Video Editing via In-Context Sparse Attention
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-06 with 13 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Shitong Shao et al. |
| Year | 2026 |
| HF Upvotes | 13 |
| arXiv | 2605.04569 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Video editing has evolved toward In-Context Learning (ICL) paradigms, yet the resulting quadratic attention costs create a critical computational bottleneck. In this work, we propose In-context Sparse Attention (ISA), the first near-lossless empirical sparse framework tailored for ICL video editing. Our design is grounded in two key insights: first, context tokens exhibit significantly lower saliency than source tokens; second, we theoretically prove and empirically validate that Query sharpness correlates with approximation error. Motivated by these findings, ISA implements an efficient pre-selection strategy to prune redundant context, followed by a dynamic query grouping mechanism that routes high-error queries to full attention and low-error ones to a computationally efficient 0-th order Taylor sparse attention. Furthermore, we build \texttt{LIVEditor} , a novel lightning video editing model via ISA and a proposed video-editing data pipeline that curated a 1.7M high-quality dataset. Extensive experiments demonstrate that LIVEditor achieves a sim60% reduction in attention-module latency while surpassing state-of-the-art methods across EditVerseBench, IVE-Bench, and VIE-Bench, delivering near-lossless acceleration without compromising visual fidelity.
Engineering Breakdown
Plain English
This paper tackles the computational cost of video editing using in-context learning (ICL), where attention mechanisms scale quadratically with input size. The authors propose In-context Sparse Attention (ISA), which prunes redundant context tokens and routes queries dynamically—sending difficult queries to full attention and easy ones to cheaper approximations—achieving near-lossless performance at significantly lower computational cost.
Key Engineering Insight
Query sharpness directly correlates with approximation error, enabling a practical routing strategy: you can safely use cheap attention for queries that are already confident in their outputs, but must use full attention only where needed. This decoupling of computational cost from model capacity is rare in attention mechanisms.
Why It Matters for Engineers
Video editing models are moving toward ICL paradigms for flexibility, but quadratic attention makes them infeasible for production. ISA directly addresses this by reducing compute without model retraining, making ICL-based video editing actually deployable on standard hardware. This matters for any system processing long sequences where most tokens contribute little signal.
Research Context
Prior work on efficient attention (sparse, linear, local) didn't specifically optimize for ICL video editing's unique token importance patterns. This paper advances the field by proving that context and source tokens have fundamentally different roles, and that query-level routing enables dramatically better efficiency than one-size-fits-all pruning. It opens the door to ICL-based video systems that were previously computationally intractable.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::