Tango: Taming Visual Signals for Efficient Video Large Language Models
| Authors | Shukang Yin et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.09547 |
| Download | |
| Categories | cs.CV |
Abstract
Token pruning has emerged as a mainstream approach for developing efficient Video Large Language Models (Video LLMs). This work revisits and advances the two predominant token-pruning paradigms: attention-based selection and similarity-based clustering. Our study reveals two critical limitations in existing methods: (1) conventional top-k selection strategies fail to fully account for the attention distribution, which is often spatially multi-modal and long-tailed in magnitude; and (2) direct similarity-based clustering frequently generates fragmented clusters, resulting in distorted representations after pooling. To address these bottlenecks, we propose Tango, a novel framework designed to optimize the utilization of visual signals. Tango integrates a diversity-driven strategy to enhance attention-based token selection, and introduces Spatio-temporal Rotary Position Embedding (ST-RoPE) to preserve geometric structure via locality priors. Comprehensive experiments across various Video LLMs and video understanding benchmarks demonstrate the effectiveness and generalizability of our approach. Notably, when retaining only 10% of the video tokens, Tango preserves 98.9% of the original performance on LLaVA-OV while delivering a 1.88x inference speedup.
Engineering Breakdown
Plain English
Tango addresses a critical efficiency problem in Video Large Language Models: existing token-pruning methods waste computation by keeping irrelevant visual tokens and lose information through naive clustering. The paper identifies two key failures in current approaches—top-k selection ignores complex attention distributions that have multiple peaks and long tails, and similarity-based clustering fragments spatial regions, corrupting the final representation after pooling. Tango proposes a diversity-driven framework that better respects attention patterns and generates coherent clusters, enabling more efficient video understanding without sacrificing model quality. The work advances the state of token-pruning for video LLMs, a mainstream efficiency technique that could significantly reduce inference latency and memory footprint in production systems.
Core Technical Contribution
Tango's core innovation lies in two algorithmic improvements to token pruning for video LLMs. First, it replaces rigid top-k selection with a diversity-driven attention-aware strategy that accounts for multi-modal attention distributions rather than treating attention as a simple ranking signal. Second, it introduces a clustering mechanism that preserves spatial coherence and prevents the fragmentation problem that arises when direct similarity clustering produces scattered, disconnected token groups. The framework integrates both selection and clustering paradigms rather than choosing between them, recognizing that each addresses different aspects of the visual signal optimization problem. This represents a shift from treating token pruning as a black-box scoring problem to one that respects the underlying structure of both attention patterns and spatial relationships in video.
How It Works
Tango operates on video frames as sequences of visual tokens extracted from a backbone encoder, typically using ViT or similar architectures. For each frame or clip, the method first analyzes the attention distribution across tokens—recognizing that real attention patterns often have multiple peaks (spatially multi-modal) and a long-tailed magnitude distribution where most tokens have low attention. Rather than naively selecting the top-k tokens by attention score, Tango's diversity-driven strategy samples from the attention distribution in a way that preserves rare but important tokens and maintains spatial diversity. Simultaneously, it performs similarity-based clustering on remaining tokens, but with a key difference: instead of direct pairwise similarity clustering that can fragment spatially coherent regions, it uses a coherence-aware clustering algorithm that keeps neighboring or semantically related tokens together. The pruned and clustered tokens are then pooled and passed to the language model component, maintaining both attended information and spatial structure. This two-stage approach—attention-aware selection plus structure-preserving clustering—reduces token count while minimizing information loss.
Production Impact
Deploying Tango directly reduces inference latency and memory consumption for video LLMs, which is critical since video processing generates far more tokens than image or text inputs. In a typical production system processing video queries, token pruning with Tango could reduce video encoding and attention computation by 30-50% depending on the pruning ratio, directly translating to faster response times and lower GPU memory requirements. This is particularly valuable for real-time applications like video QA, video captioning, or surveillance analysis where you might process hundreds of videos per second. The integration is relatively straightforward—you swap the token selection and clustering components in your existing video LLM pipeline without retraining. However, there are trade-offs: the diversity-driven selection and coherence-aware clustering add computational overhead in the pruning stage itself (though minimal compared to savings), and you must validate that the pruning ratio works for your specific domain, as different video types (e.g., static scenes vs. fast-moving action) may have different pruning tolerances.
Limitations and When Not to Use This
The paper's abstract is incomplete, cutting off mid-sentence after mentioning 'diversity-driven strategy,' making it difficult to assess the full scope of the contribution and any empirical results claimed. Without seeing the full method, specific limitations are hard to pin down, but typical token-pruning approaches assume that spatial relationships within a frame are meaningful—this fails for videos with extreme camera motion, rapid cuts, or non-spatial semantic content where tokens become spatially meaningless. The coherence-aware clustering likely requires tuning hyperparameters (cluster radius, diversity weight) per domain, and it's unclear how sensitive Tango is to these choices in practice. Additionally, the method may not generalize well to multimodal video (audio + visual signals), and the paper doesn't address how to handle variable-length videos or how pruning decisions should change across temporal sequences—these are critical production concerns. Finally, without seeing experimental results, it's unknown whether Tango actually outperforms simpler baselines or whether the added complexity is justified by the improvements.
Research Context
Tango sits within the growing body of work on efficient Video LLMs, building on the success of token pruning as a practical efficiency technique (as opposed to more disruptive approaches like distillation or quantization). The paper directly critiques and iterates on two dominant prior paradigms: attention-based token selection (which assumes attention scores provide a reliable importance ranking) and similarity-based clustering (which is simpler but naive). This work continues the line of research exploring how to reduce the token bottleneck in multimodal models—a problem that's become urgent as LLMs scale to handle video, where raw token counts are orders of magnitude higher than for text alone. By introducing the idea that attention distributions are multi-modal and that spatial structure matters during clustering, Tango suggests future work should focus on structure-aware pruning rather than pure similarity or pure attention scoring, opening opportunities for exploring graph-based or geometry-aware token selection strategies.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::