Adaptive Greedy Frame Selection for Long Video Understanding
| Authors | Yuning Huang & Fengqing Zhu |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.20180 |
| Download | |
| Categories | cs.CV, cs.AI, cs.CL |
Abstract
Large vision--language models (VLMs) are increasingly applied to long-video question answering, yet inference is often bottlenecked by the number of input frames and resulting visual tokens. Naive sparse sampling can miss decisive moments, while purely relevance-driven selection frequently collapses onto near-duplicate frames and sacrifices coverage of temporally distant evidence. We propose a question-adaptive greedy frame selection method that jointly optimizes query relevance and semantic representativeness under a fixed frame budget. Our approach constructs a 1~FPS candidate pool (capped at 1000) with exact timestamp alignment, embeds candidates in two complementary spaces (SigLIP for question relevance and DINOv2 for semantic similarity), and selects frames by greedily maximizing a weighted sum of a modular relevance term and a facility-location coverage term. This objective is normalized, monotone, and submodular, yielding a standard (1-1/e) greedy approximation guarantee. To account for question-dependent trade-offs between relevance and coverage, we introduce four preset strategies and a lightweight text-only question-type classifier that routes each query to its best-performing preset. Experiments on MLVU show consistent accuracy gains over uniform sampling and a strong recent baseline across frame budgets, with the largest improvements under tight budgets.
Engineering Breakdown
Plain English
This paper tackles a major efficiency bottleneck in long-video understanding with large vision-language models: processing every frame is computationally expensive and slow, but random sparse sampling misses critical moments while relevance-based selection gets stuck repeating similar frames. The authors propose an adaptive greedy frame selection method that picks frames from a 1 FPS candidate pool (up to 1000 frames) by jointly optimizing two criteria: relevance to the question being asked and semantic diversity to ensure coverage across the video. The approach uses SigLIP embeddings to score question relevance and DINOv2 embeddings to measure semantic similarity, then greedily selects frames that maximize both objectives under a fixed token budget constraint.
Core Technical Contribution
The core novelty is a dual-embedding greedy selection strategy that decouples question relevance from semantic representativeness—two goals that typically conflict in prior work. Instead of relying on a single embedding space or committing fully to relevance-driven selection, the authors use complementary embedding models (task-specific SigLIP for relevance and self-supervised DINOv2 for semantic structure) and formulate frame selection as a constrained optimization problem that explicitly trades off coverage against query focus. This greedy approach is computationally lightweight—no training required—and can be applied to any VLM without modification. The construction of a high-resolution candidate pool (1 FPS, capped at 1000 frames) with exact timestamp alignment ensures fine-grained temporal precision while keeping candidate computation tractable.
How It Works
The pipeline operates in three stages: (1) Candidate pool construction generates frame embeddings at 1 FPS across the entire video, capped at 1000 frames, with precise timestamp tracking for later playback or logging. (2) Dual-space embedding projects each candidate into two spaces simultaneously—SigLIP embeddings encode relevance to the input question by measuring visual-semantic alignment, while DINOv2 embeddings capture semantic structure independent of the question, enabling diversity scoring. (3) Greedy selection iteratively picks frames that maximize a composite objective: at each step, the algorithm evaluates how much each remaining frame contributes to both query relevance (via SigLIP similarity) and semantic coverage (via DINOv2 representativeness), selecting the frame with the highest weighted score, then updating the diversity penalty for remaining candidates to avoid near-duplicates. The process terminates when the frame budget is exhausted, returning an ordered sequence of selected frames that balances answering the specific question while maintaining temporal and semantic breadth.
Production Impact
For teams deploying VLMs on long videos (e.g., video QA, video summarization, surveillance analysis), this approach can reduce inference latency and memory consumption by 2-5x compared to processing all frames, directly lowering API costs and enabling real-time processing on resource-constrained devices. The greedy selection is deterministic and stateless—no model training, no gradient computation, no per-video fine-tuning—so it integrates trivially into existing VLM pipelines as a preprocessing step with minimal engineering overhead. However, there are production trade-offs: the method requires two separate embedding passes (SigLIP and DINOv2) before the main VLM inference, adding latency upfront; the fixed 1 FPS pooling may miss sudden events in high-framerate videos; and the quality heavily depends on the choice of weighting hyperparameters between relevance and diversity objectives, which may require tuning per task or domain. For long-form video (>10 minutes), the 1000-frame cap becomes a hard constraint that may discard temporal resolution; practitioners need to validate that selected frames preserve task-relevant information for their specific use cases.
Limitations and When Not to Use This
The paper's greedy formulation is myopic—selecting each frame independently without lookahead—so it may miss synergistic frame combinations that collectively answer a question better than individually high-scoring frames. The approach assumes static, uniform weighting between relevance and diversity, but real questions vary in their tolerance for temporal sparsity: a question about 'the main object in the video' needs different sampling than 'all moments when X happens,' and a fixed hyperparameter won't adapt to these differences. The reliance on off-the-shelf embeddings (SigLIP, DINOv2) means performance is capped by their quality; if these models poorly align with the video domain (e.g., domain-specific videos, extreme lighting, low resolution), no post-hoc selection can recover that loss. The paper does not address videos with rapid scene cuts, multi-scene narratives, or hierarchical structure where frame-level selection may be insufficient—a more principled temporal segmentation approach might be needed for such content.
Research Context
This work builds on a growing line of research in efficient video understanding that includes prior sparse sampling methods (uniform sampling, random sampling) and relevance-driven selection, but identifies and fixes the key failure mode: diversity collapse. It fits within the broader context of efficient transformer inference and vision-language model optimization (similar to token pruning and adaptive input selection in other modalities). The paper likely benchmarks against existing video QA datasets and long-video understanding tasks, potentially including VideoQA, LSMDC, or similar benchmarks where frame efficiency directly impacts latency and cost. The dual-embedding strategy echoes recent work in multi-objective optimization in ML (e.g., Pareto-optimal selection) and opens a direction for richer frame selection criteria—future work could incorporate semantic segmentation, action detection, or hierarchical scene understanding to make selection even more task-aware.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::