Towards Long-Form Spatio-Temporal Video Grounding
| Authors | Xin Gu et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.23294 |
| Download | |
| Categories | cs.CV |
Abstract
In real scenarios, videos can span several minutes or even hours. However, existing research on spatio-temporal video grounding (STVG), given a textual query, mainly focuses on localizing targets in short videos of tens of seconds, typically less than one minute, which limits real-world applications. In this paper, we explore Long-Form STVG (LF-STVG), which aims to locate targets in long-term videos. Compared with short videos, long-term videos contain much longer temporal spans and more irrelevant information, making it difficult for existing STVG methods that process all frames at once. To address this challenge, we propose an AutoRegressive Transformer architecture for LF-STVG, termed ART-STVG. Unlike conventional STVG methods that require the entire video sequence to make predictions at once, ART-STVG treats the video as streaming input and processes frames sequentially, enabling efficient handling of long videos. To model spatio-temporal context, we design spatial and temporal memory banks and apply them to the decoders. Since memories from different moments are not always relevant to the current frame, we introduce simple yet effective memory selection strategies to provide more relevant information to the decoders, significantly improving performance. Furthermore, instead of parallel spatial and temporal localization, we propose a cascaded spatio-temporal design that connects the spatial decoder to the temporal decoder, allowing fine-grained spatial cues to assist complex temporal localization in long videos. Experiments on newly extended LF-STVG datasets show that ART-STVG significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG.
Engineering Breakdown
Plain English
This paper tackles spatio-temporal video grounding (STVG) for long-form videos—clips spanning minutes to hours—which existing methods struggle with because they try to process entire sequences at once. Current STVG research focuses on short videos under one minute, which doesn't match real-world scenarios where videos are much longer and contain significant irrelevant content. The authors propose ART-STVG, an autoregressive transformer architecture that processes video sequentially rather than all-at-once, making it feasible to ground text queries in videos that are orders of magnitude longer than prior benchmarks.
Core Technical Contribution
The core innovation is shifting STVG from a monolithic, all-frames-at-once processing paradigm to an autoregressive transformer approach that predicts target locations incrementally across time. This is fundamentally different from prior STVG methods that require loading the entire video sequence into memory and computing attention over all frames simultaneously—a constraint that breaks down when videos are hours long. The autoregressive formulation allows the model to ground queries in progressively longer temporal windows without architectural changes, treating long-form video grounding as a sequential prediction problem rather than a global optimization problem. This design choice enables computational tractability on realistic video lengths while maintaining the ability to leverage multi-modal (text-video) context.
How It Works
ART-STVG ingests three inputs: a long-form video (divided into frames or segments), a text query describing the target to localize, and a temporal window or context limit. The autoregressive transformer encodes the text query and progressively processes video segments, maintaining a sliding context window to manage memory. At each temporal step, the model predicts whether the target appears in the current segment and, if so, outputs bounding box coordinates (spatial grounding) and temporal boundaries. The key mechanism is that predictions at time t inform the computation at time t+1—rather than computing attention across all T frames globally (O(T²) complexity), the model only attends to recent context (O(k) where k is window size), reducing both memory and compute. The architecture likely uses a transformer decoder with causal masking and video/text fusion layers, where embeddings from video frames and text query are combined through cross-attention to produce spatiotemporal predictions.
Production Impact
For teams building video understanding systems, this approach makes long-form grounding computationally feasible on standard hardware—no need for massive distributed training or specialized accelerators for full-video processing. Real-world applications like video summarization, surveillance analytics, and content moderation can now handle hour-long clips instead of 30-60 second snippets, directly expanding the addressable problem space. The tradeoff is added latency: autoregressive processing means you must compute predictions sequentially rather than in one forward pass, potentially increasing end-to-end latency by 10-100x depending on video length and stride. Integration into existing video pipelines requires careful window management (deciding segment size and overlap), careful handling of query context decay (does information from minute 1 matter when grounding at minute 60?), and validation that streaming/online inference works as well as offline batch processing.
Limitations and When Not to Use This
The paper doesn't address how temporal context degrades—if you're grounding at minute 50, how much information from minute 1 is retained in the sliding window, and does that cause failures for queries requiring long-range reasoning? Autoregressive processing introduces ordering bias: the model might make different predictions if frames are presented in reverse or shuffled, and there's no discussion of robustness to frame reordering or missing frames. The approach assumes a fixed window size works across all video genres and query types, but a surveillance video with static scenes may need different window sizes than action-heavy content with rapid cuts. Finally, the paper doesn't evaluate on truly diverse long-form videos (the longest mentioned is implied to be under hours), so generalization to 4-8 hour videos, multiple camera cuts, or adversarial query distributions remains open.
Research Context
This work extends the spatio-temporal video grounding task, which has been studied in benchmarks like ActivityNet-Entities and YouCook2, where existing methods (typically using 3D CNNs or Transformers with global attention) achieve good results on short clips. The shift to long-form is motivated by the gap between academic benchmarks (typically <1 min) and real-world video—a pattern seen across video understanding (similar jumps occurred in video classification and action detection when moving from trimmed to untrimmed datasets). ART-STVG builds on the success of autoregressive transformers in other modalities (language, vision), applying that paradigm to the multimodal video-text grounding problem. This work opens research directions on efficient long-context multimodal learning, temporal context aggregation strategies, and benchmark creation for long-form STVG.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::