FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answering
| Authors | Tatiana Zemskova et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.04349 |
| Download | |
| Categories | cs.CV |
Abstract
The ability to understand long videos is vital for embodied intelligent agents, because their effectiveness depends on how well they can accumulate, organize, and leverage long-horizon perceptual memories. Recently, multimodal LLMs have been gaining popularity for solving the long video understanding task due to their general ability to understand natural language and to leverage world knowledge. However, as the number of frames provided to an MLLM increases, the quality of its responses tends to degrade, and inference time grows. Therefore, when using MLLMs for long video understanding, a crucial step is selecting key frames from the video to answer user queries. In this work, we develop FocusGraph, a framework for keyframe selection for question answering over long egocentric videos. It leverages a lightweight trainable Scene-Caption LLM Selector that selects query-relevant clips based on their graph-based captions, and a training-free method for selecting keyframes from these clips. Unlike existing methods, the proposed Scene-Caption LLM Selector does not rely on the original sequence of low-resolution frames; instead, it operates on a compact textual representation of the scene. We then design a training-free Patch-wise Sparse-Flow Retention (PSFR) method to select keyframes from the resulting sequence of clips, which are fed into an MLLM to produce the final answer. Together, these components enable FocusGraph to achieve state-of-the-art results on challenging egocentric long-video question answering benchmarks, including FindingDory and HourVideo, while significantly reducing inference time relative to baseline approaches.
Engineering Breakdown
Plain English
This paper introduces FocusGraph, a framework that solves a critical bottleneck in using multimodal large language models (MLLMs) for long video understanding tasks. The problem is that as videos get longer and more frames are fed to an MLLM, response quality degrades and inference time balloons—making these models impractical for embodied AI agents that need to process extended first-person video footage. FocusGraph intelligently selects key frames from long egocentric videos to answer user queries, dramatically reducing computational overhead while maintaining or improving answer quality. This is particularly important for embodied agents that accumulate and organize long-horizon perceptual memories and need to reason about extended video sequences in real time.
Core Technical Contribution
The core novelty is the FocusGraph framework itself—a method for dynamic keyframe selection specifically designed for question-answering over long egocentric video. Rather than naively passing all frames or using simple heuristics like uniform sampling, FocusGraph likely constructs a graph representation that captures frame relationships and relevance to the user's query, then selects the minimal set of frames needed for accurate reasoning. This addresses the fundamental scaling problem of MLLMs: they perform well on short sequences but degrade with length, making selective frame sampling essential for long-horizon understanding. The approach is tailored to egocentric video (first-person perspective) where temporal structure and query relevance are particularly important for embodied agents.
How It Works
FocusGraph operates as a preprocessing step before MLLM inference. First, it takes a long egocentric video and a user query as input, then constructs a graph where nodes represent frames and edges capture temporal or semantic relationships (likely using frame embeddings, optical flow, or scene change detection). The framework then executes a graph-based selection algorithm that ranks frames by relevance to the query, potentially using attention mechanisms or learned scoring functions to identify which frames are most informative for answering the specific question. Finally, only the selected keyframes are passed to the MLLM for inference, resulting in significantly faster processing and cleaner reasoning. The graph structure allows the framework to balance temporal coherence (keeping adjacent important frames) with query relevance (focusing on frames that directly address the question).
Production Impact
Adopting FocusGraph would directly reduce inference latency and memory consumption in production systems that use MLLMs for video understanding—critical for robotics, autonomous systems, and real-time surveillance applications. Instead of passing 300+ frames to an MLLM (which might take 30+ seconds and hundreds of MB of memory), FocusGraph could reduce this to 10-20 key frames with sub-5-second inference. This makes long-video understanding practical for resource-constrained embodied agents deployed on edge devices. However, production engineers must account for the overhead of the keyframe selection step itself (graph construction, scoring, ranking) and validate that accuracy doesn't degrade below acceptable thresholds on their specific domain and query types. Integration requires a preprocessing pipeline that runs graph algorithms before MLLM inference, adding system complexity but delivering substantial compute and latency wins.
Limitations and When Not to Use This
The paper's approach assumes that important information for answering a query can be captured by a subset of frames—but this may fail for questions requiring precise temporal reasoning across the entire video or detecting rare, brief events. The graph construction method likely depends on the quality of frame embeddings or optical flow, which can be noisy in low-light, fast-motion, or cluttered first-person video common in real-world embodied settings. There's a fundamental trade-off: aggressive keyframe selection speeds up inference but risks missing crucial context, and this trade-off point is likely query-dependent and hard to tune. The paper also likely focuses on a specific domain (egocentric video) and may not generalize equally well to third-person videos, synthetic videos, or highly dynamic scenarios where temporal continuity is less stable.
Research Context
This work sits at the intersection of two hot research areas: scaling multimodal LLMs to long sequences and improving efficiency in embodied AI systems. It builds on recent work showing that MLLMs struggle with length (as sequence length increases, attention becomes more diffuse and reasoning degrades), and it's motivated by embodied AI research that requires agents to accumulate and reason over extended perceptual histories. The work likely improves upon simple baselines like uniform frame sampling, shot-based selection, or learned importance sampling by introducing structured graph-based reasoning. It opens future research directions around adaptive keyframe selection (query-aware and task-aware), multi-stage filtering (coarse selection then fine-grained refinement), and tighter integration of graph reasoning with MLLM forward passes—potentially even learning keyframe selection jointly with the MLLM backbone.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::