Towards Spatio-Temporal World Scene Graph Generation from Monocular Videos
| Authors | Rohith Peddi et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.13185 |
| Download | |
| Categories | cs.CV |
Abstract
Spatio-temporal scene graphs provide a principled representation for modeling evolving object interactions, yet existing methods remain fundamentally frame-centric: they reason only about currently visible objects, discard entities upon occlusion, and operate in 2D. To address this, we first introduce ActionGenome4D, a dataset that upgrades Action Genome videos into 4D scenes via feed-forward 3D reconstruction, world-frame oriented bounding boxes for every object involved in actions, and dense relationship annotations including for objects that are temporarily unobserved due to occlusion or camera motion. Building on this data, we formalize World Scene Graph Generation (WSGG), the task of constructing a world scene graph at each timestamp that encompasses all interacting objects in the scene, both observed and unobserved. We then propose three complementary methods, each exploring a different inductive bias for reasoning about unobserved objects: PWG (Persistent World Graph), which implements object permanence via a zero-order feature buffer; MWAE (Masked World Auto-Encoder), which reframes unobserved-object reasoning as masked completion with cross-view associative retrieval; and 4DST (4D Scene Transformer), which replaces the static buffer with differentiable per-object temporal attention enriched by 3D motion and camera-pose features. We further design and evaluate the performance of strong open-source Vision-Language Models on the WSGG task via a suite of Graph RAG-based approaches, establishing baselines for unlocalized relationship prediction. WSGG thus advances video scene understanding toward world-centric, temporally persistent, and interpretable scene reasoning.
Engineering Breakdown
Plain English
This paper tackles the problem of understanding dynamic scenes from monocular video by generating world scene graphs that track all interacting objects over time, even when they're temporarily occluded or out of frame. Existing methods fail because they operate frame-by-frame in 2D and discard objects when the camera moves or they disappear. The authors introduce ActionGenome4D, a dataset that converts Action Genome videos into 4D scenes using feed-forward 3D reconstruction and world-frame bounding boxes, plus dense relationship annotations for occluded objects. They then formalize World Scene Graph Generation (WSGG), a new task that constructs complete scene graphs at each timestamp encompassing all interacting objects, addressing the fundamental limitation of current frame-centric approaches.
Core Technical Contribution
The core innovation is shifting from frame-centric to world-centric scene graph reasoning—maintaining a persistent representation of objects and their relationships in world coordinates rather than camera coordinates, which requires tracking objects through occlusions and camera motion. The technical contribution includes the ActionGenome4D dataset itself, which bridges the gap between 2D action recognition datasets and full 3D spatiotemporal understanding by adding 4D reconstruction (3D space + time) and explicit annotations for occluded objects. The formalization of WSGG as a distinct task (not just frame-by-frame scene graph generation) establishes a new evaluation paradigm where the model must reason about objects that are currently invisible but still interacting. This requires architectural changes to maintain world state across frames and handle incomplete observations, moving beyond existing approaches that only reason about visible entities.
How It Works
The pipeline starts with monocular video input and performs feed-forward 3D reconstruction to establish world-frame geometry for each frame. For every detected object, the system generates world-frame oriented bounding boxes (rather than image-space boxes) so that the same object maintains consistent spatial identity across frames despite camera motion. The core WSGG model maintains a persistent world scene representation—likely a graph where nodes are objects with world coordinates and edges are spatial/temporal relationships. At each timestamp, the model must predict which objects exist in the world (including occluded ones), their 3D positions and interactions, and update relationships as objects move or interact. The key mechanism is decoupling object identity and spatial relationships from camera viewpoint: instead of asking 'what's visible now?', the model asks 'what exists in the world and how do they relate?'. Dense relationship annotations for occluded objects during training provide supervision signal for reasoning about temporarily unobserved entities.
Production Impact
For robotics and autonomous systems, this approach enables more robust scene understanding because the world model persists across occlusions and viewpoint changes—critical for tasks like manipulation where you need to track objects even when they're temporarily hidden. In surveillance and activity understanding, maintaining complete world scene graphs allows for better long-horizon reasoning about complex multi-object interactions (group activities, collaborative tasks) rather than inferring only from visible frames. The trade-off is significant: you now require 3D reconstruction as a preprocessing step, which adds computational overhead (feed-forward 3D reconstruction must be run per frame), and the world-space representation requires substantially more memory to maintain object state. For deployment, this means higher GPU memory requirements and increased latency per frame—not suitable for real-time edge applications but viable for offline video analysis or systems with compute resources (cloud-based video analysis). Integration requires retooling downstream tasks to work with world-space graphs rather than image-space graphs, which is a non-trivial change to existing action recognition or interaction understanding pipelines.
Limitations and When Not to Use This
The approach critically depends on accurate 3D reconstruction from monocular video, which remains fundamentally ambiguous—scale and depth are not fully recoverable, meaning the world-space representation may not align perfectly with true geometry. The reliance on dense annotations for occluded objects makes the ActionGenome4D dataset expensive to create and limits scalability; extending this to new domains requires substantial annotation effort. The paper doesn't clarify how the method handles long-term occlusions (what if an object is hidden for 10+ seconds?) or temporary objects entering/leaving the scene, which are common in real-world video. Computational cost is also under-explored—the paper doesn't provide latency numbers or memory requirements compared to frame-centric baselines, making it unclear whether this is practical for long videos or real-time applications. Finally, the evaluation is limited to Action Genome videos, which have relatively clean, actor-centric content; generalization to cluttered scenes, outdoor environments, or crowded scenarios is unvalidated.
Research Context
This work builds on the scene graph generation literature (which has focused on single-image understanding) and extends it to temporal domains, drawing from video understanding and action recognition research. It addresses a fundamental limitation in existing video scene graphs that treat frames independently and cannot reason about occluded objects. The ActionGenome4D dataset is an upgrade of the Action Genome dataset (a widely-used benchmark for action/interaction understanding) into 3D/4D space, similar to how other works have progressively added depth and temporal information to foundational datasets. This opens a research direction toward persistent, world-centric representations for video understanding—moving away from the dominant frame-centric paradigm that has dominated computer vision, with potential implications for embodied AI, autonomous agents, and dynamic scene understanding.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::