Envisioning the Future, One Step at a Time
| Authors | Stefan Andreas Baumann et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.09527 |
| Download | |
| Categories | cs.CV, cs.AI, cs.LG |
Abstract
Accurately anticipating how complex, diverse scenes will evolve requires models that represent uncertainty, simulate along extended interaction chains, and efficiently explore many plausible futures. Yet most existing approaches rely on dense video or latent-space prediction, expending substantial capacity on dense appearance rather than on the underlying sparse trajectories of points in the scene. This makes large-scale exploration of future hypotheses costly and limits performance when long-horizon, multi-modal motion is essential. We address this by formulating the prediction of open-set future scene dynamics as step-wise inference over sparse point trajectories. Our autoregressive diffusion model advances these trajectories through short, locally predictable transitions, explicitly modeling the growth of uncertainty over time. This dynamics-centric representation enables fast rollout of thousands of diverse futures from a single image, optionally guided by initial constraints on motion, while maintaining physical plausibility and long-range coherence. We further introduce OWM, a benchmark for open-set motion prediction based on diverse in-the-wild videos, to evaluate accuracy and variability of predicted trajectory distributions under real-world uncertainty. Our method matches or surpasses dense simulators in predictive accuracy while achieving orders-of-magnitude higher sampling speed, making open-set future prediction both scalable and practical. Project page: http://compvis.github.io/myriad.
Engineering Breakdown
Plain English
This paper tackles the problem of predicting how complex scenes will evolve over long time horizons by abandoning dense pixel-space or latent-space prediction in favor of sparse point trajectory modeling. Most existing video prediction models waste computational capacity on dense appearance reconstruction when the core challenge is actually predicting the sparse, underlying motion of objects and actors in a scene. The authors propose an autoregressive diffusion model that predicts future scene dynamics by stepping through sparse point trajectories one time step at a time, representing uncertainty explicitly and enabling efficient exploration of multiple plausible futures without the computational overhead of dense prediction.
Core Technical Contribution
The key novelty is reformulating multi-modal, long-horizon scene prediction as step-wise autoregressive inference over sparse point trajectories rather than dense video or latent codes. Instead of a model that predicts entire frames or compressed representations, this approach focuses computational resources exclusively on predicting where objects will be, treating motion prediction as a trajectory extrapolation problem. The use of a diffusion model operating in trajectory space allows the model to naturally represent the inherent uncertainty and multi-modality in future scene evolution—multiple plausible futures can be explored through different samples from the diffusion process. This is fundamentally different from prior dense-prediction methods because it decouples appearance modeling from motion modeling, allowing the model to specialize in the harder problem (predicting what-moves-where) without wasting capacity on reconstructing static or weakly-informative visual details.
How It Works
The model takes a scene representation and extracts or maintains a set of sparse point trajectories—essentially the 2D or 3D positions of key objects or actors in the scene across recent frames. At each prediction step, the diffusion model conditions on the current trajectory state and iteratively denoises a trajectory prediction, advancing all points forward by one short time step. These locally predictable transitions are chained together autoregressively: the predicted trajectory at step t becomes part of the context for predicting step t+1, allowing the model to build up long-horizon predictions by composing many short, easier steps. The diffusion formulation provides a principled way to represent uncertainty—different noise samples during the diffusion process generate different trajectory realizations, naturally producing a distribution over plausible futures. This sparse, trajectory-centric approach reduces the input dimensionality and memory footprint compared to dense prediction, making it feasible to explore many hypotheses or perform long-horizon rollouts without prohibitive computational cost.
Production Impact
In a production system, this approach would replace dense video prediction pipelines (which typically require large compute for frame generation) with a leaner trajectory prediction engine, dramatically reducing inference latency and memory footprint when the downstream task cares primarily about object motion and interactions. For applications like autonomous driving, robotics, or scene understanding, you would feed raw video or frame sequences into a point-tracking module (or use tracked detections), then run this diffusion model to predict future object positions and interactions—enabling efficient multi-hypothesis planning or risk assessment. The trade-off is that you lose high-fidelity visual detail about appearance changes, occlusions, or lighting—acceptable for planning and prediction tasks but problematic if your pipeline requires realistic future video frames for human review or pixel-level downstream tasks. Integration is straightforward: the model accepts sparse input (point coordinates and trajectories) and outputs point positions, fitting naturally between a perception/tracking front-end and a downstream planning or decision module. The explicit uncertainty representation through diffusion sampling enables better uncertainty quantification in planning, which is crucial for risk-aware decision making in safety-critical domains.
Limitations and When Not to Use This
The paper's primary limitation is its dependence on reliable point tracking or correspondence—if the upstream tracking module fails, drifts, or loses tracks, the trajectory prediction model has no ground truth to work with, and errors compound over long horizons. The approach may struggle with novel object interactions, open-set scenarios with unknown actor types, or situations requiring detailed appearance reasoning (e.g., predicting whether a person will trip based on subtle posture cues that aren't captured by sparse points). Long-horizon prediction (beyond a few seconds) still faces the fundamental challenge that trajectory uncertainty grows exponentially; the paper's step-wise approach mitigates but does not solve the divergence problem inherent to autoregressive prediction. Additionally, the method assumes scenes are well-described by point motion and may not handle scene changes like occlusions, deformations, or topological changes elegantly—sparse point trajectories lose contact with occluded objects, and reconstructing visibility or handling contact dynamics is non-trivial.
Research Context
This work builds on the growing trend of moving away from dense pixel-space prediction toward sparse, structured scene representations (related to neural radiance fields, 3D object detection, and trajectory forecasting). It directly addresses critiques of prior video prediction work (like those from the video generation community) that highlighted the computational inefficiency of predicting dense pixels when motion and dynamics are sparse. The paper likely advances over prior point trajectory work (such as TAP-Vid or similar point-tracking benchmarks) by adding principled uncertainty quantification through diffusion and demonstrating that trajectory prediction alone, without dense appearance, can scale to complex, diverse scenes. The sparse trajectory formulation opens a new direction for scene prediction that naturally connects to physics-based simulation, differentiable rendering, and world models for embodied AI, suggesting that future work may combine this trajectory prediction with learned dynamics priors or neural physical models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::