Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction
| Authors | Efstathios Karypidis et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2604.11707 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix, a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency compared to strong diffusion baselines. We provide the implementation code at https://github.com/Sta8is/Re2Pix
Engineering Breakdown
Plain English
Re2Pix tackles the hard problem of predicting future video frames in complex dynamic scenes like autonomous driving by splitting the task into two stages: first predicting what the scene will look like semantically (using a frozen vision foundation model's feature space), then generating photorealistic RGB frames conditioned on those predictions using a latent diffusion model. This two-stage decomposition lets the model handle scene dynamics and appearance generation separately, which is harder to do when predicting pixels directly. The key innovation is leveraging pre-trained vision models as semantic anchors to maintain consistency rather than trying to predict everything end-to-end in pixel space, addressing both visual fidelity and scene semantic accuracy simultaneously.
Core Technical Contribution
The core novelty is the hierarchical decomposition of video prediction into semantic representation prediction followed by appearance synthesis, rather than direct pixel prediction. By freezing a foundation model and predicting in its feature space first, the approach decouples scene structure forecasting from pixel-level rendering, allowing each stage to optimize for different objectives. This is fundamentally different from prior end-to-end pixel prediction methods (like ConvLSTM or transformer-based approaches) because it leverages the semantic understanding already learned by large pre-trained vision models as an inductive bias. The use of latent diffusion as a conditional decoder on predicted representations is a technically sound way to handle the complexity of photorealistic synthesis while maintaining grounding in semantically consistent feature space.
How It Works
The pipeline operates as follows: (1) Input: a sequence of past video frames is encoded using a frozen vision foundation model to extract semantic representations (learned features, not RGB pixels). (2) Prediction stage: a temporal forecasting module predicts future semantic representations in the foundation model's feature space, operating on higher-level structure rather than pixel coordinates. (3) Synthesis stage: a latent diffusion model is conditioned on these predicted semantic representations and generates future RGB frames in latent space, then decoded back to pixel space. (4) The foundation model remains frozen throughout training, preventing catastrophic forgetting of its semantic understanding while the forecasting and synthesis modules learn task-specific dynamics. This hierarchical structure means the first stage focuses on 'what will happen' (scene structure) and the second stage handles 'how it will look' (photorealistic detail), reducing the burden on any single model to simultaneously understand physics, semantics, and pixel-level statistics.
Production Impact
For autonomous driving systems and other safety-critical applications requiring future prediction, this approach provides two major benefits: (1) improved consistency of semantic scene understanding, which is critical for planning and safety modules downstream, and (2) better visual fidelity for simulation and testing. In production, you would replace your existing video prediction backbone with this two-stage system, meaning your infrastructure needs to support frozen foundation model inference (compute-light but memory-intensive) plus diffusion-based synthesis (slower than deterministic decoders but higher quality). The main trade-off is latency: diffusion sampling adds non-trivial inference time compared to direct pixel prediction methods, requiring evaluation of whether your system can tolerate this (e.g., 100-200ms per frame for diffusion vs. 10-50ms for feedforward networks). Data requirements are moderate since you leverage pre-trained models, but you still need video sequences with semantic ground truth labels for proper evaluation, making this most practical for domains with existing annotation infrastructure.
Limitations and When Not to Use This
The abstract hints at a 'train-test mismatch' problem that isn't fully explained, likely referring to the gap between training with clean foundation model encodings and testing with the actual predicted representations (which accumulate error over time). The approach assumes access to a suitable frozen foundation model for your domain—if you're working with specialized sensors (thermal, lidar) or very different visual domains than what the foundation model was trained on, the semantic anchors may be misaligned. The reliance on latent diffusion means you inherit all its limitations: stochastic output (non-deterministic predictions), computational overhead, and the need to tune sampling parameters per application. Additionally, the paper doesn't discuss how to handle occlusions, out-of-distribution objects, or distribution shifts that occur between training data and real-world autonomous driving scenarios, which are critical for safety-critical deployment.
Research Context
This work builds on the established line of hierarchical video prediction research (prior work like Denton et al., Babaeizadeh et al.) but incorporates recent advances in large vision foundation models and diffusion-based generation. It sits at the intersection of three active research areas: (1) foundation models as frozen feature extractors for downstream tasks, (2) diffusion models for conditional image/video synthesis, and (3) future frame prediction for autonomous systems. The contribution is timely given the shift toward leveraging pre-trained models rather than training everything end-to-end, and it opens research directions around better temporal consistency in diffusion decoding, handling the train-test mismatch more robustly, and scaling to longer prediction horizons. Benchmarks likely include standard video prediction datasets (likely KITTI for autonomous driving, possibly UCF-101 or Kinetics subsets for general video), with metrics spanning both pixel-level quality (SSIM, LPIPS) and semantic consistency (segmentation accuracy on predicted frames).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::