CityRAG: Stepping Into a City via Spatially-Grounded Video Generation
| Authors | Gene Chou et al. |
| Year | 2026 |
| HF Upvotes | 17 |
| arXiv | 2604.19741 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We address the problem of generating a 3D-consistent, navigable environment that is spatially grounded: a simulation of a real location. Existing video generative models can produce a plausible sequence that is consistent with a text (T2V) or image (I2V) prompt. However, the capability to reconstruct the real world under arbitrary weather conditions and dynamic object configurations is essential for downstream applications including autonomous driving and robotics simulation. To this end, we present CityRAG, a video generative model that leverages large corpora of geo-registered data as context to ground generation to the physical scene, while maintaining learned priors for complex motion and appearance changes. CityRAG relies on temporally unaligned training data, which teaches the model to semantically disentangle the underlying scene from its transient attributes. Our experiments demonstrate that CityRAG can generate coherent minutes-long, physically grounded video sequences, maintain weather and lighting conditions over thousands of frames, achieve loop closure, and navigate complex trajectories to reconstruct real-world geography.
Engineering Breakdown
Plain English
CityRAG is a video generation model that creates 3D-consistent, navigable environments grounded in real-world locations by leveraging large corpora of geo-registered data. Unlike existing text-to-video or image-to-video models that generate plausible sequences from prompts, CityRAG reconstructs real places under varying weather and object configurations by anchoring generation to physical scene context while maintaining learned priors for motion and appearance changes. The key innovation is using temporally unaligned training data—footage from the same location captured at different times—to teach the model how real environments evolve without requiring perfectly synchronized sequences. This approach enables applications like autonomous driving simulation and robotics testing by producing spatially grounded, physically consistent video of real-world locations.
Core Technical Contribution
The core novelty is a retrieval-augmented generation (RAG) framework applied to video synthesis that grounds generation in geo-registered spatial context rather than relying solely on text or image prompts. Most video generative models learn appearance and motion from paired examples; CityRAG uniquely exploits temporally unaligned multi-view data from the same real-world location to establish spatial consistency without requiring temporally aligned supervision. The technical contribution is a mechanism to condition a video diffusion model on retrieved geo-spatial context (from a large corpus of location-specific data) while preserving the model's ability to handle novel weather, lighting, and dynamic object variations. This bridges the gap between unconditioned generative models and precise scene reconstruction by treating real-world location data as a retrievable knowledge base that constrains and guides generation.
How It Works
The system operates in three main stages. First, given a query location (latitude/longitude or image) and generation parameters (weather, time of day, object configuration), the model retrieves relevant geo-registered video frames and metadata from a large indexed corpus of city-scale data. Second, these retrieved context frames are encoded into a spatial embedding that represents the scene geometry and appearance at that location, which is then injected into a video diffusion model through cross-attention conditioning layers (similar to how text conditions image generation in models like Stable Diffusion). Third, the conditioned diffusion model iteratively denoises a random video tensor, guided by both the spatial context embeddings and any additional constraints (e.g., 'rainy conditions'), generating a sequence of frames that maintains 3D consistency with the real location while incorporating the requested appearance variations. The temporally unaligned training data teaches the model to separate static geometry (building structure, street layout) from dynamic factors (weather, objects, lighting) so it can recombine them in novel ways during inference.
Production Impact
For autonomous driving and robotics teams, this enables generation of realistic, location-specific simulation data without expensive on-site filming under every weather condition or scenario—you could generate training data for your deployment city under rain, snow, or night conditions by querying the model with that location's coordinates. The geo-grounding ensures consistent street layouts and building structures across generated sequences, reducing sim-to-real transfer errors compared to generic video models that might hallucinate unrealistic geometry. Production adoption requires building an indexed geo-registered video corpus (requiring significant data collection and geo-tagging infrastructure), implementing the retrieval system with low-latency lookups, and integrating the diffusion inference pipeline—expect 5-30 second generation times per video on current hardware (A100 GPUs), making it suitable for offline dataset synthesis but not real-time control. The main trade-off is data density: the model performs best in well-covered urban areas with abundant existing footage; sparse or novel locations will degrade gracefully to vanilla text-to-video behavior.
Limitations and When Not to Use This
CityRAG assumes abundant geo-registered training data exists for target locations, making it impractical for rural areas, indoor environments, or newly developed regions with limited footage history. The approach doesn't explicitly handle occlusions or precise object-level control—if you need a specific car at a specific pixel location at a specific frame, this model provides less control than deterministic rendering engines. The temporal unaligned training data assumption may not hold for rapidly changing urban environments (construction sites, temporary installations); the model could generate outdated versions of locations. Fundamental open questions remain about quantifying 3D consistency beyond perceptual metrics and scaling to global coverage—how do you efficiently retrieve relevant context in a corpus of petabyte-scale geo-video data while maintaining inference latency?
Research Context
This work extends video diffusion models (building on Imagen Video and similar architectures) by incorporating retrieval-augmented generation, a technique proven effective in language models (RAG, Retrieval-Augmented Generation) and recently adapted to multimodal tasks. It addresses a gap in the video generation literature: most work focuses on semantic fidelity to prompts, while this tackles physical grounding to real locations—a requirement largely ignored by T2V/I2V papers until now. The paper opens a new research direction combining geo-spatial data indexing with generative models, likely to influence future work in location-aware synthesis, urban computing, and sim-to-real transfer. It also demonstrates that temporally unaligned data, which was previously considered problematic noise, can be valuable training signal when properly leveraged—inverting assumptions in prior video understanding work.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::