Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories
| Authors | Wonbong Jang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.09429 |
| Download | |
| Categories | cs.CV, cs.AI, cs.LG |
Abstract
Recovering camera parameters from images and rendering scenes from novel viewpoints have long been treated as separate tasks in computer vision and graphics. This separation breaks down when image coverage is sparse or poses are ambiguous, since each task needs what the other produces. We propose Rays as Pixels, a Video Diffusion Model (VDM) that learns a joint distribution over videos and camera trajectories. We represent each camera as dense ray pixels (raxels) and denoise them jointly with video frames through Decoupled Self-Cross Attention mechanism. A single trained model handles three tasks: predicting camera trajectories from video, jointly generating video and camera trajectory from input images, and generating video from input images along a target camera trajectory. Because the model can both predict trajectories from a video and generate views conditioned on its own predictions, we evaluate it through a closed-loop self-consistency test, demonstrating that its forward and inverse predictions agree. Notably, trajectory prediction requires far fewer denoising steps than video generation, even a few denoising steps suffice for self-consistency. We report results on pose estimation and camera-controlled video generation.
Engineering Breakdown
Plain English
This paper proposes Rays as Pixels, a video diffusion model that jointly learns camera parameters and novel view synthesis from sparse image coverage. Instead of treating camera pose estimation and video generation as separate problems, the authors unify them into a single model that can perform three tasks: predicting camera trajectories from video, jointly generating video and camera parameters from images, and generating video along a specified camera path. The key innovation is representing cameras as dense ray pixels (raxels) and denoising them together with video frames using a Decoupled Self-Cross Attention mechanism. A single trained model handles all three scenarios, which is significant because sparse or ambiguous poses typically require one task to feed results into the other.
Core Technical Contribution
The core novelty is decoupling the traditionally intertwined problems of camera pose estimation and novel view synthesis into a joint probabilistic framework using video diffusion. Rather than solving camera recovery first and then rendering (or vice versa), the authors treat both as part of the same generative process, learning a joint distribution over videos and camera trajectories. The technical innovation is the Decoupled Self-Cross Attention mechanism that allows the model to denoise camera ray representations (raxels) alongside video frames without information leakage between them. This architectural choice enables multi-task capability: the same trained model can work in different modes depending on what inputs are provided and what outputs are desired, which previous methods could not achieve with a single model.
How It Works
The model represents each camera viewpoint as a set of dense ray pixels (raxels)—essentially discretizing the camera's ray manifold into a learnable pixel-like representation. During the diffusion process, both video frames and raxel tensors are progressively denoised from Gaussian noise. The Decoupled Self-Cross Attention mechanism processes video frames and camera rays with separate self-attention branches that can exchange information only through carefully gated cross-attention, preventing the model from collapsing one modality into the other. The input can be: (1) a video sequence from which to infer camera motion, (2) a set of sparse images from which to jointly infer camera positions and generate intermediate frames, or (3) sparse images plus a target camera trajectory that guides video synthesis. During sampling, the diffusion reverse process iteratively refines both the video and camera parameters, with the attention mechanism ensuring they remain consistent with each other at every denoising step.
Production Impact
In production 3D reconstruction and novel-view-synthesis pipelines, this approach eliminates the need to maintain two separate models and their associated engineering complexity. Teams could replace multi-stage systems (camera tracker → novel-view generator → refinement) with a single inference pass, reducing latency and memory overhead. For sparse image datasets—common in robotics, autonomous vehicles, and photogrammetry—this model's ability to jointly optimize camera and scene geometry should improve reconstruction quality compared to pipelines where camera errors compound through stages. The trade-offs include computational cost: diffusion models require multiple denoising steps (typically 20-50), making inference slower than direct regression methods; you'd need substantial GPU memory to hold video tensors during denoising; and the model requires paired video/camera trajectory training data, which is expensive to collect at scale. Integration would require retraining on your specific domain's visual distribution and camera conventions, and handling different camera models (intrinsics) may require model variants.
Limitations and When Not to Use This
The paper assumes access to reasonably clean training data pairing video frames with ground-truth camera trajectories, which is expensive to obtain at production scale—synthetic data may not transfer well to real imagery. The approach inherits all limitations of diffusion models: slow inference due to iterative denoising (likely 5-10 seconds per video on standard hardware), sensitivity to sampling hyperparameters, and difficulty guaranteeing physically plausible camera trajectories in all cases. The method works best when image coverage is sparse but still informative; it may struggle with extreme viewpoint changes or severely occluded scenes. The paper does not address dynamic scenes or time-varying camera intrinsics, limiting applicability to static reconstruction tasks. Follow-up work is needed on efficient sampling strategies to reduce inference time, better geometric priors to prevent physically invalid solutions, and scalability to higher resolutions and longer video sequences.
Research Context
This work builds on recent advances in video diffusion models (like Video Diffusion Models by Ho et al.) and neural rendering techniques, extending them to the joint camera-recovery problem that has traditionally required separate modules (e.g., COLMAP for SfM, NeRF for rendering). It advances the broader trend of unifying downstream vision tasks through generative modeling, similar to how diffusion has unified image inpainting, editing, and generation. The paper likely improves on benchmarks like ScanNet or standard novel-view-synthesis datasets by showing that joint optimization recovers better geometry than sequential approaches. This opens a research direction toward end-to-end differentiable 3D understanding where all task-specific module boundaries dissolve, and generative models become the default tool for vision problems with ambiguous or incomplete data.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::