Representation Learning for Spatiotemporal Physical Systems
| Authors | Helen Qu et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2603.13227 |
| Download | |
| Categories | cs.LG, cs.CV |
Abstract
Machine learning approaches to spatiotemporal physical systems have primarily focused on next-frame prediction, with the goal of learning an accurate emulator for the system's evolution in time. However, these emulators are computationally expensive to train and are subject to performance pitfalls, such as compounding errors during autoregressive rollout. In this work, we take a different perspective and look at scientific tasks further downstream of predicting the next frame, such as estimation of a system's governing physical parameters. Accuracy on these tasks offers a uniquely quantifiable glimpse into the physical relevance of the representations of these models. We evaluate the effectiveness of general-purpose self-supervised methods in learning physics-grounded representations that are useful for downstream scientific tasks. Surprisingly, we find that not all methods designed for physical modeling outperform generic self-supervised learning methods on these tasks, and methods that learn in the latent space (e.g., joint embedding predictive architectures, or JEPAs) outperform those optimizing pixel-level prediction objectives. Code is available at https://github.com/helenqu/physical-representation-learning.
Engineering Breakdown
Plain English
This paper reframes how we evaluate machine learning models for simulating physical systems. Rather than measuring success solely on next-frame prediction accuracy (which suffers from compounding errors during long rollouts), the authors propose evaluating models on downstream physics tasks like estimating a system's governing parameters. This approach provides a more physically meaningful way to assess whether models have actually learned the underlying physics, not just memorized visual patterns. The work evaluates self-supervised learning methods as a way to learn these physics-grounded representations more efficiently than supervised approaches.
Core Technical Contribution
The key innovation is shifting the evaluation paradigm away from autoregressive next-frame prediction error toward downstream physics estimation tasks that directly measure physical understanding. Rather than building yet another transformer-based video predictor, the authors treat representation learning as the primary objective—asking what properties a learned representation must have to solve physics problems beyond simple frame synthesis. They demonstrate that self-supervised methods can learn these physics-grounded representations effectively, and that performance on parameter estimation tasks is a more reliable signal of whether a model has captured true physical dynamics versus overfitting to visual details. This reframing exposes a fundamental limitation in prior work: a model can predict frames accurately while having learned almost nothing about the actual physics.
How It Works
The approach takes sequences of spatiotemporal observations from physical systems as input and processes them through a representation learning backbone (trained with self-supervised objectives) to produce embeddings that capture system state. These learned representations are then evaluated on multiple downstream tasks: the primary task is estimating governing physical parameters (e.g., elasticity, viscosity, gravitational constant) from trajectories, which requires understanding the causal dynamics rather than surface-level correlations. The paper likely uses a two-stage pipeline where the encoder is first trained with contrastive or other self-supervised losses on raw videos, then frozen feature extractors feed into lightweight prediction heads for parameter estimation. By measuring parameter estimation accuracy, the authors can directly compare how well different representation learning methods capture the true underlying physics, with better representations enabling more accurate parameter recovery. The self-supervised pretraining avoids the need for expensive labeled physics annotations while still learning representations validated against ground-truth physical laws.
Production Impact
This approach directly addresses a critical pain point in physics-informed ML: production systems need models that generalize to novel initial conditions and longer rollouts, but next-frame prediction metrics don't guarantee this generalization. By validating representations against physics parameter estimation, engineers can detect when a model has merely fit visual patterns versus learning transferable physical understanding—critical for safety-critical applications like climate modeling or structural simulation. The efficiency gain is substantial: self-supervised pretraining requires no manual physics annotations, reducing the annotation burden compared to supervised approaches. However, there's a computational trade-off: you need access to diverse physical systems during pretraining to learn general representations, and parameter estimation evaluation requires ground truth physics values (which may not always be available in production). Integration into existing pipelines requires replacing the standard video prediction loss with a multi-task objective that includes downstream physics supervision, and care must be taken to ensure the learned parameters are actually meaningful for the application domain.
Limitations and When Not to Use This
The paper's reliance on parameter estimation as a proxy for physical understanding assumes that recovering accurate parameters is both necessary and sufficient for downstream applications—this may not hold for all physics tasks (e.g., some applications only care about qualitative behavior, not quantitative parameter values). The approach requires access to systems where ground-truth governing parameters are known during evaluation, limiting applicability to well-understood physical domains; real-world systems with unknown or ill-defined parameters may not benefit from this evaluation metric. Self-supervised pretraining requires sufficient diversity of physical phenomena during training, and performance likely degrades significantly on out-of-distribution physics (e.g., novel regime parameters or interaction types not seen in pretraining data). The paper appears incomplete in the abstract—the statement about self-supervised methods trails off without stating actual results or quantitative improvements, making it unclear what the empirical gains are or whether self-supervised learning actually outperforms simpler baselines.
Research Context
This work addresses a well-known problem in neural physics simulators: the divergence between next-frame prediction metrics and actual physical accuracy, which has been studied in video prediction literature for years but rarely tackled directly. It builds on the self-supervised learning movement (contrastive methods like SimCLR, momentum contrast) and applies them to a new domain—physical system understanding—where downstream task performance provides inherent validation. The paper challenges the implicit assumption in the field that next-frame prediction loss is the right objective, opening up research into what representations actually matter for physics. This direction aligns with emerging interest in evaluating foundation models on physics understanding rather than visual fidelity, potentially inspiring new benchmarks for physics-aware representation learning beyond standard datasets like TaylorGreen or SinusoidalGaussian.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::