Physics Informed Viscous Value Representations
| Authors | Hrishikesh Viswanath et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2602.23280 |
| Download | |
| Categories | cs.LG, cs.RO |
Abstract
Offline goal-conditioned reinforcement learning (GCRL) learns goal-conditioned policies from static pre-collected datasets. However, accurate value estimation remains a challenge due to the limited coverage of the state-action space. Recent physics-informed approaches have sought to address this by imposing physical and geometric constraints on the value function through regularization defined over first-order partial differential equations (PDEs), such as the Eikonal equation. However, these formulations can often be ill-posed in complex, high-dimensional environments. In this work, we propose a physics-informed regularization derived from the viscosity solution of the Hamilton-Jacobi-Bellman (HJB) equation. By providing a physics-based inductive bias, our approach grounds the learning process in optimal control theory, explicitly regularizing and bounding updates during value iterations. Furthermore, we leverage the Feynman-Kac theorem to recast the PDE solution as an expectation, enabling a tractable Monte Carlo estimation of the objective that avoids numerical instability in higher-order gradients. Experiments demonstrate that our method improves geometric consistency, making it broadly applicable to navigation and high-dimensional, complex manipulation tasks. Open-source codes are available at https://github.com/HrishikeshVish/phys-fk-value-GCRL.
Engineering Breakdown
Plain English
This paper addresses a critical problem in offline goal-conditioned reinforcement learning: accurately estimating value functions when you only have a static dataset that doesn't cover the full state-action space. The authors propose using physics-informed regularization based on viscosity solutions of the Hamilton-Jacobi-Bellman (HJB) equation to constrain value function learning. Rather than using simpler PDE constraints like the Eikonal equation (which can be ill-posed in high dimensions), their approach grounds value estimation in the underlying physics of optimal control, providing a more theoretically sound inductive bias for learning in sparse-data regimes.
Core Technical Contribution
The core novelty is replacing ad-hoc physics-informed regularization with a principled viscosity solution framework derived from the HJB equation, which governs optimal control in continuous spaces. Previous work applied Eikonal equation constraints, but this becomes mathematically ill-posed in complex, high-dimensional settings. By leveraging viscosity solutions—a well-established theory in optimal control that handles non-smooth value functions—the authors provide a mathematically rigorous way to regularize value estimation that better captures the underlying structure of optimal policies. This is fundamentally different because it's not just adding a geometric constraint; it's enforcing that the learned value function satisfies the actual differential equation governing optimal control.
How It Works
The method works by formulating offline GCRL as a constrained optimization problem where the value function must approximate solutions to the HJB equation. Given a pre-collected dataset of state-action-reward tuples, the learning process simultaneously fits a neural network value function V(s, g) (predicting expected returns to goal g from state s) while satisfying viscosity solution constraints. The viscosity formulation allows the regularizer to handle non-differentiable value functions that arise in optimal control—these are specified through inequalities rather than equations, making them numerically stable. The regularization term is added to the standard RL loss, penalizing violations of the HJB constraint. During inference, the learned value function guides policy extraction through gradient ascent or similar optimization, with the physics constraint ensuring that extrapolation to unseen states remains physically plausible rather than producing unbounded or degenerate estimates.
Production Impact
For engineers building offline RL systems (robotics, autonomous vehicles, recommendation systems), this addresses a major practical pain point: offline learning from logged data without online interaction often produces value estimates that catastrophically fail on out-of-distribution states. Adopting this approach would replace standard value regularization (like L2 penalties or uncertainty bounds) with physics-informed constraints, requiring integration of viscosity solution solvers into the training loop. The concrete benefit is more robust extrapolation and fewer distribution shift failures in deployment, though at the cost of additional computational overhead during training—you're solving PDE constraints in addition to fitting neural networks. This is most valuable in high-stakes domains (robotics, safety-critical control) where value function failures directly cause system failures; it's less critical in data-rich regimes where you can learn directly without extrapolation.
Limitations and When Not to Use This
The approach assumes access to a reward function and that the underlying optimal control problem is well-modeled by the HJB equation—it will fail or provide no benefit if rewards are sparse, delayed, or fundamentally misaligned with control objectives. The viscosity solution formulation adds theoretical guarantees but requires solving PDEs in high-dimensional spaces, which becomes computationally prohibitive (curse of dimensionality) and may introduce numerical instability at scale. The paper is currently in abstract form, so critical details are missing: how exactly the viscosity constraints are discretized for neural networks, what computational overhead is incurred, and whether the approach actually outperforms simpler baselines on realistic benchmarks with 50+ dimensional state spaces. Additionally, the method requires the offline dataset to have reasonable coverage of important states; in extremely sparse coverage regimes, no amount of physics-informed regularization can overcome the underlying data poverty.
Research Context
This builds on a recent trend of physics-informed machine learning in control, extending prior work that applied Eikonal equation constraints and other PDE-based regularizations to RL value functions. It directly improves upon approaches like constraint-based value learning by leveraging deeper optimal control theory—specifically, viscosity solution theory developed in the 1980s-1990s for analyzing non-smooth value functions in deterministic optimal control. The work fits within the broader offline RL research area, which has matured significantly since 2020 with methods like conservative Q-learning and IQL, and represents a push toward more theoretically grounded and extrapolation-safe approaches. This likely opens a research direction around other optimal control PDEs (e.g., risk-sensitive variants, stochastic control equations) as inductive biases for deep RL, and may influence how the community thinks about physics-informed inductive biases in learning-based control.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::