RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework
| Authors | Hao Gao et al. |
| Year | 2026 |
| HF Upvotes | 26 |
| arXiv | 2604.15308 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
High-level autonomous driving requires motion planners capable of modeling multimodal future uncertainties while remaining robust in closed-loop interactions. Although diffusion-based planners are effective at modeling complex trajectory distributions, they often suffer from stochastic instabilities and the lack of corrective negative feedback when trained purely with imitation learning. To address these issues, we propose RAD-2, a unified generator-discriminator framework for closed-loop planning. Specifically, a diffusion-based generator is used to produce diverse trajectory candidates, while an RL-optimized discriminator reranks these candidates according to their long-term driving quality. This decoupled design avoids directly applying sparse scalar rewards to the full high-dimensional trajectory space, thereby improving optimization stability. To further enhance reinforcement learning, we introduce Temporally Consistent Group Relative Policy Optimization, which exploits temporal coherence to alleviate the credit assignment problem. In addition, we propose On-policy Generator Optimization, which converts closed-loop feedback into structured longitudinal optimization signals and progressively shifts the generator toward high-reward trajectory manifolds. To support efficient large-scale training, we introduce BEV-Warp, a high-throughput simulation environment that performs closed-loop evaluation directly in Bird's-Eye View feature space via spatial warping. RAD-2 reduces the collision rate by 56% compared with strong diffusion-based planners. Real-world deployment further demonstrates improved perceived safety and driving smoothness in complex urban traffic.
Engineering Breakdown
Plain English
RAD-2 addresses a fundamental problem in autonomous driving: motion planners must generate diverse future trajectory predictions while remaining stable during closed-loop execution and avoiding the reward collapse that occurs in pure imitation learning. The paper proposes a two-stage architecture where a diffusion model generates multiple candidate trajectories, and a reinforcement learning-trained discriminator ranks these candidates by their long-term driving quality. This decoupled approach avoids the instability of directly applying sparse scalar RL rewards to high-dimensional trajectory distributions, instead using the discriminator as a reranking mechanism. The framework unifies generative modeling (diversity through diffusion) with RL-based evaluation (quality through learned discrimination).
Core Technical Contribution
The core novelty is the generator-discriminator decomposition for closed-loop autonomous driving planning, which breaks the traditional end-to-end RL pipeline into two interpretable stages with different optimization objectives. Rather than training a single model to both generate trajectories and optimize them with RL rewards, RAD-2 keeps the generative model (diffusion) focused on modeling trajectory distributions through imitation learning, then applies RL only to a learned discriminator that scores trajectories. This design avoids the curse of dimensionality when applying sparse rewards to full trajectory sequences—a key failure mode in naive RL approaches to planning. The discriminator can be efficiently trained because it consumes structured candidate scores rather than high-dimensional trajectory tensors, making RL optimization more stable and sample-efficient.
How It Works
The pipeline operates in three stages: (1) A diffusion-based generator, pre-trained on expert driving demonstrations, produces N diverse trajectory candidates by iteratively denoising from Gaussian noise conditioned on current state and road context. (2) A discriminator network—trained with RL—consumes each trajectory and outputs a scalar Q-value predicting long-term driving quality (safety, comfort, efficiency, time-to-goal). (3) The discriminator reranks the N candidates and the top-K trajectories are selected for execution. During training, the discriminator receives rewards from closed-loop simulation rollouts: trajectories that lead to successful navigation and avoid collisions receive positive signal, while those causing accidents or inefficiency receive negative signal. The generator remains frozen during discriminator training, so it continues modeling the true multimodal distribution of expert trajectories rather than collapsing toward a single mode. The RL objective for the discriminator is to maximize cumulative driving reward (collision avoidance, progress toward goal, jerk minimization, etc.) over fixed-length rollouts.
Production Impact
This architecture directly solves three production pain points in autonomous driving systems: (1) Multimodal prediction—the diffusion generator naturally captures the uncertainty in future trajectories (multiple valid paths through intersections, lane changes, obstacle avoidance), unlike deterministic planners; (2) Closed-loop stability—by keeping the generative model static and only training the discriminator, the system avoids mode collapse where RL rewards push the generator toward a single high-reward trajectory, which fails catastrophically when that trajectory becomes invalid; (3) Sample efficiency—reranking N trajectories with a lightweight discriminator is far more sample-efficient than training a monolithic RL planner on full trajectory space. In practice, engineers would replace their rule-based or simple neural trajectory selectors with this discriminator, feeding it candidates from any generator (diffusion, VAE, or ensemble sampling). Compute-wise, you pay the cost of diffusion sampling (multiple forward passes through a large model) plus one discriminator forward pass per candidate, which is tractable on modern vehicle compute (inference latency likely 50-200ms depending on N). The training pipeline requires substantial closed-loop simulation data, but avoids the label scarcity problem since rewards are automatically computed from simulation rollouts.
Limitations and When Not to Use This
The paper's approach critically depends on having a good generator pre-trained on imitation learning; if the diffusion model doesn't already capture safe trajectories, the discriminator cannot rerank its way to safety. The method assumes the discriminator's learned reward function generalizes to novel driving scenarios not seen during training—distribution shift in edge cases (construction zones, new road types) could degrade performance. The decoupling between generator and discriminator may lose information: if the discriminator identifies a failure pattern, the generator cannot adapt because it's frozen, so the system can only mask generator failures rather than correct root causes. Additionally, the paper does not address computational tractability at scale—sampling N trajectories and evaluating each through the discriminator multiplies latency; the optimal N and discriminator architecture tradeoffs are likely dataset-dependent and underexplored. Finally, there's no principled way to set N or determine how many candidates the discriminator needs to see to avoid filtering out the true optimal trajectory; this hyperparameter likely requires empirical tuning per scenario.
Research Context
RAD-2 builds on two mature research directions: (1) diffusion models for trajectory generation (following recent work like Diffusion Policy and Latent Diffusion Models for control), which have proven effective at modeling multimodal distributions without mode collapse; (2) discriminator-based reranking and learned cost functions in planning (building on ideas from GANs, reward learning, and preference-based RL). It addresses a known failure of naive RL-from-pixels approaches to planning—direct RL on high-dimensional outputs causes instability and reward hacking. The work sits in the intersection of imitation learning (diffusion generator) and RL (discriminator optimization), a pattern increasingly common in recent autonomous driving research. Benchmarks would likely be evaluated on closed-loop simulation platforms like CARLA or real-world metrics from manufacturer datasets. The research opens directions in: (1) joint optimization of generator and discriminator through alternating updates, (2) hierarchical discriminators for long-horizon planning, and (3) uncertainty quantification of the discriminator's rankings (avoiding overconfidence on out-of-distribution candidates).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::