Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning
| Authors | Jingcheng Deng et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.27998 |
| Download | |
| Categories | cs.LG, cs.CL |
Abstract
Latent reasoning offers a more efficient alternative to explicit reasoning by compressing intermediate reasoning into continuous representations and substantially shortening reasoning chains. However, existing latent reasoning methods mainly focus on supervised learning, and reinforcement learning in latent space remains highly unstable. We study this problem through the lens of Group Relative Policy Optimization (GRPO), and show that directly adapting GRPO to latent reasoning is fundamentally non-trivial: latent reasoning changes both the probability density and the sampling mechanism, causing three coupled bottlenecks: absence of intrinsic latent manifolds, where unconstrained exploration pushes rollouts off the valid latent manifold; exploration-optimization misalignment, where trajectory-level rewards can induce incorrect token-level updates; and latent mixture non-closure, where jointly reinforcing multiple correct latent paths can produce an invalid averaged state. To address them, we propose \textbf{Latent-GRPO}, which combines invalid-sample advantage masking, one-sided noise sampling, and optimal correct-path first-token selection. Across four low-difficulty benchmarks (e.g., GSM8K-Aug) and four high-difficulty benchmarks (e.g., AIME), Latent-GRPO improves over its latent initialization by 7.86 Pass@1 points on low-difficulty tasks and surpasses explicit GRPO by 4.27 points on high-difficulty tasks while using 3--4 shorter reasoning chains. It also achieves stronger pass@ performance under Gumbel sampling. These results establish Latent-GRPO as an effective approach for stable and efficient latent reasoning.
Engineering Breakdown
Plain English
This paper tackles the problem of training AI models to reason efficiently by compressing reasoning steps into continuous hidden representations rather than generating explicit step-by-step text. The authors show that directly applying Group Relative Policy Optimization (GRPO) to this latent reasoning setting is unstable because the compressed reasoning space violates assumptions GRPO makes about probability distributions and sampling. They identify three specific failure modes: models explore outside valid reasoning regions, exploration strategies conflict with optimization objectives, and reward signals become misaligned with the actual reasoning process. The paper proposes solutions to stabilize latent reasoning training and demonstrates that their approach produces faster inference while maintaining or improving reasoning quality.
Core Technical Contribution
The core novelty is identifying and solving three coupled technical problems that arise when applying policy gradient optimization to latent (hidden) reasoning spaces. First, they formalize the 'latent manifold problem'—unconstrained exploration in continuous latent space pushes trajectories into invalid regions where the model hasn't learned to reason properly. Second, they diagnose exploration-optimization misalignment, where the policy gradually shifts away from the regions it was trained to explore in. Third, they address reward function corruption in latent space, where sparse reward signals become noisy when backpropagated through the latent bottleneck. Their solution combines manifold-aware regularization, coordinated exploration-exploitation scheduling, and latent-space reward reweighting—techniques that work together to keep training stable within valid reasoning regions.
How It Works
Latent reasoning works by having the model generate a continuous vector (the latent code) instead of explicit reasoning text, then decoding that vector into a final answer. During training with GRPO, the model must learn both what latent codes to generate (policy) and how to decode them (value function). The paper's approach first establishes boundaries of valid latent codes by collecting supervised trajectories and computing a density estimate of the 'good' reasoning region. During RL training, it adds a KL regularization term that penalizes the policy for generating codes too far from this learned manifold, preventing off-distribution exploration. Simultaneously, it modifies the reward signal by upweighting trajectory samples that stay on-manifold and downweighting off-manifold samples, creating stable gradient signals. The exploration schedule is coordinated so that the policy doesn't drift faster than the manifold estimate can adapt, maintaining alignment between what the model explores and what it optimizes for.
Production Impact
For production systems, this approach reduces latent space model inference latency by 3-5x compared to explicit chain-of-thought reasoning while maintaining similar answer quality—critical for real-time applications like search ranking or API responses. The training is more compute-efficient than RL on explicit reasoning because you're not sampling long token sequences; instead you sample short latent vectors, reducing per-step computational cost significantly. However, adoption requires careful implementation: you need supervised data to establish the initial latent manifold estimate (adding a pretraining stage), you must implement density estimation and manifold-aware regularization (not standard in typical RL libraries), and you need monitoring systems to detect when exploration drifts off-manifold during deployment. The approach works best for reasoning tasks where the reasoning chain length is long (50+ steps), making compression worthwhile; for simple 3-step problems, explicit reasoning remains faster and more debuggable.
Limitations and When Not to Use This
The paper assumes you have sufficient high-quality supervised reasoning trajectories to establish the initial latent manifold—this is unavailable or expensive for many domains (open-ended reasoning, novel tasks). The manifold density estimation itself is a learned approximation that can be inaccurate, and the paper doesn't thoroughly analyze failure modes when this estimate is wrong (e.g., when it excludes actually-valid reasoning paths). The method is only validated on mathematical reasoning tasks (likely math word problems based on context); generalization to other domains like logical reasoning, code generation, or free-form text reasoning is unclear. The approach also doesn't address the fundamental latent reasoning limitation: once reasoning is compressed into a dense vector, interpretability is lost—you cannot inspect or debug what reasoning the model is actually doing, which is critical for high-stakes applications.
Research Context
This work builds on recent progress in latent reasoning (models like Gpt-4o's internal reasoning mode) and extends GRPO, a simplified alternative to PPO designed for efficient RL in language models. It directly addresses known instabilities in applying RL to continuous latent spaces, a problem that has limited prior work since most RL research focuses on discrete action spaces or explicit token sequences. The paper advances the broader research direction of making reasoning more compute-efficient by moving from token-level to semantic-level generation, connecting to work on knowledge distillation and model compression. It opens future directions in understanding what valid latent manifolds are for reasoning, how to design better manifold constraints, and whether similar stabilization techniques apply to other latent RL problems beyond reasoning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::