SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
| Authors | Tianyi Wang et al. |
| Year | 2026 |
| HF Upvotes | 29 |
| arXiv | 2604.08865 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting training throughput. In this paper, we introduce Sequence-Level PPO (SPPO), a scalable algorithm that harmonizes the sample efficiency of PPO with the stability of outcome-based updates. SPPO reformulates the reasoning process as a Sequence-Level Contextual Bandit problem, employing a decoupled scalar value function to derive low-variance advantage signals without multi-sampling. Extensive experiments on mathematical benchmarks demonstrate that SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods, offering a resource-efficient framework for aligning reasoning LLMs.
Engineering Breakdown
Plain English
This paper tackles a critical problem in training large language models for reasoning tasks: standard token-level PPO (Proximal Policy Optimization) becomes unstable and memory-intensive when applied to long chain-of-thought sequences, while critic-free alternatives like GRPO require so many samples that they become computationally impractical. The authors introduce Sequence-Level PPO (SPPO), which reformulates reasoning as a sequence-level contextual bandit problem, combining the sample efficiency of PPO with the stability of outcome-based updates. This approach eliminates the need for a separate value model and reduces the sample overhead of baseline estimation, enabling more efficient training of reasoning-capable LLMs without sacrificing the theoretical guarantees that make PPO popular in practice.
Core Technical Contribution
The core innovation is reformulating the long-horizon reasoning problem from a token-level MDP (Markov Decision Process) into a sequence-level contextual bandit framework, which fundamentally changes how credit assignment and policy updates work. Instead of computing advantages at every token step (which becomes unstable over long CoT sequences), SPPO computes advantages at the sequence level using only the final outcome reward, eliminating temporal credit assignment instability. This is combined with a reformulated PPO objective that maintains theoretical convergence properties while removing the dependency on a separate critic/value model, reducing memory overhead. The approach keeps the computational efficiency advantage of outcome-based methods while retaining the sample efficiency that makes PPO superior to pure GRPO.
How It Works
SPPO operates by first collecting trajectories of full reasoning sequences (complete chain-of-thought responses) conditioned on problem inputs, then computing a single advantage value for each entire sequence based on whether it reached a verifiable correct answer. Rather than unrolling rewards and values backward through each token position, SPPO treats each sequence as an atomic unit in a contextual bandit setting where the context is the problem statement and the action is the entire generated sequence. The policy update uses a PPO-style clipped objective function applied at the sequence level, which computes the importance-weighted ratio of new-to-old policy probabilities for the full sequence. A lightweight baseline for variance reduction is computed across a batch of sequences without maintaining a separate neural network value model, reducing both memory (no value model to store and backprop through) and computation (no dual forward passes). The algorithm iterates by collecting new sequence samples under the updated policy, recomputing sequence-level advantages, and applying PPO updates until convergence.
Production Impact
For production systems training reasoning LLMs, SPPO offers immediate practical benefits: you eliminate the 2x GPU memory cost of maintaining both policy and value models, enabling larger model sizes or batch sizes on the same hardware. The removal of token-level advantage computation eliminates a major source of training instability that practitioners encounter in multi-step reasoning tasks (math, code generation, scientific reasoning), translating to faster convergence and fewer training runs needed to find stable hyperparameter regimes. The sample efficiency remains comparable to standard PPO while being orders of magnitude better than critic-free baselines, meaning you need fewer GPU-hours of sampling to reach a given performance target. However, this approach requires your reward signal to be sparse and outcome-based (correct/incorrect) rather than dense per-token; if you need fine-grained token-level feedback, SPPO is not suitable. Integration into existing RL training pipelines is straightforward since it uses standard PPO-style updates, but teams need to refactor their value model infrastructure and advantage computation code.
Limitations and When Not to Use This
SPPO fundamentally requires outcome-based reward signals (pass/fail at sequence end) and cannot leverage dense intermediate rewards, limiting applicability to domains where you can only verify final answers; this rules out tasks where partial credit or step-wise feedback is valuable. The sequence-level abstraction loses fine-grained credit assignment information, which may be suboptimal for tasks where different intermediate steps contribute differently to the final outcome—token-level approaches theoretically capture this better even if practically unstable. The paper's contextual bandit formulation assumes independence between sequences, which may not hold if you want to teach the model to learn from its own error patterns across multiple attempts on related problems. Early results mentioned in the abstract suggest improvements over token-level PPO and GRPO, but without seeing full experimental results (numbers, datasets, baselines), it's unclear how much of the stability gain comes from the algorithm versus simply forcing longer-horizon batching. The approach also doesn't address the underlying problem that verifiable rewards are expensive to obtain at scale—you still need a verifier or ground truth, just applied once per sequence instead of per-token.
Research Context
This work directly addresses known pain points in the practical application of PPO to reasoning tasks, building on years of RLHF research for LLMs (exemplified by InstructGPT and subsequent work) while incorporating insights from outcome-based approaches like GRPO (Group Relative Policy Optimization). The sequence-level contextual bandit formulation connects to classical RL theory but applies it in a novel way to the long-horizon reasoning setting where standard Bellman decomposition becomes computationally prohibitive. It sits in the broader trend of simplifying and stabilizing RL for large language models, following other recent work that trades off fine-grained credit assignment for stability and computational efficiency. The paper implicitly opens research questions around hybrid approaches (can you combine sequence-level and token-level updates?), the theoretical convergence rates of sequence-level PPO, and how to extend this to domains with dense rewards or multiple reward signals.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::