Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
| Authors | Zhen Zhang et al. |
| Year | 2026 |
| HF Upvotes | 19 |
| arXiv | 2604.27039 |
| Download | |
| Code | https://github.com/eric-ai-lab/Length-Value-Model |
Abstract
Token serves as the fundamental unit of computation in modern autoregressive models, and generation length directly influences both inference cost and reasoning performance. Despite its importance, existing approaches lack fine-grained length modeling, operating primarily at the coarse-grained sequence level. We introduce the Length Value Model (LenVM), a token-level framework that models the remaining generation length. By formulating length modeling as a value estimation problem and assigning a constant negative reward to each generated token, LenVM predicts a bounded, discounted return that serves as a monotone proxy for the remaining generation horizon. This formulation yields supervision that is annotation-free, dense, unbiased, and scalable. Experiments on LLMs and VLMs demonstrate LenVM provides a highly effective signal at inference time. On the LIFEBench exact length matching task, applying LenVM to a 7B model improves the length score from 30.9 to 64.8, significantly outperforming frontier closed-source models. Furthermore, LenVM enables continuous control over the trade off between performance and efficiency. On GSM8K at a budget of 200 tokens, LenVM maintains 63% accuracy compared to 6 percent for token budget baseline. It also accurately predicts total generation length from the prompt boundary. Finally, LenVM's token-level values offer an interpretable view of generation dynamics, revealing how specific tokens shift reasoning toward shorter or longer regimes. Results demonstrate that LenVM supports a broad range of applications and token length can be effectively modeled as a token-level value signal, highlighting the potential of LenVM as a general framework for length modeling and as a length-specific value signal that could support future RL training. Code is available at https://github.com/eric-ai-lab/Length-Value-Model.
Engineering Breakdown
Plain English
This paper introduces Length Value Model (LenVM), a framework that predicts how many tokens an LLM should generate at the token level rather than deciding sequence length upfront. The core insight is treating length prediction as a value estimation problem: each generated token receives a constant negative reward, and the model learns to predict remaining generation length as a bounded, discounted return. This approach produces supervision automatically without manual annotations, scales efficiently during pretraining, and experiments show it improves both inference cost and reasoning performance on language and vision-language models. The method is token-granular rather than sequence-level, enabling fine-grained control over generation length with dense, unbiased training signals.
Core Technical Contribution
The key novelty is formulating token-level length modeling as a value estimation problem rather than treating it as a classification or regression task. By assigning a constant negative reward per token and computing discounted returns, the authors create a theoretically grounded framework where the predicted value monotonically reflects remaining generation horizon. This is fundamentally different from prior coarse-grained approaches that decide length once per sequence; LenVM operates at token granularity and generates supervision automatically from any generation trajectory without external labels. The bounded, discounted return formulation ensures the learning signal is both mathematically principled and practically scalable during pretraining.
How It Works
LenVM augments an autoregressive model with a value head that predicts remaining generation length at each token position. During generation, when the model produces a token, it receives a constant negative reward (typically -1), and the value head estimates the discounted sum of future rewards from that point forward. If the actual sequence length is L and we're at position t with discount factor γ, the target value is approximately -γ^(L-t), creating a monotone decreasing signal as generation progresses. The supervision is annotation-free: given any completed sequence, you can compute ground-truth length values for every intermediate position by working backwards. During inference, the model can use these length predictions to decide when to stop, balance compute vs. reasoning capability, or provide uncertainty estimates about generation length. The architecture integrates a small value prediction head alongside the language modeling head, trained jointly with standard next-token prediction loss plus L2 regression loss on the length predictions.
Production Impact
This approach directly addresses the inference cost vs. reasoning quality trade-off that engineers face in production LLM deployments. Instead of fixed generation budgets, systems can dynamically allocate tokens per query based on predicted complexity, reducing wasted computation on simple questions while spending more on reasoning-heavy tasks. The token-level granularity enables adaptive stopping criteria and uncertainty quantification about when generation should terminate, improving both latency and result quality. Integration is relatively straightforward—add a value head to your existing model, train it alongside language modeling, and use length predictions to gate generation budget. Key trade-offs include modest compute overhead during training (one additional regression head) and the need to decide on discount factors and reward magnitudes; for production systems, you'd likely need to tune these hyperparameters empirically on your domain and measure actual cost-quality curves before deployment.
Limitations and When Not to Use This
The paper assumes constant negative rewards per token, which may not reflect the true cost-quality trade-off in domain-specific applications where certain token types are more expensive or valuable than others. The approach is evaluated primarily on general reasoning tasks; its effectiveness on highly constrained generation (summarization, translation with fixed-length targets) or domains where token budgets are externally determined remains unclear. The bounded return formulation requires choosing discount factors, and the paper provides limited guidance on setting these across different model scales and task types—this hyperparameter sensitivity could be problematic in production. Additionally, the value estimates are only useful if the model actually learns to correlate predicted length with optimal stopping, which may fail in domains where ideal length is ambiguous or context-dependent.
Research Context
This work builds on the intersection of value-based reinforcement learning (where discount returns are standard) and language model fine-tuning, extending ideas from process reward models and chain-of-thought length prediction into a unified token-level framework. It advances beyond prior sequence-level length modeling by introducing density and granularity, similar to how recent work on token-level value estimation has improved over state-level approaches in RL. The paper addresses a growing concern in LLM scaling: generation efficiency and the cost-quality Pareto frontier, which has gained urgency with larger models and longer reasoning chains. This opens directions for hybrid length modeling (combining with draft decoding, speculative execution) and extensions to other token-level attributes like confidence or complexity.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::