MARS: Enabling Autoregressive Models Multi-Token Generation
| Authors | Ziqi Jin et al. |
| Year | 2026 |
| HF Upvotes | 34 |
| arXiv | 2604.07023 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Autoregressive (AR) language models generate text one token at a time, even when consecutive tokens are highly predictable given earlier context. We introduce MARS (Mask AutoRegreSsion), a lightweight fine-tuning method that teaches an instruction-tuned AR model to predict multiple tokens per forward pass. MARS adds no architectural modifications, no extra parameters, and produces a single model that can still be called exactly like the original AR model with no performance degradation. Unlike speculative decoding, which maintains a separate draft model alongside the target, or multi-head approaches such as Medusa, which attach additional prediction heads, MARS requires only continued training on existing instruction data. When generating one token per forward pass, MARS matches or exceeds the AR baseline on six standard benchmarks. When allowed to accept multiple tokens per step, it maintains baseline-level accuracy while achieving 1.5-1.7x throughput. We further develop a block-level KV caching strategy for batch inference, achieving up to 1.71x wall-clock speedup over AR with KV cache on Qwen2.5-7B. Finally, MARS supports real-time speed adjustment via confidence thresholding: under high request load, the serving system can increase throughput on the fly without swapping models or restarting, providing a practical latency-quality knob for deployment.
Engineering Breakdown
Plain English
MARS is a fine-tuning method that teaches autoregressive language models to predict multiple tokens in a single forward pass, addressing the inefficiency of generating one token at a time even when the next tokens are highly predictable from context. The approach requires no architectural changes, no additional parameters, and produces a single model that remains fully compatible with standard autoregressive inference—you can call it exactly like the original model with no performance degradation. Unlike competing methods like speculative decoding (which needs a separate draft model) or Medusa (which adds extra prediction heads), MARS only needs continued training on existing instruction data. The key finding is that MARS matches or exceeds the original model's performance when generating a single token per forward pass, while enabling efficient multi-token generation when the context supports it.
Core Technical Contribution
The core novelty is a lightweight training-only method that enables multi-token generation without modifying the model architecture or adding parameters. MARS works by continuing to train an instruction-tuned autoregressive model with a masking strategy that teaches it to predict multiple consecutive tokens during training, but the model remains a standard AR model at inference time. The technical insight is that you can achieve multi-token prediction capability through careful training objectives rather than architectural tricks, avoiding the complexity and parameter overhead of approaches like Medusa's multiple heads or speculative decoding's dual-model overhead. This is a pure training methodology rather than an inference or architectural innovation, making it dramatically simpler to integrate into existing production systems.
How It Works
During training, MARS presents the model with sequences where multiple future tokens are masked (hidden), requiring the model to learn to predict several tokens simultaneously given the visible context. The input is standard instruction-tuning data, but the training loss is computed over multiple token positions at once rather than one position per example. The model learns internal representations and attention patterns that can predict token sequences, essentially learning when and how to generate multiple coherent tokens together. At inference time, the model can operate in two modes: standard single-token generation (backward compatible with existing code), or multi-token generation when you explicitly request it. The mechanism relies on the fact that the model already has the capacity to predict multiple tokens—MARS simply teaches it to leverage that capacity by training on masked multi-token targets. Because no architecture changes, the same model weights work for both single-token and multi-token generation.
Production Impact
This approach solves the throughput bottleneck of autoregressive inference without requiring infrastructure changes or model redeployment, since the same weights work for both single and multi-token generation modes. Engineers can adopt MARS by running continued fine-tuning on existing instruction data—no new data collection, no architectural overhaul, no separate model serving. For latency-sensitive applications, MARS enables up to N× speedup when generating N tokens, since N tokens can come from a single forward pass instead of N passes, reducing memory bandwidth pressure and enabling better hardware utilization on accelerators. The trade-off is that training time increases due to the multi-token supervision signal, but the cost is paid once during fine-tuning, not per inference request. Integration complexity is minimal since the model's API and calling convention remain identical to the original—your inference code doesn't need to change.
Limitations and When Not to Use This
MARS's effectiveness likely depends on context coherence and token predictability; highly unpredictable or long-tail sequences may not benefit from multi-token generation, and the paper doesn't clarify how performance degrades when attempting to generate many tokens simultaneously. The approach assumes you have access to the base instruction-tuned model and enough compute to run continued training—this may not be feasible for practitioners using only API-based models or under strict budget constraints. The paper doesn't address how MARS interacts with techniques like quantization, pruning, or other compression methods that are standard in production systems, nor does it discuss performance on code generation, reasoning tasks, or other domains beyond general instruction-following. Finally, the mechanism for automatically determining when and how many tokens to generate per forward pass appears to still be manual or heuristic-based, rather than learned, which limits the method's adaptability to variable context quality.
Research Context
MARS builds on the growing recognition that standard autoregressive decoding is fundamentally inefficient—prior work like speculative decoding and Medusa explored multi-token generation but with architectural or parameter overhead that MARS eliminates. This work is part of a broader effort to close the gap between the theoretical capacity of large models and their practical inference efficiency, addressing the fact that modern LLMs waste compute generating predictable continuations one token at a time. The research connects to instruction-tuning literature (showing that the training objective can shape generative capabilities beyond just accuracy) and extends the success of lightweight fine-tuning methods like LoRA by demonstrating that training modifications alone can enable qualitatively different inference behaviors. MARS opens a research direction around training-centric solutions to inference efficiency, potentially enabling future work on adaptive multi-token generation, dynamic decoding strategies, or hybrid approaches that combine MARS with other efficiency techniques.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::