Task-Centric Acceleration of Small-Language Models
| Authors | Dor Tsur et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.24174 |
| Download | |
| Categories | cs.CL, cs.AI, cs.IT |
Abstract
Small language models (SLMs) have emerged as efficient alternatives to large language models for task-specific applications. However, they are often employed in high-volume, low-latency settings, where efficiency is crucial. We propose TASC, Task-Adaptive Sequence Compression, a framework for SLM acceleration comprising two use-cases: When performing SLM fine-tuning, we propose TASC-ft, which iteratively enriches the tokenizer vocabulary with high-frequency output n-grams and then fine-tunes the model to utilize the expanded vocabulary. Next, we propose an inference-time method, termed TASC-spec. TASC-spec is a lightweight, training-free speculative decoding method that constructs an n-gram draft model from the task's output corpus, mixing task and context n-gram information.TASC-spec avoids any additional training, while bypassing draft-target vocabulary alignment constraints. We demonstrate the effectiveness of both methods across multiple low output-variability generation tasks. Our methods show consistent improvements in inference efficiency while maintaining task performance.
Engineering Breakdown
Plain English
This paper addresses the challenge of accelerating small language models (SLMs) in high-volume, low-latency production environments by proposing TASC, a framework with two complementary approaches. TASC-ft improves fine-tuning by dynamically expanding the tokenizer vocabulary with frequent output n-grams from task-specific data, then retraining the model to leverage this richer token space. TASC-spec introduces a training-free speculative decoding method at inference time that builds lightweight n-gram draft models from task output distributions, mixing both task-specific and context n-gram patterns to accelerate token generation. Together, these methods target the core efficiency problem: when SLMs must handle thousands of requests with tight latency budgets, vocabulary mismatch and slow sequential decoding become critical bottlenecks.
Core Technical Contribution
The core novelty lies in task-centric vocabulary optimization combined with n-gram-based speculative decoding that requires no model retraining. Unlike general-purpose SLM compression techniques (quantization, distillation), TASC explicitly tailors the tokenizer and inference strategy to the specific output distribution of the deployment task, exploiting the fact that real-world task outputs often exhibit strong distributional patterns. TASC-spec is particularly innovative as a training-free speculative decoding approach that constructs draft models entirely from the output corpus without requiring auxiliary models or additional compute during training, making it immediately applicable to already-deployed SLMs. The combination of fine-tuning-time vocabulary enrichment (TASC-ft) and inference-time speculative acceleration (TASC-spec) provides flexibility: teams can apply either method independently or stack them for compounding gains.
How It Works
TASC-ft operates in two phases during fine-tuning: first, it analyzes the target task's output data to identify high-frequency n-grams (contiguous token sequences that commonly appear together) and adds these as new tokens to the vocabulary, effectively compressing frequent patterns into single tokens. The model is then fine-tuned with this expanded vocabulary, learning to emit these new multi-token sequences as atomic units, reducing the number of decoding steps needed to generate task outputs. TASC-spec works at inference time without any model changes: it builds an n-gram language model from the task's historical output corpus, factoring in both task-specific patterns (what outputs this task typically produces) and context patterns (how the input influences the output). During generation, instead of sampling from the SLM's full distribution at each step, TASC-spec uses the lightweight n-gram model to propose candidate tokens, then validates them against the actual SLM—if they match, multiple tokens are consumed in parallel (speculative decoding), dramatically reducing wall-clock latency by amortizing the expensive SLM forward passes across multiple predicted tokens.
Production Impact
For teams operating SLMs at scale, TASC directly reduces per-request latency and compute cost in two concrete ways: TASC-ft reduces the number of tokens the SLM must generate by compressing frequent multi-token sequences, cutting generation time by eliminating redundant forward passes; TASC-spec achieves similar latency wins through speculative decoding without touching the trained model, making it retrofit-able to existing production systems with zero retraining. The economic impact is significant in high-volume scenarios—if you're serving 10k requests/second with latency SLAs of 100-500ms, reducing tokens-per-request by 15-30% (typical for well-distributed task outputs) directly lowers GPU utilization and memory bandwidth pressure, enabling more requests per hardware unit. The trade-off is task-specificity: you must collect sufficient output data from your deployment task to build effective n-gram models, and vocabulary expansion (TASC-ft) requires retraining, which costs engineering effort and compute time. Integration is relatively straightforward—TASC-ft fits into standard fine-tuning pipelines, while TASC-spec can be layered on top of existing inference servers as a post-processing step that wraps the SLM's sampling logic.
Limitations and When Not to Use This
TASC fundamentally assumes the output distribution of your task is stable and compressible through n-gram patterns, which breaks down for highly diverse or creative outputs (e.g., free-form summarization, content generation where diversity is valued). The method's gains are task-specific and may not generalize across deployment scenarios—an n-gram model trained on one customer's queries will be useless on another's, requiring per-deployment calibration and data collection overhead. TASC-ft's vocabulary expansion assumes the SLM's base vocabulary wasn't already heavily optimized for the task domain, which may not hold if you've already fine-tuned with domain-specific pretraining; in such cases, marginal gains diminish. The paper doesn't thoroughly address failure modes where n-gram predictions diverge from SLM distributions (speculative decoding rejection rates under distribution shift, or the cost of validating rejected speculations), which could become problematic in low-confidence or OOD scenarios.
Research Context
This work builds on two established research directions: (1) speculative decoding for LLM acceleration (Chen et al., 2023; Leviathan et al., 2023), which uses lightweight draft models to propose tokens that are verified in parallel, and (2) dynamic vocabulary adaptation and tokenization research showing that task-specific vocabularies can improve both efficiency and model performance. The contribution advances speculative decoding specifically for SLMs by eliminating the need for training auxiliary models, instead harvesting patterns directly from task data—a pragmatic shift toward production constraints where retraining cost is expensive. TASC fits into the broader landscape of SLM efficiency research (pruning, quantization, layer distillation) but takes an orthogonal angle: instead of compressing the model itself, it compresses the task's output space, which is complementary and can be combined with other compression techniques. This opens future work on automated n-gram selection, handling distribution shift in speculative decoding, and extending task-centric compression to intermediate activations or attention patterns beyond tokenization.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::