TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification
| Authors | Adam Rida |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2604.14531 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Every call to an LLM classification endpoint produces a labeled input-output pair already retained in production logs. These pairs constitute a free, growing training set: a lightweight surrogate trained on them can absorb a significant portion of future traffic at near-zero marginal inference cost. The open questions are when the surrogate is reliable enough to deploy, what it handles versus defers, and how that boundary evolves as data accumulates. We introduce TRACER (Trace-based Adaptive Cost-Efficient Routing), an open-source system that trains ML surrogates on an LLM's own production traces and governs deployment through a parity gate: the surrogate is activated only when its agreement with the LLM exceeds a user-specified threshold α. To make the routing boundary transparent, TRACER generates interpretability artifacts describing which input regions the surrogate handles, where it plateaus, and why it defers. On a 77-class intent benchmark with a Sonnet 4.6 teacher, TRACER achieves 83-100% surrogate coverage depending on the quality target α; on a 150-class benchmark, the surrogate fully replaces the teacher. On a natural language inference task, the parity gate correctly refuses deployment because the embedding representation cannot support reliable separation. The system is available as open-source software.
Engineering Breakdown
Plain English
TRACER is a system that automatically creates lightweight surrogate models from production logs of LLM classification calls, allowing cheaper inference for future requests. Every LLM classification already produces a labeled input-output pair stored in logs—TRACER captures this free training data and trains small ML models on it. The system deploys the surrogate model only when it agrees with the LLM above a threshold α, creating a cost-efficient routing layer that handles easy cases locally while deferring harder cases to the expensive LLM. This approach lets production systems absorb significant traffic at near-zero marginal cost while maintaining quality guarantees through the parity gate mechanism.
Core Technical Contribution
The core novelty is treating production logs as a free, continuously growing training signal that can power adaptive routing in real time. TRACER introduces the parity gate concept: a dynamic agreement threshold that governs when a surrogate is reliable enough to handle traffic, making the surrogate's deployment decision transparent and auditable. This is fundamentally different from static surrogate deployment—instead of training once and deploying a fixed decision boundary, TRACER continuously retrains as new labeled data arrives and adjusts routing thresholds based on measured agreement with the production LLM. The system architecture explicitly models the cost-reliability trade-off as a tunable parameter α, giving engineers direct control over the speed-accuracy frontier.
How It Works
The system operates in a continuous loop: (1) production LLM calls generate labeled input-output pairs that are logged, (2) a lightweight surrogate model is trained periodically on accumulated traces, (3) incoming requests are scored by both the surrogate and LLM in parallel, and (4) routing decisions use the parity gate—if surrogate and LLM agreement exceeds threshold α, the request is handled by the cheap surrogate; otherwise it defers to the LLM for a definitive answer. The agreement metric tracks how often the surrogate produces the same output as the LLM on held-out test sets, creating an empirical measure of surrogate reliability that updates as new data accumulates. The surrogate itself is typically a small, efficient model (not specified in abstract but contextually would be a classifier far cheaper than the LLM) that learns decision boundaries from the accumulated traces. Over time, as more traces accumulate, the surrogate becomes more capable and the parity gate activates more frequently, routing more traffic away from the expensive LLM and toward the cheap surrogate.
Production Impact
For teams operating LLM classification services, TRACER directly reduces inference costs by routing high-confidence requests through cheap surrogates instead of calling expensive LLM endpoints. If a surrogate can absorb 50-80% of traffic (realistic for many classification tasks) while maintaining agreement threshold α, you reduce infrastructure spend proportionally—a significant saving in large-scale deployments. The system is backward-compatible: it observes but doesn't require changes to existing LLM pipelines, and can be bolted onto production systems that already log LLM calls. The trade-offs are: (1) surrogate latency overhead for agreement checking, (2) complexity of managing retraining schedules and threshold tuning, (3) dependency on LLM output quality being reliable ground truth (if the LLM is wrong, the surrogate learns wrong labels), and (4) need for careful monitoring of agreement metrics to catch distribution shift where the surrogate agreement degrades.
Limitations and When Not to Use This
TRACER assumes production logs contain high-quality labels from the LLM, but LLMs make mistakes—systematic errors in the LLM get replicated in the surrogate, creating a garbage-in-garbage-out problem that cascades. The system doesn't address how to set α adaptively; choosing the threshold requires empirical tuning and domain knowledge, and wrong choices (too high α = poor cost savings; too low α = quality degradation) aren't addressed. Distribution shift is a critical blind spot: when future request distributions differ from training traces (new domains, seasonal patterns, adversarial inputs), surrogate agreement degrades but may not be caught until real user impact occurs. The paper doesn't discuss how to handle multi-class classification, imbalanced classes, or how surrogate architecture choice (linear vs. deep) affects cost-benefit trade-offs—these are open questions for practitioners.
Research Context
This work sits at the intersection of machine learning efficiency (reducing inference costs) and online learning (continuous retraining on production data). It builds on decades of work in cascade/mixture-of-experts routing but applies it in a cost-aware production setting with clear measurement of agreement as a proxy for quality. The implicit benchmark is cost per correct classification compared to baseline full-LLM systems, though specific baselines aren't mentioned in the abstract. TRACER opens the research direction of log-driven model optimization—using production telemetry not just for monitoring but as an active training signal, a paradigm increasingly relevant as LLMs dominate production inference workloads.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::