Strait: Perceiving Priority and Interference in ML Inference Serving
| Authors | Haidong Zhao & Nikolaos Georgantas |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.28175 |
| Download | |
| Categories | cs.LG |
Abstract
Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. However, limited support for task prioritization and insufficient latency estimation under concurrent execution may restrict their applicability in on-premises scenarios. We present \emph{Strait}, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization. To improve latency estimation, Strait models potential contention during data transfer and accounts for kernel execution interference through an adaptive prediction model. By drawing on these predictions, it performs priority-aware scheduling to deliver differentiated handling. Evaluation results under intense workloads suggest that Strait reduces deadline violations for high-priority tasks by 1.02 to 11.18 percentage points while incurring acceptable costs on low-priority tasks. Compared to software-defined preemption approaches, Strait also exhibits more equitable performance.
Engineering Breakdown
Plain English
Strait is an ML inference serving system that addresses a critical gap in production deployments: handling multiple priority tiers of inference requests while maintaining tight latency bounds under high GPU utilization. The paper identifies that existing serving systems lack both accurate latency prediction under concurrent GPU execution and priority-aware scheduling that accounts for interference between workloads. Strait's core innovation is an adaptive prediction model that captures contention during data transfer and kernel execution interference, then uses these predictions to perform priority-aware scheduling. The system is evaluated under intense, mixed-priority workloads on multi-GPU deployments, targeting improved deadline satisfaction rates compared to baseline approaches.
Core Technical Contribution
The key technical novelty lies in modeling GPU interference effects during concurrent execution rather than treating requests independently. Strait introduces an adaptive prediction model that explicitly accounts for two sources of latency variation: (1) data transfer contention when multiple requests compete for PCIe/NVLink bandwidth, and (2) kernel execution interference when multiple CUDA kernels run simultaneously on the same GPU, causing cache conflicts and reduced per-kernel throughput. This interference-aware latency model then feeds into a priority-aware scheduler that can make informed trade-offs between low-priority and high-priority requests. Prior systems either ignore interference entirely (leading to deadline misses) or use overly pessimistic estimates (wasting GPU capacity), whereas Strait dynamically adapts its predictions based on observed execution patterns.
How It Works
The system operates in three stages: (1) Profiling and Model Training, where Strait runs representative batches of mixed-priority requests on the target GPU to measure actual latencies under various contention scenarios, building an empirical interference model. (2) Latency Prediction, where incoming requests are characterized by their model type, batch size, and estimated compute demand, then the adaptive model predicts tail latency accounting for currently-queued requests and their interference patterns. (3) Priority-Aware Scheduling, where the scheduler uses these predictions to make admission and ordering decisions—high-priority requests receive favorable queue positions and may trigger preemption of low-priority work, while the system continuously monitors actual versus predicted latency to recalibrate the interference model. The adaptive component is critical: rather than a static model, Strait adjusts its interference coefficients online as GPU utilization patterns shift, making it robust to workload variations that would break a pre-trained model.
Production Impact
For engineers building on-premises ML inference systems, Strait directly solves the deadline satisfaction problem in multi-tenant or prioritized inference scenarios—think serving both latency-critical user-facing requests and batch analytics jobs on the same GPU cluster. Implementation would require: (1) integrating the profiling stage into your model deployment pipeline to build interference models for each model+GPU combination, (2) instrumenting your request scheduler to track queue state and feed it to the latency predictor, (3) adding request metadata (priority tier, model ID, expected batch size) to your serving RPC. The main trade-offs are operational overhead (periodic retraining of interference models as hardware ages or workloads drift) and complexity (the adaptive model adds significant code vs. simpler FCFS schedulers). However, for systems where missing SLAs is costly (e.g., financial services, real-time recommendations), the improvement in deadline satisfaction likely justifies this complexity. Integration points are primarily in your scheduler (most ML serving frameworks like Clipper, Ansor, or custom systems) and your request router (to track priority tiers).
Limitations and When Not to Use This
The approach assumes that interference effects are relatively stable and predictable across similar request patterns—this breaks down with highly heterogeneous workloads or dynamic GPU sharing with non-ML workloads, where interference becomes chaotic. Strait requires a profiling phase on target hardware before deployment, which adds operational friction and means models trained on one GPU architecture may not transfer to another (a 100-GPU cluster means 100 sets of profiles). The paper's evaluation is described as 'intense workloads' but concrete throughput numbers and deadline miss rates vs. baselines are absent from the abstract, making it unclear how much improvement is actually achieved in real scenarios. Additionally, the approach doesn't address model serving patterns with variable-size batches or dynamic batch composition, which are common in production when request arrival is bursty. The system also requires explicit priority signals in requests—it cannot infer priority from request metadata or learn it end-to-end, limiting applicability in scenarios where priority is implicit or changing.
Research Context
This work sits at the intersection of GPU scheduling and ML inference serving, building on prior research in both areas. It extends systems like REEF and INFaaS (which focus on multi-model scheduling) by adding explicit interference modeling—a capability that prior systems like Ansor and vLLM have largely ignored in favor of simplified contention assumptions. The paper draws on microbenchmarking techniques from GPU systems research (e.g., understanding CUDA kernel interference through shared cache behavior) and applies them to the inference scheduling domain. This opens a research direction toward 'interference-aware inference serving' where the scheduler is GPU-architecture-aware, which could extend to other accelerators (TPUs, custom inference chips) and heterogeneous clusters. The work also complements concurrent research in predictable ML inference (e.g., work on tail latency reduction) by providing a principled foundation for trading off throughput and latency under priority constraints.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::