TingIS: Real-time Risk Event Discovery from Noisy Customer Incidents at Enterprise Scale
| Authors | Jun Wang et al. |
| Year | 2026 |
| HF Upvotes | 12 |
| arXiv | 2604.21889 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Real-time detection and mitigation of technical anomalies are critical for large-scale cloud-native services, where even minutes of downtime can result in massive financial losses and diminished user trust. While customer incidents serve as a vital signal for discovering risks missed by monitoring, extracting actionable intelligence from this data remains challenging due to extreme noise, high throughput, and semantic complexity of diverse business lines. In this paper, we present TingIS, an end-to-end system designed for enterprise-grade incident discovery. At the core of TingIS is a multi-stage event linking engine that synergizes efficient indexing techniques with Large Language Models (LLMs) to make informed decisions on event merging, enabling the stable extraction of actionable incidents from just a handful of diverse user descriptions. This engine is complemented by a cascaded routing mechanism for precise business attribution and a multi-dimensional noise reduction pipeline that integrates domain knowledge, statistical patterns, and behavioral filtering. Deployed in a production environment handling a peak throughput of over 2,000 messages per minute and 300,000 messages per day, TingIS achieves a P90 alert latency of 3.5 minutes and a 95% discovery rate for high-priority incidents. Benchmarks constructed from real-world data demonstrate that TingIS significantly outperforms baseline methods in routing accuracy, clustering quality, and Signal-to-Noise Ratio.
Engineering Breakdown
Plain English
TingIS is an enterprise incident discovery system that automatically detects and correlates technical anomalies in large-scale cloud services by combining efficient event indexing with Large Language Models. The system addresses a critical pain point: customer-reported incidents contain vital signals about system failures that monitoring systems miss, but the extreme volume, noise, and semantic complexity across different business units make it nearly impossible to extract actionable intelligence manually. The core innovation is a multi-stage event linking engine that uses LLMs to intelligently decide which events should be grouped together, enabling stable and accurate incident extraction. For cloud operators managing services where minutes of downtime cost millions, this represents a practical bridge between raw incident data and actual root-cause discovery.
Core Technical Contribution
The primary innovation is a multi-stage event linking engine that synergizes efficient indexing techniques with LLM-based decision making for event merging and incident extraction. Rather than treating incident discovery as a simple classification or clustering problem, the authors recognize it as a semantic linking task where LLMs can understand the business context and relationship between disparate system events. The system architecture separates the indexing problem (how to efficiently retrieve candidate events) from the merging problem (whether events belong to the same incident), allowing each stage to be optimized independently. This decomposition is novel because prior approaches either relied on rigid rule-based correlation or applied generic ML classifiers without leveraging the reasoning capabilities of foundation models.
How It Works
The system operates in multiple stages: first, incoming events (customer reports, alerts, logs) are indexed using efficient retrieval techniques to create candidate sets of related events. Second, for each candidate set, the LLM receives a structured representation of the events and determines whether they should be merged into a single incident or remain separate. The LLM leverages semantic understanding of error messages, service dependencies, and temporal relationships to make these decisions. Third, the merged event clusters are processed to extract actionable incident descriptions and metadata. The architecture is designed to handle high throughput (implicit from 'extreme noise, high throughput' language) while maintaining low latency for real-time detection. The key insight is that LLMs can understand when two superficially different errors (e.g., 'timeout in service A' and 'connection refused in service B') are actually symptoms of the same underlying incident.
Production Impact
For engineers running cloud services, TingIS would dramatically reduce Mean Time To Detection (MTTD) and Mean Time To Resolution (MTTR) by automatically correlating noisy incident signals that current monitoring dashboards treat as separate alerts. In practice, this means your on-call team would receive pre-grouped incident summaries instead of hundreds of independent alerts, reducing alert fatigue and enabling faster root-cause analysis. The integration complexity is moderate: you need to connect your incident data sources (support tickets, logs, alerts) to the system's indexing layer and configure LLM prompts for your specific business domain. Trade-offs include LLM API costs (per inference), latency overhead from LLM calls (likely 500ms-2s per merge decision), and the requirement to tune the system's sensitivity for false positive/negative rates in incident grouping—aggressive merging might hide distinct incidents, while conservative merging defeats the purpose.
Limitations and When Not to Use This
The paper's abstract does not detail the accuracy metrics or failure modes, leaving unclear how often the LLM's merging decisions are correct in practice—particularly for novel or adversarial incident patterns not seen during development. The system assumes that LLMs can reliably extract semantic meaning from incident data across all business domains, but this assumption may break down for highly technical or domain-specific errors that require deep infrastructure knowledge beyond training data. The reliance on LLM inference introduces latency and cost constraints; for very high-throughput environments (millions of events/day), the compute cost might become prohibitive. Additionally, the paper does not address how the system handles concept drift—as systems evolve and new failure modes emerge, the LLM's understanding of incident relationships may become stale without continuous retraining or prompt updates.
Research Context
This work builds on decades of research in event correlation and anomaly detection in distributed systems, modernizing the approach by replacing hand-crafted rules and classical ML with LLM-based reasoning. It sits at the intersection of three research areas: log/event analysis (where techniques like template-based parsing are standard), LLM-as-a-tool paradigms (prompt engineering for structured decision-making), and applied systems reliability. The contribution is primarily in systems and application rather than fundamental ML research—it demonstrates how to effectively deploy LLMs in a real operational context where latency, cost, and interpretability matter. This opens future directions around few-shot learning for new incident types, active learning to improve the LLM's merging accuracy over time, and integration with root-cause analysis systems that could automatically propose remediation steps.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::