Latent Adversarial Detection: Adaptive Probing of LLM Activations for Multi-Turn Attack Detection
| Authors | Prashant Kulkarni |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2604.28129 |
| Download | |
| Categories | cs.CR, cs.AI |
Abstract
Multi-turn prompt injection follows a known attack path -- trust-building, pivoting, escalation but text-level defenses miss covert attacks where individual turns appear benign. We show this attack path leaves an activation-level signature in the model's residual stream: each phase shift moves the activation, producing a total path length far exceeding benign conversations. We call this adversarial restlessness. Five scalar trajectory features capturing this signal lift conversation-level detection from 76.2% to 93.8% on synthetic held-out data. The signal replicates across four model families (24B-70B); probes are model-specific and do not transfer across architectures. Generalization is source-dependent: leave-one-source-out evaluation shows each of synthetic, LMSYS-Chat-1M, and SafeDialBench captures distinct attack distributions, with detection on real-world LMSYS reaching 47-71% when its distribution is represented in training. Combined three-source training achieves 89.4% detection at 2.4% false positive rate on a held-out mixed set. We further show that three-phase turn-level labels(benign/pivoting/adversarial) unique to our synthetic dataset are essential: binary conversation-level labels produce 50-59% false positives. These results establish adversarial restlessness as a reliable activation-level signal and characterize the data requirements for practical deployment.
Engineering Breakdown
Plain English
This paper identifies a new way to detect multi-turn prompt injection attacks by analyzing activation patterns in a language model's residual stream rather than looking at text alone. The authors discovered that adversarial attack sequences (trust-building, pivoting, escalation) create a distinctive 'restlessness' signal—a characteristic movement through activation space that benign conversations don't produce. Using five scalar trajectory features to capture this signal, they achieve 93.8% detection accuracy on held-out synthetic data, up from a baseline of 76.2% using text-level defenses. Critically, the approach works across four model families (24B-70B parameters) but the detection probes are model-specific and don't transfer between architectures.
Core Technical Contribution
The core novelty is detecting adversarial multi-turn attacks at the activation level by measuring what the authors call 'adversarial restlessness'—the cumulative path length and directional changes in activation space as an attack unfolds across conversation turns. Rather than trying to catch attacks through textual analysis of individual turns (which the authors show misses covert attacks), they leverage the fact that different conversational phases (trust-building, pivoting, escalation) leave measurable signatures in how the model's internal representations shift. They develop five scalar features that summarize these trajectory patterns and use them for conversation-level binary classification. This represents a shift from defense-in-text to defense-in-activation-space, exploiting the model's own internal dynamics to reveal adversarial intent that individual turns may conceal.
How It Works
The pipeline starts with conversation histories (either synthetic adversarial sequences or benign examples from LMSYS-Chat-1M and SafeDial). For each conversation, the model processes all turns and the authors extract activations from the residual stream at each step, treating this as a sequence of high-dimensional vectors. They then compute five scalar features that characterize the trajectory: these likely include metrics like cumulative L2 distance traveled, angular dispersion between consecutive steps, maximum displacement from origin, variance in step magnitudes, and temporal clustering. These five numbers are fed into a classifier (details on the exact classifier aren't provided in the abstract) to produce a binary decision: benign or adversarial conversation. The key insight is that the geometry of the activation path itself—not the text content—carries the signal; an attack that systematically shifts representations in consistent directions will show a characteristic trajectory signature distinct from the random-walk-like pattern of benign multi-turn chat.
Production Impact
For teams deploying LLMs in safety-critical settings, this offers a concrete detection layer that complements existing text-based safeguards. You could integrate this as a post-hoc conversation monitor: after each turn, extract residual stream activations, compute the five trajectory features incrementally, and flag conversations that cross a detection threshold. The latency overhead is low—computing residual activations is a single forward pass you're already doing, and five scalar features are negligible to compute. However, there are practical constraints: first, the probes are model-specific, so you'd need to train separate detection heads for each model family you deploy, adding engineering and validation overhead. Second, the generalization results are incomplete (the abstract cuts off), so you'll need to validate that performance holds on your specific data distribution and attack variations before relying on it in production. The 93.8% detection rate on synthetic data is strong, but real-world adversarial creativity may differ, requiring continuous monitoring and retraining.
Limitations and When Not to Use This
The most critical limitation is that model-specific probes don't transfer across architectures, meaning you cannot train once and deploy universally—you need separate detection heads for 7B, 13B, 34B, 70B model families, and likely for different base model architectures entirely. The paper only evaluates on synthetic attacks and two datasets (LMSYS-Chat-1M and SafeDial), so generalization to novel, naturally-occurring adversarial patterns remains unclear; adversarial creativity in the wild may sidestep the activation signature. The abstraction cuts off before revealing leave-one-source-out evaluation results, which is critical for understanding whether the detector truly generalizes or overfits to dataset-specific artifacts. Additionally, the approach assumes white-box access to residual streams—a significant constraint in federated or API-only deployment scenarios where you cannot extract intermediate activations. Finally, the paper doesn't address whether an adaptive adversary who becomes aware of activation-level monitoring could deliberately craft attacks that mimic benign activation trajectories, a cat-and-mouse dynamic that requires ongoing research.
Research Context
This work builds on a growing body of research into activation-space analysis for model understanding and safety (e.g., mechanistic interpretability, hidden-space anomaly detection). It advances the prompt injection defense literature by moving beyond rule-based and text-embedding defenses to exploit the model's own computational dynamics; prior work focused on input validation, output filtering, or semantic similarity checks, all of which operate on text representation alone. The paper contributes to the safety evaluation benchmark landscape by introducing a detection task with synthetic and real-world conversational data, extending benchmarks like TrustGPT and others that test robustness to adversarial prompts. The trajectory-feature-based classification approach opens a research direction: what other adversarial attack types leave distinctive activation signatures, and can a unified activation monitor detect multiple threats (prompt injection, jailbreaking, exfiltration) through geometric anomaly detection?
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::