LLM Safety From Within: Detecting Harmful Content with Internal Representations
| Authors | Difan Jiao et al. |
| Year | 2026 |
| HF Upvotes | 15 |
| arXiv | 2604.18519 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Guard models are widely used to detect harmful content in user prompts and LLM responses. However, state-of-the-art guard models rely solely on terminal-layer representations and overlook the rich safety-relevant features distributed across internal layers. We present SIREN, a lightweight guard model that harnesses these internal features. By identifying safety neurons via linear probing and combining them through an adaptive layer-weighted strategy, SIREN builds a harmfulness detector from LLM internals without modifying the underlying model. Our comprehensive evaluation shows that SIREN substantially outperforms state-of-the-art open-source guard models across multiple benchmarks while using 250 times fewer trainable parameters. Moreover, SIREN exhibits superior generalization to unseen benchmarks, naturally enables real-time streaming detection, and significantly improves inference efficiency compared to generative guard models. Overall, our results highlight LLM internal states as a promising foundation for practical, high-performance harmfulness detection.
Engineering Breakdown
Plain English
This paper introduces SIREN, a lightweight safety guard model that detects harmful content in LLM prompts and responses by leveraging internal layer representations rather than just the final output layer. Traditional guard models only look at terminal-layer features, but SIREN identifies safety-relevant neurons distributed across all layers through linear probing and combines them with an adaptive weighting strategy. The results are compelling: SIREN outperforms state-of-the-art open-source guard models while using 250 times fewer trainable parameters, and it generalizes better to unseen attack patterns. This represents a significant efficiency gain for deploying safety mechanisms without modifying the base language model.
Core Technical Contribution
The core innovation is recognizing and exploiting safety-relevant information distributed across internal LLM layers, rather than treating the model as a black box that only exposes final representations. SIREN's technical novelty lies in two complementary mechanisms: (1) systematic identification of safety neurons via linear probing across all transformer layers, and (2) an adaptive layer-weighted aggregation strategy that learns to combine these neurons with different importance weights. This approach enables a lightweight safety classifier (250× fewer parameters) that doesn't require modifying or fine-tuning the underlying LLM, contrasting sharply with prior work that either adds trainable safety layers or requires full model adaptation. The architectural insight—that safety signals are learnable linear functions of internal activations—enables exceptional parameter efficiency while maintaining or exceeding performance.
How It Works
SIREN operates in two phases: probing and aggregation. In the probing phase, for each transformer layer in the LLM, SIREN trains lightweight linear probes on top of the hidden states to predict a harmfulness label (binary or multi-class). These probes identify which neurons and layer combinations contain safety-relevant information. During inference, SIREN extracts activations from all layers of the LLM when processing a prompt or response, runs the trained linear probes on each layer's representations, and aggregates their predictions using learned layer weights. The adaptive weighting mechanism allows SIREN to automatically learn that some layers are more informative for safety than others—for example, middle layers might capture semantic content while later layers refine it. The final harmfulness prediction is a weighted combination of all these per-layer probe outputs, producing a single safety score that determines whether content should be flagged. Critically, this requires no modifications to the LLM itself; SIREN only stores lightweight probe weights and aggregation parameters.
Production Impact
For production systems, SIREN offers a compelling trade-off: you get state-of-the-art safety detection while keeping inference latency minimal and avoiding expensive fine-tuning of large models. Deployment is straightforward—add a forward pass through the LLM to collect internal activations, then pass them through tiny linear probes (essentially matrix multiplications) and aggregate predictions. The 250× parameter reduction means SIREN can run efficiently on CPU or GPU edge devices, making it feasible to deploy safety checks at scale without the compute overhead of running separate large guard models. You would integrate SIREN by: (1) calibrating the probes on a labeled safety dataset, (2) storing probe weights alongside your LLM inference service, and (3) querying probes at inference time before allowing responses to reach users. The main trade-off is that you're locked to a specific LLM architecture—changing models requires retraining the probes—and the probes depend on access to internal activations, which some inference frameworks may not easily expose.
Limitations and When Not to Use This
The paper doesn't address the critical question of whether safety signals learned from one LLM transfer to others, which is essential for production use where you might swap models. SIREN assumes you have labeled safety training data and access to internal activations of the base LLM—a constraint if you're using proprietary closed-source models or inference APIs that don't expose hidden states. The abstract cuts off discussing generalization claims, so it's unclear how well SIREN handles adversarial attacks specifically designed to fool it or how it performs on novel harm categories unseen during probe training. Additionally, the approach conflates different types of harm (toxic language, jailbreaks, misinformation) into a single linear probe per layer, which may not capture domain-specific nuances; a harmful prompt in one context might be benign in another. The paper would benefit from ablations on which layers matter most and analysis of failure cases where the probes gave incorrect predictions.
Research Context
This work builds on a growing body of research showing that safety-relevant information is encoded in LLM internals, extending beyond earlier work that treated models as black boxes. It relates to mechanistic interpretability research (identifying specific neurons responsible for behaviors) and refines the observation that linear probes can extract useful information from hidden states. SIREN advances the practical application of this insight by showing that lightweight aggregation of layer-wise probes achieves better performance than state-of-the-art dedicated guard models like OpenAI's Moderation API or similar open-source baselines. The paper opens a research direction on multi-layer safety signal fusion and could inspire follow-up work on which layer combinations are most informative for different harm categories, how these signals change across model scales and architectures, and whether probe weights can be distilled or transferred across related models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::