Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
| Authors | Rongyuan Tan et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.17761 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Interpretability tools are increasingly used to analyze failures of Large Language Models (LLMs), yet prior work largely focuses on short prompts or toy settings, leaving their behavior on commonly used benchmarks underexplored. To address this gap, we study contrastive, LRP-based attribution as a practical tool for analyzing LLM failures in realistic settings. We formulate failure analysis as contrastive attribution, attributing the logit difference between an incorrect output token and a correct alternative to input tokens and internal model states, and introduce an efficient extension that enables construction of cross-layer attribution graphs for long-context inputs. Using this framework, we conduct a systematic empirical study across benchmarks, comparing attribution patterns across datasets, model sizes, and training checkpoints. Our results show that this token-level contrastive attribution can yield informative signals in some failure cases, but is not universally applicable, highlighting both its utility and its limitations for realistic LLM failure analysis. Our code is available at: https://aka.ms/Debug-XAI.
Engineering Breakdown
Plain English
This paper introduces a practical framework for understanding why Large Language Models fail on real-world benchmarks by using contrastive attribution methods based on Layer-wise Relevance Propagation (LRP). The authors reformulate failure analysis as a problem of attributing logit differences between incorrect and correct output tokens to input tokens and internal model states, then propose an efficient extension that scales to long-context inputs by constructing cross-layer attribution graphs. They conduct a systematic empirical study across multiple benchmarks to identify patterns in how model failures occur, moving beyond prior work that only studied toy problems or short prompts. This gives engineers concrete interpretability insights into LLM failure modes on production-relevant tasks.
Core Technical Contribution
The core novelty is reformulating LLM failure analysis as a contrastive attribution problem—rather than explaining why a model made a wrong prediction in isolation, the framework explains why it chose an incorrect token over the correct one by computing the logit difference attribution. The authors introduce an efficient algorithm for constructing cross-layer attribution graphs that handles long-context inputs, which prior LRP approaches couldn't scale to. This makes attribution analysis practical for real benchmarks and realistic prompt lengths instead of toy settings. The key insight is that contrastive attribution is more informative for debugging because it isolates the specific decision boundary the model got wrong.
How It Works
The framework starts with an LLM generating an incorrect output token when a correct alternative existed—for example, predicting 'false' when the correct answer is 'true.' The method computes the logit score difference between these two tokens (logit_correct - logit_incorrect). Using LRP-based attribution, this logit difference is backpropagated through the neural network to decompose it into contributions from each input token and intermediate layer activations. The efficient extension enables cross-layer graph construction by carefully tracking which internal hidden states contribute most to the final logit difference, avoiding the exponential complexity of naive multi-layer attribution. The output is a visualization or quantitative ranking showing which input positions and model layers are most responsible for the incorrect choice, enabling root cause analysis of specific failure modes.
Production Impact
For engineers debugging LLM failures in production, this gives a principled way to identify whether errors stem from tokenization issues, early token misunderstandings, attention failures at specific layers, or final token prediction issues. Instead of black-box error analysis, you can now attribute failures to concrete input positions and model components, enabling targeted fine-tuning or prompt engineering fixes. The cross-layer graph construction is critical for production because it scales to realistic prompt lengths (the paper handles commonly used benchmarks), whereas prior attribution methods would time out or run out of memory. The trade-off is that computing full attribution graphs for every failure is computationally expensive—you'd likely use this selectively on representative failure cases rather than all errors, and integration requires storing intermediate activations during inference. This approach is most valuable when you have budget to analyze root causes deeply rather than just log error metrics.
Limitations and When Not to Use This
The paper focuses on analyzing failures that already occurred; it doesn't directly improve model performance or propose corrective actions—it's a diagnosis tool, not a cure. The efficiency gains from the cross-layer extension are relative but still require significant compute to generate attributions for long contexts, so it may not scale to analyzing failures in real-time production systems handling millions of requests. The framework assumes that logit differences are the right target for analysis, but this may miss higher-level reasoning failures that don't manifest as simple token choice errors—for example, a model might pick the right token at the wrong point in its output. Finally, the empirical study lacks details on which specific benchmarks were tested and what attribution patterns were discovered, so practitioners don't yet have concrete playbooks for common failure modes.
Research Context
This work builds on the layer-wise relevance propagation (LRP) interpretation literature and extends contrastive explanation methods to the LLM domain, moving beyond prior interpretability work that tested only on short, synthetic examples. It directly addresses a gap in the interpretability literature by scaling attribution analysis to realistic benchmarks and prompt lengths where LLMs actually fail in practice. The paper positions itself as a bridge between interpretability research (which tends to be theoretical) and practical failure debugging (which tends to be manual and ad-hoc). Future work likely includes automating the discovery of failure patterns across many failures, connecting attributions to mitigation strategies (prompting, fine-tuning, architecture changes), and extending the approach to multimodal or code-generation domains.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::