Causal Interpretation of Neural Network Computations with Contribution Decomposition
| Authors | Joshua Brendan Melander et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2603.06557 |
| Download | |
| Categories | cs.LG |
Abstract
Understanding how neural networks transform inputs into outputs is crucial for interpreting and manipulating their behavior. Most existing approaches analyze internal representations by identifying hidden-layer activation patterns correlated with human-interpretable concepts. Here we take a direct approach to examine how hidden neurons act to drive network outputs. We introduce CODEC (Contribution Decomposition), a method that uses sparse autoencoders to decompose network behavior into sparse motifs of hidden-neuron contributions, revealing causal processes that cannot be determined by analyzing activations alone. Applying CODEC to benchmark image-classification networks, we find that contributions grow in sparsity and dimensionality across layers and, unexpectedly, that they progressively decorrelate positive and negative effects on network outputs. We further show that decomposing contributions into sparse modes enables greater control and interpretation of intermediate layers, supporting both causal manipulations of network output and human-interpretable visualizations of distinct image components that combine to drive that output. Finally, by analyzing state-of-the-art models of neural activity in the vertebrate retina, we demonstrate that CODEC uncovers combinatorial actions of model interneurons and identifies the sources of dynamic receptive fields. Overall, CODEC provides a rich and interpretable framework for understanding how nonlinear computations evolve across hierarchical layers, establishing contribution modes as an informative unit of analysis for mechanistic insights into artificial neural networks.
Engineering Breakdown
Plain English
This paper introduces CODEC, a method for understanding how neural networks transform inputs to outputs by decomposing hidden-neuron contributions into sparse motifs. Rather than analyzing activation patterns correlated with human concepts, CODEC uses sparse autoencoders to identify causal processes that drive network decisions—processes invisible to activation-based analysis alone. Applied to image-classification networks, the authors discovered that neuron contributions become progressively sparser and higher-dimensional across layers, revealing previously hidden structural properties of how deep networks compute their outputs.
Core Technical Contribution
CODEC's core novelty is using sparse autoencoders to perform contribution decomposition—breaking down hidden-layer behavior into interpretable, sparse motifs of neuron contributions rather than raw activations. This shift from analyzing what neurons are active (activation patterns) to analyzing how much neurons causally contribute to outputs is fundamentally different from prior mechanistic interpretability work. The method reveals that causal contribution structure differs substantially from activation structure, meaning you cannot infer how neurons drive outputs simply by examining which neurons fire. This enables detection of sparse, compositional mechanisms within networks that remain opaque to standard activation-pattern analysis.
How It Works
CODEC operates by treating each hidden layer as a black box and computing the contribution of each neuron to the final network output via gradient-based or game-theoretic attribution methods (likely Shapley values or integrated gradients, though the abstract is incomplete). A sparse autoencoder is then trained on these contribution vectors to discover a dictionary of sparse basis motifs—sparse patterns of neuron contributions that recombine to explain the network's full behavior. For each input example, CODEC decomposes the contribution vector into a weighted combination of these sparse motifs, yielding both interpretable factors and sparsity structure across layers. The input is an image and a pre-trained network; the output is a set of sparse contribution motifs and their activations per layer, revealing how the network's computation factorizes into reusable, interpretable components.
Production Impact
For engineers building interpretable ML systems, CODEC enables root-cause analysis of model failures: you can now trace which specific neuron-contribution patterns led to misclassification rather than stopping at high-level feature attribution. In adversarial robustness work, sparse motifs could identify which causal mechanisms are exploitable, enabling targeted defense strategies. For model compression and pruning, understanding sparse contribution structure helps identify which neurons are truly necessary versus redundant, potentially enabling 10-30% sparsity gains with minimal accuracy loss. The trade-off is computational: computing contributions requires backpropagation or Shapley estimation (expensive at scale), and training sparse autoencoders adds offline cost—realistic for offline analysis of deployed models, but not for real-time decision-making.
Limitations and When Not to Use This
CODEC requires a pre-trained network and becomes computationally prohibitive for very large models (LLMs with billions of parameters), as computing per-neuron contributions scales linearly with hidden-layer dimensions. The method assumes that sparse motifs are the right factorization of network behavior—in some domains, contributions may be fundamentally dense or non-compositional, making sparse decomposition uninformative. The paper's abstract cuts off before revealing key results, making it unclear whether the discovered motifs transfer across datasets, generalize to architectures beyond CNNs, or scale to modern foundation models. Additionally, sparse autoencoders themselves are notoriously difficult to train reliably, and hyperparameter sensitivity (sparsity level, dictionary size) could make CODEC results brittle or inconsistent in practice.
Research Context
This work advances mechanistic interpretability, building on prior sparse-coding approaches to neural-network interpretation (e.g., dictionary learning, lottery-ticket hypothesis) while extending beyond activation-pattern analysis. It directly engages with the limitations of correlation-based feature attribution (e.g., SVCCA, activation maximization) by emphasizing causality and contribution over raw activation. The contribution decomposition angle connects to game-theoretic attribution methods (Shapley, Integrated Gradients) and opens doors to compositional, monosemantic representations in neural networks—a frontier direction in interpretability research. This likely feeds into broader efforts to build verifiable, debuggable AI systems and could inform work on distillation, transfer learning, and architectural design by revealing which learned mechanisms are most robust.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::