Do Sparse Autoencoders Capture Concept Manifolds?
| Authors | Usha Bhalla et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.28119 |
| Download | |
| Categories | cs.LG, cs.AI |
Abstract
Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to independent linear directions. However, a growing body of evidence suggests that many concepts are instead organized along low-dimensional manifolds encoding continuous geometric relationships. This raises three basic questions: what does it mean for an SAE to capture a manifold, when do existing SAE architectures do so, and how? We develop a theoretical framework that answers these questions and show that SAEs can capture manifolds in two fundamentally different ways: globally, by allocating a compact group of atoms whose linear span contains the entire manifold, or locally, by distributing it across features that each selectively tile a restricted region of the underlying geometry. Empirically, we find that SAEs suboptimally recover continuous structures, mixing the global subspace and local tiling solutions in a fragmented regime we call dilution. This explains why manifold structure is rarely visible at the level of individual concepts and motivates post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions. More broadly, our results suggest that future representation learning methods should treat geometric objects, not just individual directions, as the basic units of interpretability.
Engineering Breakdown
Plain English
This paper challenges a core assumption in sparse autoencoders (SAEs)—that interpretable features align with independent linear directions. Instead, the authors present evidence that many neural network concepts are organized as low-dimensional manifolds with continuous geometric structure. They develop a theoretical framework answering three questions: what it means for SAEs to capture manifolds, when existing SAE architectures succeed at this, and how the mechanism works. The key finding is that SAEs can capture manifolds in two distinct ways: globally by clustering atoms whose linear span contains the full manifold, or locally by distributing the manifold across multiple sparse features.
Core Technical Contribution
The paper's novelty is a formal theoretical framework for understanding how SAEs interact with manifold-structured concepts rather than point features. Prior work treated SAE atoms as capturing independent concepts; this paper rigorously characterizes the geometric conditions under which compact groups of atoms can represent entire low-dimensional manifolds. The authors identify two fundamentally different capture mechanisms—global concentration versus local distribution—and provide mathematical conditions for when each occurs. This shifts SAE interpretation from a feature-centric model to a manifold-aware model, enabling principled design of SAE architectures for capturing continuous semantic structure.
How It Works
The framework starts by modeling concepts as low-dimensional manifolds embedded in high-dimensional activation spaces (e.g., a 2D face-pose manifold in a 10,000-dimensional model hidden layer). An SAE learns a sparse set of atoms (basis vectors) and reconstructs activations via a linear combination of these atoms using sparse coefficients. For global capture, a tight cluster of atoms spans a subspace large enough to contain the entire manifold with bounded reconstruction error. For local capture, different atoms activate for different regions of the manifold—as you move along the manifold, the active set of atoms changes, distributing the manifold across features. The theoretical contribution characterizes the atom count and geometric positioning required for each strategy and derives conditions on the manifold's intrinsic dimension and curvature that determine which strategy is optimal.
Production Impact
For engineers deploying SAEs for model interpretability, this framework provides concrete guidance on architecture choices. If you know your concepts are manifold-structured (continuous variations like pose, scale, or style), you can now predict whether to use denser atom clusters (global strategy) or sparser, more distributed representations (local strategy). This directly impacts interpretability—global captures give you a single region to analyze; local captures require understanding how atoms cluster and interact across manifold regions. The computational trade-off is that verifying manifold structure requires additional analysis (dimensionality testing, manifold learning), adding preprocessing overhead. For safety applications (identifying deceptive concepts), manifold awareness is critical: a single atom can no longer be assumed to cleanly represent a concept, so circuits and mechanistic interpretability workflows need manifold-specific debugging tools.
Limitations and When Not to Use This
The paper assumes concepts are actually organized as low-dimensional manifolds, which may not hold for all learned representations—some features could be genuinely high-dimensional or scattered. The theoretical framework doesn't provide algorithmic guidance for automatically detecting which capture strategy a real SAE is using, requiring manual analysis. The work focuses on characterizing existing SAE architectures but doesn't propose new SAE training objectives specifically optimized for manifold capture, leaving open whether current sparse-coding losses naturally encourage the right behavior. Additionally, the analysis applies to linear SAEs; nonlinear SAE variants and more complex manifold topologies (with self-intersections or non-convex geometry) are not addressed.
Research Context
This paper advances the growing literature on geometric structure in neural network representations, building on work showing that concepts like object poses and semantic attributes organize along manifolds rather than as isolated points. It directly challenges the implicit linearity assumption in SAE interpretability work (e.g., Anthropic's recent SAE papers) and connects to broader research on manifold hypothesis in deep learning. The manifold-SAE connection opens a research direction: designing SAE training objectives that explicitly encourage manifold-aware sparse codes, integrating topological learning into the sparse coding formulation. This work strengthens the foundation for mechanistic interpretability by clarifying what SAEs can and cannot isolate, which is essential for safety work on adversarial examples and hidden concept detection.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::