Towards Faithful Multimodal Concept Bottleneck Models
| Authors | Pierre Moreau et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.13163 |
| Download | |
| Categories | cs.CV, cs.LG |
Abstract
Concept Bottleneck Models (CBMs) are interpretable models that route predictions through a layer of human-interpretable concepts. While widely studied in vision and, more recently, in NLP, CBMs remain largely unexplored in multimodal settings. For their explanations to be faithful, CBMs must satisfy two conditions: concepts must be properly detected, and concept representations must encode only their intended semantics, without smuggling extraneous task-relevant or inter-concept information into final predictions, a phenomenon known as leakage. Existing approaches treat concept detection and leakage mitigation as separate problems, and typically improve one at the expense of predictive accuracy. In this work, we introduce f-CBM, a faithful multimodal CBM framework built on a vision-language backbone that jointly targets both aspects through two complementary strategies: a differentiable leakage loss to mitigate leakage, and a Kolmogorov-Arnold Network prediction head that provides sufficient expressiveness to improve concept detection. Experiments demonstrate that f-CBM achieves the best trade-off between task accuracy, concept detection, and leakage reduction, while applying seamlessly to both image and text or text-only datasets, making it versatile across modalities.
Engineering Breakdown
Plain English
This paper addresses a critical gap in interpretable AI: how to build multimodal concept bottleneck models (CBMs) that faithfully explain their predictions while maintaining accuracy. CBMs route decisions through human-understandable concepts rather than black-box features, but existing approaches struggle when concepts leak task-relevant information or when working across vision and language modalities simultaneously. The authors introduce f-CBM, a framework that tackles both concept detection and leakage mitigation together on a vision-language backbone, rather than treating them as separate problems that trade off against each other.
Core Technical Contribution
The core novelty is a unified framework that addresses the faithfulness problem in multimodal CBMs by explicitly handling information leakage at the concept bottleneck layer. Prior work either focused on improving concept detection accuracy or preventing leakage, but typically degraded performance on the other dimension. f-CBM introduces architectural and training mechanisms that allow concepts to be properly learned while simultaneously constraining what information flows through them into the final prediction layer. The framework leverages vision-language models as a foundation, enabling concepts to be grounded in both visual and textual semantics rather than isolated to a single modality.
How It Works
The f-CBM architecture takes multimodal inputs (images and text) and processes them through a shared vision-language encoder to extract rich feature representations. These representations are then passed to a concept prediction layer that learns to map inputs to interpretable, human-recognizable concepts (e.g., 'has_fur', 'is_furry_animal'). A critical component is the bottleneck mechanism that enforces leakage constraints—it ensures the concept representations contain only their intended semantic information by regularizing against task-relevant but concept-unrelated signal. The final prediction layer takes only the bottleneck concept outputs (not the original features), forcing the model to rely on faithful concept representations rather than reconstructing leaked information. During training, the framework jointly optimizes concept detection accuracy and leakage prevention, using techniques like mutual information minimization or conditional independence constraints to prevent concepts from encoding task shortcuts.
Production Impact
For production systems, f-CBM enables building AI pipelines where predictions come with faithful explanations that stakeholders can actually audit and trust. In regulated domains (healthcare, finance, lending), this means you can demonstrate to regulators exactly which concepts influenced a decision, rather than providing post-hoc explanations that may not reflect the model's actual reasoning. The multimodal aspect is critical for real-world applications—many decisions require both visual context and text (medical imaging with reports, e-commerce with product images and descriptions). The trade-off is increased training complexity and latency: you need to optimize three objectives simultaneously (concept accuracy, leakage prevention, and task accuracy), which typically requires careful hyperparameter tuning and longer training time. Integration requires defining your concept taxonomy upfront in collaboration with domain experts, adding a non-trivial engineering overhead before model training begins.
Limitations and When Not to Use This
The paper assumes concepts can be clearly defined and labelled consistently by humans, which breaks down in domains with subjective or emergent concepts. The approach requires multimodal training data with concept annotations, which is expensive and may not exist for specialized domains. Scalability to very large concept vocabularies (hundreds or thousands of concepts) remains unclear—the mutual information constraints may become computationally intractable. The framework's reliance on vision-language backbones like CLIP means it inherits those models' limitations (potential bias, dataset-specific biases, limited reasoning capability). The paper doesn't address temporal dynamics or sequential decision-making, limiting applicability to problems where a single prediction suffices.
Research Context
This work extends concept bottleneck models, which originated in interpretable ML but faced accuracy-interpretability trade-offs when scaled to complex tasks. Recent work by Been Kim's group and others showed CBMs could be viable with proper concept learning, but the multimodal setting with leakage concerns remained open. The paper builds on the rise of vision-language models (CLIP, BLIP) as foundation models that can ground concepts in both modalities. It fits into a broader research direction on mechanistic interpretability and faithful explanations, as opposed to post-hoc explanation techniques like LIME or attention visualization. The work opens questions about how to scale concept bottlenecks to even larger models and whether concepts learned this way transfer across domains or tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::