A multimodal slice discovery framework for systematic failure detection and explanation in medical image classification
| Authors | Yixuan Liu et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.24183 |
| Download | |
| Categories | cs.CV, cs.LG |
Abstract
Despite advances in machine learning-based medical image classifiers, the safety and reliability of these systems remain major concerns in practical settings. Existing auditing approaches mainly rely on unimodal features or metadata-based subgroup analyses, which are limited in interpretability and often fail to capture hidden systematic failures. To address these limitations, we introduce the first automated auditing framework that extends slice discovery methods to multimodal representations specifically for medical applications. Comprehensive experiments were conducted under common failure scenarios using the MIMIC-CXR-JPG dataset, demonstrating the framework's strong capability in both failure discovery and explanation generation. Our results also show that multimodal information generally allows more comprehensive and effective auditing of classifiers, while unimodal variants beyond image-only inputs exhibit strong potential in scenarios where resources are constrained.
Engineering Breakdown
Plain English
This paper presents an automated auditing framework that finds systematic failures in medical image classifiers by analyzing multimodal representations (combining image features with metadata/structured data). Using the MIMIC-CXR chest X-ray dataset, the researchers show their approach can detect failure patterns that traditional single-modality auditing methods miss, and automatically generate explanations for why those failures occur.
Key Engineering Insight
Systematic failures in medical classifiers often hide in patterns that only become visible when you combine image features with metadata—auditing just pixels or just patient data separately will miss critical failure modes that matter in production.
Why It Matters for Engineers
Medical AI systems in production fail silently on specific subgroups (age ranges, certain pathologies, imaging equipment types), and those failures often cause real patient harm. Current auditing catches obvious bugs but misses the 'works 99% of the time except for this specific combination of conditions' problems that slice discovery can uncover.
Research Context
Prior work on slice discovery focused on finding data subgroups in unimodal settings (either images or tabular data). This paper extends that to the multimodal medical domain where images plus patient metadata together determine outcomes—enabling systematic testing of combinations that single-modality auditing can't catch, which is essential before deploying classifiers in clinical workflows.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::