CXReasonAgent: Evidence-Grounded Diagnostic Reasoning Agent for Chest X-rays
| Authors | Hyungyung Lee et al. |
| Year | 2026 |
| Field | AI / Agents |
| arXiv | 2602.23276 |
| Download | |
| Categories | cs.AI |
Abstract
Chest X-ray plays a central role in thoracic diagnosis, and its interpretation inherently requires multi-step, evidence-grounded reasoning. However, large vision-language models (LVLMs) often generate plausible responses that are not faithfully grounded in diagnostic evidence and provide limited visual evidence for verification, while also requiring costly retraining to support new diagnostic tasks, limiting their reliability and adaptability in clinical settings. To address these limitations, we present CXReasonAgent, a diagnostic agent that integrates a large language model (LLM) with clinically grounded diagnostic tools to perform evidence-grounded diagnostic reasoning using image-derived diagnostic and visual evidence. To evaluate these capabilities, we introduce CXReasonDial, a multi-turn dialogue benchmark with 1,946 dialogues across 12 diagnostic tasks, and show that CXReasonAgent produces faithfully grounded responses, enabling more reliable and verifiable diagnostic reasoning than LVLMs. These findings highlight the importance of integrating clinically grounded diagnostic tools, particularly in safety-critical clinical settings.
Engineering Breakdown
Plain English
CXReasonAgent addresses a critical gap in clinical AI: large vision-language models generate plausible-sounding diagnoses from chest X-rays that aren't actually grounded in visual evidence, making them unreliable for clinical deployment. The paper presents an agent architecture that combines an LLM with specialized diagnostic tools to perform step-by-step, evidence-grounded reasoning directly from X-ray images, where each diagnostic conclusion is tied to specific visual findings. The authors introduce CXReasonDial, a multi-turn diagnostic dialogue dataset, to evaluate whether the agent can provide transparent, verifiable reasoning that clinicians can trust. This approach avoids the need for expensive model retraining when adding new diagnostic capabilities, improving both adaptability and reliability in real clinical workflows.
Core Technical Contribution
The core innovation is reframing chest X-ray interpretation as an agentic reasoning problem rather than a pure end-to-end vision-language task. Instead of training a monolithic LVLM to directly output diagnoses, CXReasonAgent uses an LLM as a reasoning engine that orchestrates calls to clinically-grounded diagnostic tools—these tools encapsulate domain knowledge and return interpretable, image-derived evidence. This tool-use framework creates an explicit chain of reasoning where each step is auditable: the agent identifies regions of interest, extracts visual evidence (e.g., nodules, consolidation), and chains these observations into structured diagnostic conclusions. The approach is fundamentally plug-and-play: new diagnostic tasks require adding new tools, not retraining the base model, which is a significant departure from the LVLMs-for-everything paradigm.
How It Works
The system operates in a multi-step loop: (1) the LLM receives a chest X-ray and clinical context, then reasons about what diagnostic tools to invoke and in what order; (2) specialized diagnostic tools (likely computer vision models or region detectors fine-tuned on radiology datasets) analyze the image and return structured evidence—locations and confidence scores for findings like nodules, fractures, or infiltrates; (3) the LLM integrates this evidence into a diagnostic hypothesis, citing specific image regions as justification; (4) if uncertainty remains or new information is needed, the agent selects additional tools or asks for clarification. The tool outputs are standardized (bounding boxes, confidence scores, feature descriptions), making the agent's reasoning transparent and verifiable against the raw image. The entire interaction is dialogue-based, allowing iterative refinement and human-in-the-loop verification—clinicians can see exactly which image regions support each conclusion.
Production Impact
For engineers deploying diagnostic AI in hospitals, this architecture dramatically improves clinical acceptance and regulatory compliance. Instead of a black-box model that occasionally hallucinates diagnoses, you get a system whose reasoning is auditable: each claim is backed by identifiable evidence in the image, which radiologists can verify. This transparency is critical for liability and trust—a clinician can confidently override or adjust the system's output because they understand its reasoning. The tool-based design also reduces retraining burden: to support a new pathology or diagnostic protocol, you add a fine-tuned detection tool rather than retraining the entire system, shrinking the ML engineering overhead. Trade-offs include increased latency (multiple sequential tool calls vs. single forward pass), higher infrastructure complexity (managing tool versions and dependencies), and the need to build and validate each diagnostic tool—this isn't a plug-and-play drop-in replacement but rather a framework requiring domain expertise to instantiate properly.
Limitations and When Not to Use This
The paper's evaluation relies on CXReasonDial, a synthetic or curated dataset of diagnostic dialogues, which may not capture the full distribution of real-world clinical ambiguity, edge cases, or the messy iterative nature of radiology consultation. The approach assumes that diagnostic tools are accurate and well-calibrated—if a tool fails silently or returns spurious detections, the LLM's reasoning becomes unreliable, and the appearance of evidence-grounding becomes a liability rather than an asset. The paper does not address how to handle conflicting evidence from multiple tools or how to weight tool outputs when they disagree, which is common in radiology (e.g., one model flags a lesion, another doesn't). Additionally, the framework requires significant upfront investment in tool development and validation for each diagnostic domain; it's not clear how the approach scales to rare pathologies with limited training data or how it performs on out-of-distribution X-ray types (e.g., portable chest X-rays, non-standard views).
Research Context
This work sits at the intersection of agentic AI, vision-language models, and clinical decision support. It builds on recent trends in tool-use for LLMs (exemplified by ReAct, Toolformer) but applies them to a safety-critical visual domain where hallucination is unacceptable. The paper implicitly critiques the LVLM-for-everything approach (e.g., GPT-4V, LLaVA) by showing that grounding requires explicit structure and domain-specific tools rather than scale alone. It contributes to the broader effort to make medical AI interpretable and trustworthy—related work includes attention visualization, concept-based explanations, and prototype learning in medical imaging. The introduction of CXReasonDial adds a benchmark for evaluating diagnostic reasoning quality, filling a gap between existing vision-language benchmarks (which test perception) and clinical decision support benchmarks (which test diagnosis).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::