RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography
| Authors | Mélanie Roschewitz et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2604.15231 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Vision-language models (VLM) have markedly advanced AI-driven interpretation and reporting of complex medical imaging, such as computed tomography (CT). Yet, existing methods largely relegate clinicians to passive observers of final outputs, offering no interpretable reasoning trace for them to inspect, validate, or refine. To address this, we introduce RadAgent, a tool-using AI agent that generates CT reports through a stepwise and interpretable process. Each resulting report is accompanied by a fully inspectable trace of intermediate decisions and tool interactions, allowing clinicians to examine how the reported findings are derived. In our experiments, we observe that RadAgent improves Chest CT report generation over its 3D VLM counterpart, CT-Chat, across three dimensions. Clinical accuracy improves by 6.0 points (36.4% relative) in macro-F1 and 5.4 points (19.6% relative) in micro-F1. Robustness under adversarial conditions improves by 24.7 points (41.9% relative). Furthermore, RadAgent achieves 37.0% in faithfulness, a new capability entirely absent in its 3D VLM counterpart. By structuring the interpretation of chest CT as an explicit, tool-augmented and iterative reasoning trace, RadAgent brings us closer toward transparent and reliable AI for radiology.
Engineering Breakdown
Plain English
This paper introduces RadAgent, an AI agent system that generates chest CT radiology reports through a step-by-step, interpretable process rather than as opaque black-box outputs. The key innovation is that every report comes with a complete trace showing how the agent made decisions and used tools, allowing radiologists to inspect and validate the reasoning. The system outperforms CT-Chat, a 3D vision-language model baseline, across multiple evaluation dimensions. This addresses a critical gap in medical AI deployment: clinicians need to understand not just what findings are reported, but how the system derived them.
Core Technical Contribution
RadAgent's core contribution is the marriage of tool-using agents with medical image interpretation, creating an interpretability-first architecture for report generation. Rather than feeding CT scans directly into a single VLM for end-to-end report generation, the system decomposes the problem into intermediate steps where the agent can call specialized tools, make explicit decisions, and maintain a queryable trace of its reasoning. This is fundamentally different from prior VLM approaches (like CT-Chat) that treat report generation as a monolithic neural transformation. The innovation lies not in a single new algorithm, but in the systems-level design that prioritizes clinician interpretability through agent scaffolding and tool composition.
How It Works
RadAgent operates as an agentic loop: given a CT scan input, it begins by invoking tools to extract localized findings, anatomical structures, or quantitative measurements from the image. Each tool invocation is logged with its inputs, outputs, and confidence signals. The agent then uses these intermediate results to build a structured representation of findings, filtering and synthesizing them into clinically relevant observations. Next, it applies rule-based or learned ranking to prioritize findings by clinical significance. Finally, the agent generates the natural language report by conditioning on the ranked findings and their supporting tool outputs, with each claim traceable back to a specific tool call and its intermediate result. The full execution trace—every tool call, parameter, result, and decision point—remains attached to the final report for clinician inspection.
Production Impact
Deploying RadAgent in a radiology workflow shifts the integration model significantly. Instead of replacing radiologists with a black-box system, it becomes a second-reader or assisted-reporting tool where clinicians actively validate intermediate findings before they're finalized into a report. This reduces liability risk and increases adoption likelihood in regulated medical settings. The trade-offs are real: the multi-step agentic approach adds latency (multiple tool calls and decision loops instead of a single forward pass) and requires maintaining separate, interpretable tools for each imaging task—higher operational complexity than a monolithic VLM. Compute cost may increase due to the iterative nature, though batch inference optimizations can mitigate this. The value proposition is strongest in high-stakes clinical environments where interpretability and clinician confidence are worth the efficiency trade-off.
Limitations and When Not to Use This
The paper's abstracts cuts off, so full quantitative comparisons aren't visible, but several limitations are evident from the framing. First, the approach assumes that intermediate tool outputs are themselves reliable and interpretable—if the tools are black boxes or produce spurious intermediate results, the trace doesn't solve the problem. Second, the system requires explicit tool definitions for each imaging modality and anatomical region, limiting generalization; scaling to new organ systems or rare pathologies requires manual tool engineering. Third, the paper doesn't address how to handle ambiguous or conflicting signals from multiple tools, or how to build tools robust enough for clinical deployment. Finally, clinician adoption depends on the trace being genuinely useful, not just verbose—poorly designed intermediate steps could add cognitive load rather than reduce uncertainty.
Research Context
This work builds on the recent success of vision-language models in medical imaging (CT-Chat and similar systems) while responding to a known weakness: VLMs lack interpretability in clinical workflows. It sits at the intersection of two research directions: (1) tool-using AI agents (popularized by work on ReAct and similar frameworks) and (2) medical AI systems that prioritize clinician integration. The paper likely evaluates on standard radiology benchmarks (chest CT datasets, possibly internal clinical datasets), pushing the frontier on both generation quality and interpretability metrics. It opens a research direction around hybrid agentic-neural systems for high-stakes domains, where pure neural scaling has hit adoption barriers due to trust and explainability requirements.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::