Healthcare AI GYM for Medical Agents
| Authors | Minbyul Jeong |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2605.02943 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Clinical reasoning demands multi-step interactions -- gathering patient history, ordering tests, interpreting results, and making safe treatment decisions -- yet a unified training environment provides the breadth of clinical domains and specialized tools to train generalizable medical AI agents through reinforcement learning remains elusive. We present a comprehensive empirical study of multi-turn agentic RL for medical AI, built on , a gymnasium-compatible environment spanning 10 clinical domains with 3.6K+ tasks, 135 domain-specific tools, and a knowledge base of 828K medical passages. Our analysis reveals that agentic multi-turn structure degrades into verbose single-turn monologues, characterized by monotonic length explosion and a simultaneous erosion of tool-use frequency. We characterize how this collapse, alongside distillation instability, stems from the misalignment of sparse terminal rewards with sequential clinical trajectories. We find that vanilla GRPO achieves strong final accuracy on some benchmarks but suffers from training instability, evidenced by significant oscillations in response length and prolonged convergence periods. To improve training efficiency and stability, we propose Turn-level Truncated On-Policy Distillation (TT-OPD), a self-distillation framework where a gradient-free EMA teacher leverages outcome-privileged information to provide dense, outcome-aware KL regularization at every conversation turn. TT-OPD achieves the best performance on 10 of 18 benchmarks with an average +3.9~pp improvement over the non-RL baseline with faster early convergence, controlled response length, and sustained multi-turn tool use.
Engineering Breakdown
Plain English
This paper presents Healthcare AI GYM, a comprehensive reinforcement learning training environment for medical AI agents that spans 10 clinical domains with 3,600+ tasks, 135 specialized medical tools, and 828K medical knowledge passages. The authors conducted an empirical study of multi-turn agentic RL for medical reasoning and discovered a critical failure mode: agents trained in this setup tend to produce verbose, single-turn monologues rather than properly structured multi-step interactions, with monotonically increasing response lengths coupled with decreasing tool usage. The work addresses a major gap in medical AI — the lack of a unified, gymnasium-compatible environment that can train generalizable agents across diverse clinical tasks and specialized domains. This empirical analysis reveals fundamental challenges in scaling agentic reasoning for healthcare applications.
Core Technical Contribution
The core contribution is Healthcare AI GYM itself — a large-scale, standardized gymnasium-compatible environment purpose-built for training and benchmarking medical AI agents. Unlike prior work that focuses on single-domain medical tasks or doesn't provide structured multi-turn environments, this paper offers breadth (10 clinical domains, 3.6K+ tasks) and depth (135 domain-specific tools, 828K medical passages) in a single unified framework. The key technical novelty is the characterization of how multi-turn agentic RL degrades in medical contexts: the authors empirically demonstrate that the standard agentic structure leads to a pathological behavior where agents abandon tool use and produce increasingly verbose outputs without meaningful interaction steps. This discovery is valuable because it surfaces a specific failure mode that hasn't been clearly documented in prior medical AI research and opens paths toward architectural or training fixes.
How It Works
The environment is built as a gymnasium-compatible simulator that represents medical decision-making as a Markov Decision Process where the agent must complete clinical tasks by sequencing interactions with specialized tools and domain knowledge. On each turn, the agent receives a task (e.g., 'diagnose patient with chest pain'), observes the current state (patient history, lab results, imaging), and must decide whether to request a specific tool (one of 135 domain-specific medical tools), query the knowledge base, or make a clinical decision. The reward structure incentivizes correct clinical reasoning — agents are scored on whether they gather appropriate diagnostic information, order sensible tests, and make safe treatment recommendations aligned with clinical guidelines. The agent operates over multiple turns, building a chain of reasoning; however, the empirical finding is that agents learn to collapse this into longer, wordier single outputs rather than using discrete tool calls. The observation traces reveal that as episode length increases during training, the agent's tool invocations per episode paradoxically decrease, suggesting the model learns a shortcut that avoids structured interaction.
Production Impact
For engineers building production medical AI systems, this work provides two immediate values: a large, open benchmark for evaluating multi-domain medical reasoning (reducing the need to build custom environments), and a cautionary empirical finding about agentic training dynamics. If you adopt this environment for training medical agents, you immediately get structured task definitions across 10 high-value clinical domains rather than training on isolated tasks, which should improve generalization and reduce the number of domain-specific models needed. However, the identified degradation mode — verbose outputs with low tool utilization — directly impacts production reliability: if deployed without addressing this, your agent might produce longer but less trustworthy outputs, with reduced interpretability and debuggability since tool calls are the audit trail in clinical AI. Engineers would need to incorporate the findings as hard constraints in their training loop: explicit penalties for output length, minimum tool-use quotas, or architectural changes (e.g., forcing discrete action selection) to prevent collapse into monologue behavior. The compute footprint is substantial (training across 3.6K+ tasks with RL), so careful consideration of model scale and environment sampling strategy is needed.
Limitations and When Not to Use This
The paper identifies a critical pathology in multi-turn agentic RL for medical tasks but does not provide solutions, only empirical characterization — the work is diagnostic rather than prescriptive, leaving open the question of how to fix verbose collapse. The assumption that gymnasium compatibility and task breadth alone are sufficient for training generalizable medical agents may not hold; the paper doesn't establish whether trained agents actually transfer to unseen clinical domains or new patient populations. The 828K medical passages knowledge base and 135 tools are curated artifacts, and the paper doesn't discuss how to maintain or update these as medical practice evolves, nor does it address how errors or hallucinations in tool outputs propagate through the multi-turn reasoning chain. Additionally, the paper appears incomplete (the abstract trails off mid-sentence), suggesting full results, ablation studies, and proposed fixes are not yet available, making it difficult to assess whether the authors have direction on resolving the degradation issue. Real medical deployment requires formal safety verification and human expert alignment, but no mention is made of how those constraints integrate with the RL training procedure.
Research Context
This work sits at the intersection of three research threads: large-scale RL benchmarking (following trends like Atari, Minecraft environments), multi-agent and agentic reasoning systems (building on recent work in tool-using LLMs and ReAct-style prompting), and medical AI evaluation (extending beyond isolated task accuracy to multi-step clinical reasoning). The paper directly addresses a gap identified by the broader medical AI community — the absence of a unified environment for multi-domain clinical reasoning comparable to what gymnasium environments provide for robotics and game-playing RL. It advances on prior medical AI work by moving beyond single-task prediction (e.g., disease classification) to sequential decision-making under uncertainty, closer to how clinicians actually reason. The discovery of the verbose-collapse failure mode is novel in the medical context and likely to inform future work on training agentic systems, suggesting that standard RL training procedures may need adaptation for safety-critical domains where interpretability and tool use are core requirements.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::