PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-04 with 8 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Ruoqi Liu et al. |
| Year | 2026 |
| HF Upvotes | 8 |
| arXiv | 2605.02240 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We introduce PhysicianBench, a benchmark for evaluating LLM agents on physician tasks grounded in real clinical setting within electronic health record (EHR) environments. Existing medical agent benchmarks primarily focus on static knowledge recall, single-step atomic actions, or action intent without verifiable execution against the environment. As a result, they fail to capture the long-horizon, composite workflows that characterize real clinical systems. PhysicianBench comprises 100 long-horizon tasks adapted from real consultation cases between primary care and subspecialty physicians, with each task independently reviewed by a separate panel of physicians. Tasks are instantiated in an EHR environment with real patient records and accessed through the same standard APIs used by commercial EHR vendors. Tasks span 21 specialties (e.g., cardiology, endocrinology, oncology, psychiatry) and diverse workflow types (e.g., diagnosis interpretation, medication prescribing, treatment planning), requiring an average of 27 tool calls per task. Solving each task requires retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. Each task is decomposed into structured checkpoints (670 in total across the benchmark) capturing distinct stages of completion graded by task-specific scripts with execution-grounded verification. Across 13 proprietary and open-source LLM agents, the best-performing model achieves only 46% success rate (pass@1), while open-source models reach at most 19%, revealing a substantial gap between current agent capabilities and the demands of real-world clinical workflows. PhysicianBench provides a realistic and execution-grounded benchmark for measuring progress toward autonomous clinical agents.
Engineering Breakdown
Plain English
PhysicsonBench is a benchmark that evaluates LLM agents on realistic physician tasks using actual EHR systems and real patient records, rather than static knowledge tests. The benchmark includes 100 long-horizon tasks derived from real clinical consultations between primary care and specialty physicians, each independently validated by physician panels, and agents interact with EHRs through the same commercial APIs that real clinicians use.
Key Engineering Insight
The core technical shift is moving from evaluating isolated, atomic LLM actions to measuring performance on multi-step clinical workflows that require sequential decision-making, state management across system interactions, and verification against actual environment outcomes—this exposes brittleness that single-action benchmarks miss.
Why It Matters for Engineers
Production clinical AI systems fail not because models lack medical knowledge, but because they can't execute complex workflows reliably in real EHR environments where each step depends on previous results and API responses. This benchmark directly tests what breaks in production: long-horizon reasoning, error recovery, and integration with legacy systems—not just whether a model can answer a medical question.
Research Context
Prior medical AI benchmarks (like MedQA, PubMedQA) optimized for knowledge recall rather than executable workflows. PhysicianBench advances the field by grounding evaluation in actual clinical environments with real patient data and standard APIs, closing the gap between what existing benchmarks measure and what deployed medical agents actually need to do in hospital systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::