SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning
| Authors | Alejandra Perez et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.06570 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Surgeons don't just see -- they interpret. When an expert observes a surgical scene, they understand not only what instrument is being used, but why it was chosen, what risk it poses, and what comes next. Current surgical AI cannot answer such questions, largely because training data that explicitly encodes surgical reasoning is immensely difficult to annotate at scale. Yet surgical video lectures already contain exactly this -- explanations of intent, rationale, and anticipation, narrated by experts for the purpose of teaching. Though inherently noisy and unstructured, these narrations encode the reasoning that surgical AI currently lacks. We introduce SUREON, a large-scale video QA dataset that systematically harvests this training signal from surgical academic videos. SUREON defines 12 question categories covering safety assessment, decision rationale, and forecasting, and uses a multi-agent pipeline to extract and structure supervision at scale. Across 134.7K clips and 170 procedure types, SUREON yields 206.8k QA pairs and an expert-validated benchmark of 354 examples. To evaluate the extent to which this supervision translates to surgical reasoning ability, we introduce two models: SureonVLM, a vision-language model adapted through supervised fine-tuning, and SureonVLM-R1, a reasoning model trained with Group Relative Policy Optimization. Both models can answer complex questions about surgery and substantially outperform larger general-domain models, exceeding 84% accuracy on the SUREON benchmark while outperforming general-domain models on standard surgical perception tasks. Qualitative analysis of SureonVLM-R1 reveals explicit reasoning behavior, such as inferring operative intent from visual context.
Engineering Breakdown
Plain English
SUREON is a large-scale surgical video QA dataset that extracts surgical reasoning from educational surgical videos by mining expert narrations. The paper addresses a critical gap in surgical AI: current systems can identify instruments and actions, but cannot explain the reasoning behind surgical decisions—why a specific instrument was chosen, what risks it poses, or what step comes next. The authors harvest this training signal from surgical lecture videos where expert surgeons naturally explain their reasoning while teaching, creating a structured dataset with 12 reasoning categories. This approach transforms unstructured educational content into a scalable source of surgical reasoning data without requiring expensive expert annotation of raw surgical footage.
Core Technical Contribution
The core novelty is a systematic method to extract surgical reasoning from naturally-occurring expert narrations in educational videos, rather than requiring dedicated annotation of surgical video. The authors define 12 explicit surgical reasoning categories (intent, rationale, risk assessment, anticipation, etc.) and develop a pipeline to align these categories with video segments and transcribed narrations. This creates a high-quality training signal from data that already exists and is freely available, solving the annotation bottleneck that has prevented surgical AI from learning reasoning. The contribution is primarily a data-centric one—not a new architecture, but a novel way to acquire and structure training data that encodes expert decision-making at scale.
How It Works
The pipeline begins by collecting surgical lecture videos from academic sources, which naturally contain expert narration explaining surgical decisions and reasoning. The authors transcribe these narrations and align them temporally with video segments to create video-text pairs. Each narration segment is then annotated with one or more of the 12 surgical reasoning categories (e.g., 'instrument choice rationale', 'risk anticipation', 'technique justification'). The resulting dataset pairs video frames with temporally-aligned text explanations labeled by reasoning type. A vision-language model is then trained on this data using a video QA objective, where the model learns to answer questions about surgical scenes by retrieving and reasoning over the aligned textual explanations. At inference time, the model can answer questions about surgical video by understanding both the visual content and the encoded reasoning patterns from the training data.
Production Impact
For surgical AI systems in production, this enables building reasoning-aware surgical assistants that can not only detect instruments and actions but also explain their understanding to surgeons in real-time. A surgical video analysis pipeline could be enhanced to output not just 'instrument X is being used' but 'instrument X is being used because [rationale], which poses [risks], and the next step is likely [Y]'. This significantly improves trust and usability for clinical adoption, since surgeons want to understand why an AI system makes recommendations. The production trade-off is that you now need a vision-language model with multimodal reasoning capabilities (not just a classification model), which increases latency and compute requirements, and requires deployment infrastructure for handling video streams. The dataset also enables fine-tuning on specific surgical domains—a hospital could augment SUREON with their own surgical videos to adapt the reasoning patterns to their institutional practices.
Limitations and When Not to Use This
The dataset quality depends entirely on the quality and completeness of educational lecture narrations, which are inherently noisy and may not cover all critical surgical reasoning or edge cases that occur in real operating rooms. The 12 reasoning categories may not capture domain-specific reasoning in specialized surgical subdisciplines, and the approach assumes that educational videos contain representative reasoning patterns—which may not hold for rare complications or emergency situations. The paper does not address how well reasoning learned from educational contexts transfers to real intraoperative settings where time pressure and incomplete information create different decision-making patterns. Additionally, the work assumes access to high-quality transcriptions and temporal alignment between video and narration, which may be unavailable or expensive for non-English-language surgical content or older video archives.
Research Context
This work builds on the intersection of surgical AI, video understanding, and vision-language models (like CLIP, LLaVA, etc.). Prior surgical AI research focused primarily on task recognition and instrument segmentation with single-task datasets; SUREON represents a shift toward multi-task reasoning and understanding surgical intent. The paper extends recent work on mining weak supervision from naturally-occurring narrations in other domains (instructional videos, documentaries) to the high-stakes surgical domain. It positions surgical reasoning as a learnable signal that can be extracted from existing educational infrastructure, opening a new direction for dataset creation in medical AI that doesn't require new annotation efforts.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::