Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach
| Authors | Haolin Li et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2604.11547 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
While large language models hold promise for complex medical applications, their development is hindered by the scarcity of high-quality reasoning data. To address this issue, existing approaches typically distill chain-of-thought reasoning traces from large proprietary models via supervised fine-tuning, then conduct reinforcement learning (RL). These methods exhibit limited improvement on underrepresented domains like rare diseases while incurring substantial costs from generating complex reasoning chains. To efficiently enhance medical reasoning, we propose MedSSR, a Medical Knowledge-enhanced data Synthesis and Semi-supervised Reinforcement learning framework. Our framework first employs rare disease knowledge to synthesize distribution-controllable reasoning questions. We then utilize the policy model itself to generate high-quality pseudo-labels. This enables a two-stage, intrinsic-to-extrinsic training paradigm: self-supervised RL on the pseudo-labeled synthetic data, followed by supervised RL on the human-annotated real data. MedSSR scales model training efficiently without relying on costly trace distillation. Extensive experiments on Qwen and Llama demonstrate that our method outperforms existing methods across ten medical benchmarks, achieving up to +5.93% gain on rare-disease tasks. Our code is available at https://github.com/tdlhl/MedSSR.
Engineering Breakdown
Plain English
This paper addresses a critical bottleneck in medical AI: the lack of high-quality reasoning data for training large language models on complex medical tasks, particularly for rare diseases. The authors propose MedSSR, a framework that synthesizes distribution-controllable reasoning questions using rare disease knowledge, then applies semi-supervised reinforcement learning to improve model performance without requiring expensive proprietary model distillation. The approach targets the problem that existing methods rely heavily on distilling reasoning traces from large commercial models through supervised fine-tuning, which is costly and performs poorly on underrepresented medical domains. By combining knowledge-enhanced data synthesis with semi-supervised RL, MedSSR aims to achieve better reasoning capabilities on rare disease cases while reducing computational overhead.
Core Technical Contribution
The core innovation is a two-stage framework that decouples data synthesis from model training: first, rare disease knowledge is used to programmatically generate reasoning questions with controllable distributions (addressing data scarcity), then semi-supervised RL optimizes the model using both labeled and unlabeled data without requiring distillation from proprietary models. This is novel because prior work either relies on expensive supervised fine-tuning from large model outputs or applies vanilla RL to limited reasoning data, whereas MedSSR leverages domain knowledge to expand the training distribution synthetically and uses semi-supervised signals to improve sample efficiency. The knowledge-enhanced synthesis component is a departure from random data augmentation—it specifically targets underrepresented domains by injecting structured rare disease information into question generation. The semi-supervised RL component reduces dependence on high-quality labeled reasoning chains, lowering the barrier to entry for specialized medical domains.
How It Works
The framework operates in two main phases. In the synthesis phase, the system takes rare disease knowledge (structured medical information, taxonomies, or knowledge graphs) and uses it to generate reasoning questions with controllable distributions—meaning the authors can steer the synthetic data toward under-covered medical conditions rather than relying on naturally occurring data distributions. These synthesized questions become the training corpus. In the RL phase, the model learns from both labeled examples (where gold reasoning traces exist) and unlabeled examples (where only the question and answer exist), using semi-supervised objectives to leverage the larger pool of unlabeled data. The RL objective likely combines a supervised signal (reward from correct reasoning steps) with unsupervised signals (consistency regularization or self-supervised learning on unlabeled data). The model iteratively improves its ability to reason through medical problems by being exposed to a broader, knowledge-biased distribution of rare disease cases without needing to distill reasoning from a large commercial model at every step.
Production Impact
For teams building medical AI systems, this approach substantially reduces the cost and complexity of developing domain-specific reasoning capabilities. Instead of maintaining agreements with large commercial LLM providers to distill reasoning traces (which adds latency, licensing complexity, and recurring costs), teams can synthesize training data using internal medical knowledge bases and domain expertise, then train models end-to-end with semi-supervised RL. This is particularly valuable for rare disease diagnosis, drug interaction prediction, and clinical decision support—domains where annotated reasoning data is scarce. However, the adoption overhead includes: (1) building or integrating a rare disease knowledge base (ontology, knowledge graph, or structured data), (2) implementing semi-supervised RL infrastructure (adding complexity to training pipelines), and (3) validating that synthesized data doesn't introduce distribution shift artifacts. The inference latency and compute cost likely remain similar to standard LLM inference, but training cost should decrease due to reduced reliance on proprietary model calls.
Limitations and When Not to Use This
The paper does not address how to automatically validate that synthesized questions are medically accurate or clinically relevant—garbage-in-garbage-out risk is real if the rare disease knowledge source is incomplete or outdated. The semi-supervised RL approach assumes that unlabeled data follows a similar distribution to labeled data; if clinical practice shifts or new rare disease subtypes emerge, the model may degrade. The paper also appears incomplete in the abstract (cuts off mid-sentence), so critical details about evaluation metrics, baseline comparisons, and actual performance gains are missing, making it impossible to assess whether improvements are statistically significant or clinically meaningful. Additionally, it's unclear how the approach scales to truly novel rare diseases with minimal prior knowledge or how it handles cross-domain generalization when a model trained on one set of rare diseases encounters a different set in production.
Research Context
This work builds on the trend of knowledge-enhanced machine learning and semi-supervised learning in NLP, extending techniques like RLHF (reinforcement learning from human feedback) and distillation to the medical domain where proprietary model access is limited and domain knowledge is abundant. It responds to recent findings that chain-of-thought reasoning improves LLM performance on complex tasks, but distilling reasoning is expensive—prior work includes methods like chain-of-thought prompting and self-consistency, but few address the data scarcity problem for specialized domains. The paper likely contributes to benchmarks like MedQA, PubMedQA, or medical reasoning datasets by showing how to improve performance on underrepresented disease categories. It opens a research direction toward knowledge-grounded data synthesis for other specialized domains (legal reasoning, scientific discovery, financial analysis) where domain expertise exists but labeled reasoning data is scarce.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::