MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts
| Authors | Weiyue Li et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.06505 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language models (LLMs) are widely explored for reasoning-intensive research tasks, yet resources for testing whether they can infer scientific conclusions from structured biomedical evidence remain limited. We introduce MedConclusion, a large-scale dataset of 5.7M PubMed structured abstracts for biomedical conclusion generation. Each instance pairs the non-conclusion sections of an abstract with the original author-written conclusion, providing naturally occurring supervision for evidence-to-conclusion reasoning. MedConclusion also includes journal-level metadata such as biomedical category and SJR, enabling subgroup analysis across biomedical domains. As an initial study, we evaluate diverse LLMs under conclusion and summary prompting settings and score outputs with both reference-based metrics and LLM-as-a-judge. We find that conclusion writing is behaviorally distinct from summary writing, strong models remain closely clustered under current automatic metrics, and judge identity can substantially shift absolute scores. MedConclusion provides a reusable data resource for studying scientific evidence-to-conclusion reasoning. Our code and data are available at: https://github.com/Harvard-AI-and-Robotics-Lab/MedConclusion.
Engineering Breakdown
Plain English
This paper introduces MedConclusion, a dataset of 5.7 million PubMed abstracts designed to evaluate how well large language models can perform evidence-to-conclusion reasoning in biomedical domains. The dataset pairs the methods/results sections of real abstracts with their original author-written conclusions, providing naturally supervised training data for conclusion generation tasks. The authors evaluate multiple LLMs using different prompting strategies and scoring metrics, enabling systematic analysis of how well these models can synthesize scientific evidence into valid conclusions across different biomedical categories and journal quality levels. This work addresses a critical gap: while LLMs excel at many tasks, we lack benchmark datasets to measure their reasoning capabilities on real scientific inference problems.
Core Technical Contribution
The core contribution is MedConclusion itself—a large-scale, naturally supervised dataset for biomedical conclusion generation that leverages real PubMed metadata for structured evaluation. Unlike synthetic datasets or datasets built from web text, this uses authentic scientific abstracts where the evidence-to-conclusion mapping comes from actual peer-reviewed publications, ensuring the reasoning task reflects real scientific practice. The dataset includes journal-level metadata (biomedical category, SJR impact factor) enabling fine-grained subgroup analysis across research domains and publication venues, which is novel compared to flat, undifferentiated benchmarks. This enables researchers to study not just whether models can do conclusion generation, but whether their reasoning generalizes across journal quality, research domains, and evidence complexity.
How It Works
The pipeline extracts 5.7 million PubMed abstracts and segments each into evidence (methods/results) and conclusion sections using the structure of published abstracts. For each abstract, the input consists of all non-conclusion sections (background, methods, results)—the actual evidence a scientist would use to derive conclusions. The target output is the original author-written conclusion, providing ground truth supervision. Each instance is enriched with metadata: the biomedical category (e.g., cardiology, immunology) and the source journal's SJR (Scimago Journal Ranking), a peer-review quality metric. During evaluation, LLMs receive the evidence sections via different prompt templates (direct conclusion generation vs. summary-then-conclude) and their outputs are scored against reference conclusions using both exact-match metrics and likely semantic similarity measures, allowing assessment across different reasoning approaches.
Production Impact
For teams building AI systems that must synthesize scientific evidence or generate domain-specific summaries, this dataset provides a production-ready benchmark to measure model reasoning quality before deployment. You could use MedConclusion to evaluate whether your chosen LLM (GPT-4, Claude, open-source model) generalizes across different biomedical domains—critical for medical AI where reasoning consistency across specialties matters. The journal metadata enables you to measure robustness: does your model perform equally well on conclusions about high-impact research vs. niche journals? This helps identify whether the model has learned genuine reasoning or surface-level pattern matching. The main trade-off is that fine-tuning on this dataset requires processing millions of examples (substantial GPU memory and training time), but the alternative—deploying untested models for evidence synthesis—carries higher real-world risk in regulated domains like healthcare.
Limitations and When Not to Use This
The dataset is limited to PubMed abstracts, which follow a standardized structure and formatting that may not reflect how evidence appears in grant proposals, clinical reports, or unstructured text—so models trained on this may not generalize to messier real-world evidence formats. The conclusions in the dataset reflect what human authors chose to emphasize, which may not capture all valid inferences from the evidence; a model could generate a different-but-correct conclusion and be penalized by reference-based metrics. The paper acknowledges evaluation is incomplete (the abstract cuts off mid-sentence describing scoring), leaving unclear how the authors handle legitimate semantic equivalence in conclusions or whether they distinguish between reasoning failures vs. minor phrasing differences. Finally, this dataset does not address whether LLMs actually perform new scientific reasoning or instead memorize common conclusion patterns from their pretraining on PubMed—distinguishing genuine reasoning from template-matching remains an open problem.
Research Context
This work builds on the growing interest in evaluating LLMs on reasoning-heavy scientific tasks, following prior research on scientific question-answering (SQuAD, HotpotQA) but specifically targeting the conclusion generation task, which hasn't been systematically benchmarked at scale before. It contributes to the broader effort to create domain-specific evaluation datasets—similar in motivation to PubMedQA and BioASQ for biomedical NLP, but focusing on a specific reasoning capability (evidence synthesis) rather than retrieval or QA. The inclusion of journal-level metadata opens a research direction on how LLM reasoning varies with domain specialization and research quality, which hadn't been explored in prior conclusion-generation work. This positions MedConclusion as a new standard for evaluating scientific reasoning in LLMs and could inform future work on improving model calibration (knowing when to abstain) and domain-aware fine-tuning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::