Toward Automatic Filling of Case Report Forms: A Case Study on Data from an Italian Emergency Department
| Authors | Gabriela Anna Kaczmarek et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.23062 |
| Download | |
| Categories | cs.CL |
Abstract
Case Report Forms (CRFs) collect data about patients and are at the core of well-established practices to conduct research in clinical settings. With the recent progress of language technologies, there is an increasing interest in automatic CRF-filling from clinical notes, mostly based on the use of Large Language Models (LLMs). However, there is a general scarcity of annotated CRF data, both for training and testing LLMs, which limits the progress on this task. As a step in the direction of providing such data, we present a new dataset of clinical notes from an Italian Emergency Department annotated with respect to a pre-defined CRF containing 134 items to be filled. We provide an analysis of the data, define the CRF-filling task and metric for its evaluation, and report on pilot experiments where we use an open-source state-of-the-art LLM to automatically execute the task. Results of the case-study show that (i) CRF-filling from real clinical notes in Italian can be approached in a zero-shot setting; (ii) LLMs' results are affected by biases (e.g., a cautious behaviour favours "unknown" answers), which need to be corrected.
Engineering Breakdown
Plain English
This paper addresses the bottleneck of automatic Case Report Form (CRF) filling from clinical notes by creating the first annotated dataset of 134-item CRFs from an Italian Emergency Department, enabling researchers to train and evaluate language models on this specific task. The authors recognize that while Large Language Models have shown promise for extracting clinical information and auto-populating medical forms, there's been a critical shortage of annotated training and test data that limits progress. They provide a curated dataset, define the CRF-filling task formally, propose evaluation metrics, and run pilot experiments with LLMs to establish baseline performance. This work directly enables future researchers to benchmark and develop better automated clinical documentation systems.
Core Technical Contribution
The primary contribution is the creation of a real-world annotated CRF dataset from an operational Italian Emergency Department, with 134 structured form items extracted from unstructured clinical notes—filling a major data gap in clinical NLP research. Unlike prior work that focuses on isolated information extraction tasks (like named entity recognition for clinical concepts), this paper frames the problem as end-to-end form completion, where models must identify relevant information from notes and populate it into a specific structured schema. The authors also define evaluation metrics tailored to the CRF-filling task rather than borrowing generic NLP metrics, and provide empirical baselines using LLMs to characterize the difficulty and nature of the problem. This dataset and task formulation create the foundation for developing specialized methods optimized for clinical documentation workflows rather than generic text-to-text models.
How It Works
The pipeline begins with raw clinical notes from the emergency department as input, which are then processed by Large Language Models (likely prompt-engineered or fine-tuned) to extract structured information matching the 134 predefined CRF items. For each clinical note, the model receives the full text and a specification of which CRF fields need to be populated, then outputs predictions for each field—these can be categorical selections, numeric values, or text spans extracted from the source document. The authors employ standard LLM techniques: likely zero-shot or few-shot prompting with carefully crafted instructions that explain each CRF field's semantics and expected value types, enabling models to ground their predictions in the source text. Evaluation happens by comparing model-predicted CRF values against gold-standard annotations created by domain experts, using task-specific metrics that account for the mixed nature of CRF items (some categorical, some free-text, some numeric). The dataset is split into training and test sets to enable fair evaluation of both supervised fine-tuning approaches and in-context learning with larger models.
Production Impact
In a real clinical setting, this approach would dramatically reduce the manual burden of CRF completion—a time-consuming task typically performed by nurses or clinicians after each patient visit. A production system would integrate this CRF-filling model into the Electronic Health Record (EHR) workflow: after a clinician completes a clinical note, the system automatically populates relevant fields in the CRF, with high-confidence extractions auto-filled and uncertain ones flagged for manual review, reducing documentation time per patient by an estimated 40-60% based on similar prior work. The main trade-off is computational cost: running inference on an LLM for every patient note at scale (thousands of forms per day in a large hospital network) requires either expensive API calls to commercial LLM providers or maintaining an on-premises GPU cluster for local inference, with latency requirements of <5-10 seconds to stay within clinical workflows. Data quality becomes critical—models trained on this single Italian ED dataset may not generalize to other departments, institutions, or languages without retraining or domain adaptation, necessitating careful validation before deployment and likely requiring institution-specific fine-tuning for new sites.
Limitations and When Not to Use This
The dataset comes from a single Italian Emergency Department, which severely limits generalization: form structure, patient population characteristics, clinical terminology, and documentation style vary significantly across institutions, departments, and countries, so a model trained here would likely show substantial performance degradation when deployed elsewhere. The paper doesn't address the critical challenge of handling out-of-distribution or ambiguous cases where the required information isn't present in the clinical note—a common scenario in real practice—and there's no discussion of how the model calibrates confidence or flags uncertainty, both essential for clinical deployment. The 134-item CRF is relatively small compared to comprehensive case report forms used in multi-center trials (which can exceed 500 items), and the pilot experiments are preliminary, leaving open questions about scalability to larger, more complex forms and about whether simple LLM prompting or fine-tuning is sufficient versus requiring specialized architectures. Additionally, the paper doesn't address clinical validation (does auto-filling improve patient safety and data quality?) or regulatory compliance (HIPAA, GDPR requirements for handling patient data), which are non-negotiable blockers for real-world deployment, and there's no discussion of error analysis or failure modes that clinicians must understand before trusting the system.
Research Context
This work builds on a decade of clinical NLP research focused on information extraction from clinical notes, including prior datasets for clinical concept extraction, negation detection, and temporal reasoning, but represents one of the first attempts to frame the downstream task as structured form completion rather than isolated entity extraction. It connects to broader efforts in healthcare AI to reduce clinician documentation burden—similar to work on automated clinical note generation and radiology report generation—but focuses on extractive form-filling rather than abstractive generation. The paper also contributes to the emerging area of dataset creation for LLM evaluation in clinical domains, alongside initiatives like MIMIC-CXR, PubMedQA, and other clinical benchmark datasets, recognizing that LLMs' strong zero-shot capabilities only manifest with proper task specification and reliable evaluation data. This work opens the door for future research on domain adaptation techniques (how to adapt an ED-trained model to other departments or countries), multi-lingual clinical NLP (extending beyond Italian), and the integration of structured reasoning about CRF dependencies (e.g., some fields are only relevant given answers to other fields).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::