Measuring research data reuse in scholarly publications using generative artificial intelligence: Open Science Indicator development and preliminary results
| Authors | Lauren Cadwallader et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2604.28061 |
| Download | |
| Categories | cs.DL, cs.CL |
Abstract
Numerous metascience studies and other initiatives have begun to monitor the prevalence of open science practices when it is more important to understand the 'downstream' effects or impacts of open science. PLOS and DataSeer have developed a new LLM-based indicator to measure an important effect of open science: the reuse of research data. Our results show a data reuse rate of 43%, which is higher than established bibliometric techniques. We show that data reuse can be measured at scale using LLMs and generative artificial intelligence. The positive effects of research data sharing and reuse may currently be underestimated.
Engineering Breakdown
Plain English
This paper presents a new LLM-based system developed by PLOS and DataSeer to measure how often research data gets reused after initial publication—a key indicator of open science impact. Previous methods using bibliometric techniques underestimated data reuse; this approach found a 43% reuse rate by leveraging large language models to detect citations and references to datasets at scale. Rather than just tracking whether data is shared, the authors tackled the harder problem of measuring actual downstream reuse, which gives a more complete picture of open science's real-world value. This work demonstrates that generative AI can automate metascience analysis that was previously manual and limited to smaller datasets.
Core Technical Contribution
The core innovation is a scalable LLM-based detection pipeline that automatically identifies instances of data reuse across the published literature without manual annotation. Unlike prior bibliometric approaches that relied on explicit citations or metadata, this system uses language understanding to infer when a dataset has been reused, even when the connection is implicit or paraphrased. The authors show this technique outperforms existing methods (43% vs. lower rates from traditional approaches), suggesting that LLMs capture semantic relationships that numerical citation counts miss. This is the first demonstration that generative AI can reliably measure data reuse at scale across large research corpora.
How It Works
The system takes as input published research papers (likely extracted abstracts or full texts) and metadata about released datasets from the PLOS ecosystem. The LLM processes each paper to identify mentions, references, or usage patterns of datasets—going beyond simple string matching to understand contextual reuse even when dataset names are modified or discussed indirectly. For each detected reuse instance, the model outputs a reuse event with metadata (source paper, target dataset, confidence score). The pipeline aggregates these signals across thousands of papers to compute prevalence statistics, with the LLM serving as a semantic matcher rather than a simple keyword search. The key advantage is that language understanding captures implicit reuse (e.g., 'we applied the methodology from Smith et al.'s open dataset') that keyword-based tools would miss. Results are then surfaced as metrics showing 43% of papers reuse prior open data.
Production Impact
For organizations managing research data repositories (like PLOS, Zenodo, or institutional repositories), this approach enables automated measurement of dataset impact without expensive manual audits—turning data reuse tracking from a quarterly manual exercise into a continuous, real-time metric. Engineering teams building metascience platforms could integrate similar LLM-based detection to provide researchers instant feedback on their data's impact, improving incentive structures for open science. However, production adoption requires careful calibration: the model needs fine-tuning on domain-specific terminology (different fields use different conventions for citing datasets), and false positive rates matter—incorrectly flagging non-reuse as reuse skews policy decisions. Latency and cost are manageable if batched (process papers weekly rather than real-time), but scaling to millions of papers may require efficient inference techniques like quantization or distillation to reduce per-document compute cost.
Limitations and When Not to Use This
The paper doesn't discuss inter-annotator agreement or gold-standard validation, so it's unclear how often the LLM incorrectly identifies reuse versus true reuse—critical for downstream policy decisions. The 43% figure may include false positives (papers discussing a dataset without actually reusing it) or false negatives (reuse via proprietary APIs or closed-source reimplementations), and without error analysis, the true reuse rate remains uncertain. The approach also assumes all data is discoverable in text form; datasets used only through database queries, proprietary interfaces, or synthetic reproductions won't be detected. The paper doesn't address how the method generalizes across disciplines (life sciences vs. social sciences) or how sensitive results are to the choice of LLM backbone—different models likely have different precision/recall trade-offs.
Research Context
This work extends the metascience agenda established by studies tracking open science adoption rates, but shifts focus from inputs (is data shared?) to outputs (is shared data actually used?), filling a gap in impact measurement. It builds on prior work in automated citation detection and knowledge extraction from unstructured text, but applies those techniques to the novel task of measuring dataset reuse—a problem that prior bibliometric tools (which rely on explicit metadata) were poorly suited to solve. The paper aligns with broader efforts to quantify open science benefits (e.g., Zenodo impact reports, arXiv citation tracking) and demonstrates that LLMs enable new measurement capabilities at scale. This opens a research direction: using language models for retrospective impact analysis of other research outputs (code, benchmarks, methods) where reuse is often implicit and hard to detect through metadata alone.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::