BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation
| Authors | Delip Rao & Chris Callison-Burch |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2604.03159 |
| Download | |
| Categories | cs.DL, cs.CL |
Abstract
Large language models with web search are increasingly used in scientific publishing agents, yet they still produce BibTeX entries with pervasive field-level errors. Prior evaluations tested base models without search, which does not reflect current practice. We construct a benchmark of 931 papers across four scientific domains and three citation tiers -- popular, low-citation, and recent post-cutoff -- designed to disentangle parametric memory from search dependence, with version-aware ground truth accounting for multiple citable versions of the same paper. Three search-enabled frontier models (GPT-5, Claude Sonnet-4.6, Gemini-3 Flash) generate BibTeX entries scored on nine fields and a six-way error taxonomy, producing ~23,000 field-level observations. Overall accuracy is 83.6%, but only 50.9% of entries are fully correct; accuracy drops 27.7pp from popular to recent papers, revealing heavy reliance on parametric memory even when search is available. Field-error co-occurrence analysis identifies two failure modes: wholesale entry substitution (identity fields fail together) and isolated field error. We evaluate clibib, an open-source tool for deterministic BibTeX retrieval from the Zotero Translation Server with CrossRef fallback, as a mitigation mechanism. In a two-stage integration where baseline entries are revised against authoritative records, accuracy rises +8.0pp to 91.5%, fully correct entries rise from 50.9% to 78.3%, and regression rate is only 0.8%. An ablation comparing single-stage and two-stage integration shows that separating search from revision yields larger gains and lower regression (0.8% vs. 4.8%), demonstrating that integration architecture matters independently of model capability. We release the benchmark, error taxonomy, and clibib tool to support evaluation and mitigation of citation hallucinations in LLM-based scientific writing.
Engineering Breakdown
Plain English
This paper evaluates how well frontier large language models (GPT-5, Claude Sonnet-4.6, Gemini-3 Flash) generate accurate BibTeX citations when augmented with web search capabilities. The authors built a benchmark of 931 papers across four scientific domains and three citation tiers (popular, low-citation, and recent post-cutoff papers) to test whether models rely on parametric memory versus live search results. They generated ~23,000 field-level BibTeX entries scored across nine fields using a six-way error taxonomy, finding 83.6% overall accuracy but revealing significant performance degradation in specific scenarios like recent papers and low-citation works.
Core Technical Contribution
The core contribution is a domain-specific citation generation benchmark that isolates the gap between search-augmented and base model performance—a distinction prior work missed by only evaluating models without search. The authors introduced version-aware ground truth that accounts for multiple citable versions of the same paper, a practical problem overlooked in prior citation benchmarks. They designed the benchmark with deliberate citation tier stratification (popular, low-citation, recent post-cutoff) to disentangle when models succeed via parametric knowledge versus when they must rely on live search results. This reveals brittleness in frontier models that appears robust on standard evaluations but breaks down systematically on low-visibility and time-sensitive references.
How It Works
The evaluation pipeline works as follows: (1) The authors curate a diverse corpus of 931 papers from four scientific domains, then stratify them into three citation tiers (popular papers with many citations, low-citation papers with few references, and recent papers published after model cutoff dates). (2) For each paper, they define version-aware ground truth—recognizing that papers may have multiple citable versions (arXiv, published, revised editions) rather than a single canonical BibTeX entry. (3) They prompt three search-enabled frontier models to generate BibTeX entries for these papers, allowing the models to perform web search as needed. (4) The generated entries are scored field-by-field across nine BibTeX fields (author, title, year, journal, etc.) using a six-way error taxonomy (missing field, incorrect field, format error, etc.). (5) They aggregate ~23,000 field-level observations to compute accuracy per field, per domain, and per citation tier, revealing performance cliffs where search fails to retrieve relevant versions or models hallucinate missing metadata.
Production Impact
For teams building scientific publishing agents or literature management systems, this work reveals a critical gap: frontier models appear capable (83.6% accuracy) but fail catastrophically on recent papers and obscure citations—precisely the papers humans need help with. If you're deploying search-augmented LLMs for BibTeX generation, you cannot rely on base model evaluations; you must test on low-citation and time-sensitive papers explicitly, as they degrade substantially. The six-way error taxonomy provides a concrete framework for building error correction pipelines—you can now distinguish between hallucinated authors (require human review) versus minor formatting errors (auto-fixable). In production, this suggests a hybrid approach: use search-enabled models for initial generation, then apply domain-specific validation rules for recent/obscure papers and fallback to user review for citations the model flags as uncertain. The version-aware ground truth requirement adds complexity to your gold standard pipeline but is essential to avoid penalizing correct citations in non-canonical formats.
Limitations and When Not to Use This
The paper evaluates only three frontier models (GPT-5, Claude Sonnet-4.6, Gemini-3 Flash) and does not compare against smaller or specialized citation models, limiting generalizability to the broader LLM landscape. The benchmark spans only four scientific domains; results may not transfer to fields with different citation conventions (medicine, law, engineering) or to multilingual citations. The paper does not analyze why models fail—whether search retrieval is the bottleneck, whether models struggle to parse search results, or whether parametric knowledge conflicts with search results—limiting the actionability of findings for improving future systems. Critically, the evaluation assumes a single correct BibTeX entry per paper, but in practice different citation styles and versions may all be valid, potentially understating model performance on papers with legitimate variant citations.
Research Context
This work extends a long line of research on factuality and grounding in LLMs, building directly on prior work showing that search augmentation improves but does not solve hallucination. It responds to the gap that existing citation benchmarks (like CitationMatch or NIST TREC-based evaluations) tested base models without search, missing how practitioners actually deploy these systems. The benchmark design parallels recent work on time-aware evaluation (papers published after cutoff) and low-resource retrieval (low-citation papers), extending those concepts specifically to the citation generation domain. This opens a new research direction: understanding when and why search augmentation fails for scientific metadata, which could drive improvements in retrieval-augmented generation (RAG) systems, better handling of version canonicalization, and domain-specific fine-tuning for academic knowledge.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::