Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
| Authors | Delip Rao et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.03173 |
| Download | |
| Categories | cs.CL |
Abstract
Large language models and deep research agents supply citation URLs to support their claims, yet the reliability of these citations has not been systematically measured. We address six research questions about citation URL validity using 10 models and agents on DRBench (53,090 URLs) and 3 models on ExpertQA (168,021 URLs across 32 academic fields). We find that 3--13% of citation URLs are hallucinated -- they have no record in the Wayback Machine and likely never existed -- while 5--18% are non-resolving overall. Deep research agents generate substantially more citations per query than search-augmented LLMs but hallucinate URLs at higher rates. Domain effects are pronounced: non-resolving rates range from 5.4% (Business) to 11.4% (Theology), with per-model effects even larger. Decomposing failures reveals that some models fabricate every non-resolving URL, while others show substantial link-rot fractions indicating genuine retrieval. As a solution, we release urlhealth, an open-source tool for URL liveness checking and stale-vs-hallucinated classification using the Wayback Machine. In agentic self-correction experiments, models equipped with urlhealth reduce non-resolving citation URLs by 6\textrm{--}79\times to under 1%, though effectiveness depends on the model's tool-use competence. The tool and all data are publicly available. Our characterization findings, failure taxonomy, and open-source tooling establish that citation URL validity is both measurable at scale and correctable in practice.
Engineering Breakdown
Plain English
This paper systematically measures how often large language models and AI research agents produce fake citation URLs when answering questions. The authors tested 10 models and agents on two benchmarks containing over 221,000 URLs total, finding that 3-13% of citations are completely hallucinated (never existed, no Wayback Machine record) while 5-18% don't resolve at all. Deep research agents that can search the web generate more citations but hallucinate URLs at higher rates than simpler search-augmented LLMs. The problem varies significantly by academic domain, ranging from 5.4% non-resolving rates in Business to 11.4% in Theology, exposing a critical reliability issue in AI-generated research.
Core Technical Contribution
The core contribution is the first systematic measurement framework for citation hallucination in LLMs and research agents, going beyond anecdotal observations. The authors developed a methodology using the Wayback Machine as a ground truth source to distinguish between three categories: hallucinated URLs (never existed), non-resolving URLs (may have existed but are dead), and valid citations. They created or adapted two evaluation benchmarks—DRBench and ExpertQA spanning 32 academic fields—that enable comparison across 13 different models and agents. This work shifts citation quality from a vague concern to a measurable, quantifiable failure mode with domain-specific patterns.
How It Works
The evaluation pipeline takes query-answer pairs from benchmark datasets and extracts URLs that each model claims support its answer. For each extracted URL, the system queries the Wayback Machine's public API to check if any historical snapshot exists of that URL. URLs with no Wayback snapshots are classified as hallucinated (likely never existed), while URLs that existed historically but don't currently resolve are classified as non-resolving. The authors then aggregate results across multiple dimensions: per-model (comparing different LLMs and agents), per-domain (business, theology, medicine, etc.), and per-query-type (comparing search-augmented vs. deep research agents). This produces coverage matrices showing hallucination and non-resolution rates as functions of model architecture and domain, enabling engineers to understand where the reliability problems concentrate.
Production Impact
For teams building research assistants, chatbots, or knowledge-intensive applications, this work reveals that blindly trusting model-generated citations creates liability and user trust issues. Production systems now have a measurable benchmark—expect 3-13% citation hallucination—to set guardrails: you might require human review for citations in high-stakes domains like medicine or law, or require live URL validation before returning responses. The finding that deep research agents hallucinate at higher rates (despite generating more citations) suggests a trade-off: more comprehensive answers come with more garbage citations, meaning you need stronger post-processing. Teams should implement citation validation layers using Wayback Machine APIs or direct HEAD requests, add confidence scoring to model outputs based on domain history, and potentially fine-tune models on datasets with verified citations rather than raw web data. The 5-18% non-resolution range implies you should cache cited URLs or maintain a verified knowledge base rather than relying on the open web.
Limitations and When Not to Use This
The paper relies on the Wayback Machine as ground truth, which has incomplete coverage—some legitimate URLs may not be archived, leading to false hallucination classification. The benchmarks (DRBench and ExpertQA) are static snapshots of question-answer pairs from specific time periods, so hallucination rates may shift as the web changes and new URLs die or are created. The work doesn't deeply analyze why certain domains (Theology at 11.4% vs. Business at 5.4%) have worse hallucination rates—whether it's training data quality, domain obscurity, or something structural about how models reason in those fields. The paper doesn't provide guidance on fixing the problem at the source (better training, RLHF with citation grounding, etc.), only measuring and characterizing the symptom, leaving open how to build models that generate fewer hallucinated citations in the first place.
Research Context
This paper builds on recent concerns about hallucination and grounding in LLMs, extending beyond the well-studied phenomenon of factual hallucination to the specific case of citation integrity. It complements work on retrieval-augmented generation (RAG) and fact-checking by showing that even models with access to search still produce broken citations at scale. The benchmarks extend prior evaluation efforts like FEVER and HotpotQA by focusing specifically on URLs and adding domain stratification across 32 academic fields. This work opens research directions in citation grounding, potentially spurring work on RLHF methods that reward verifiable citations, retrieval augmentation strategies that prioritize archival stability, or architectural changes that make models more cautious about citing specific URLs.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::