Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation
| Authors | Nazanin Jafari et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.03141 |
| Download | |
| Categories | cs.CL |
Abstract
Evaluating the factuality of long-form output generated by large language models (LLMs) remains challenging, particularly when responses are open-ended and contain many fine-grained factual statements. Existing evaluation methods primarily focus on precision: they decompose a response into atomic claims and verify each claim against external knowledge sources such as Wikipedia. However, this overlooks an equally important dimension of factuality: recall, whether the generated response covers the relevant facts that should be included. We propose a comprehensive factuality evaluation framework that jointly measures precision and recall. Our method leverages external knowledge sources to construct reference facts and determine whether they are captured in generated text. We further introduce an importance-aware weighting scheme based on relevance and salience. Our analysis reveals that current LLMs perform substantially better on precision than on recall, suggesting that factual incompleteness remains a major limitation of long-form generation and that models are generally better at covering highly important facts than the full set of relevant facts.
Engineering Breakdown
Plain English
This paper addresses a fundamental gap in how we evaluate whether LLMs generate factually accurate long-form text. Current methods only check precision—whether stated facts are true—but ignore recall—whether the response includes all relevant facts that should be mentioned. The authors propose a comprehensive evaluation framework that measures both dimensions by extracting reference facts from external knowledge sources like Wikipedia and checking whether the generated text covers those facts. This dual-metric approach gives a much more complete picture of factuality than existing single-metric methods.
Core Technical Contribution
The key innovation is reframing factuality evaluation as a recall problem alongside precision, rather than treating it as a one-directional verification task. The authors introduce a systematic method to construct reference fact sets from external knowledge sources and then measure whether generated responses cover these facts—essentially asking 'what important information was left out?' rather than only 'what was said that was wrong?'. This joint precision-recall framework provides a more balanced assessment of factuality and catches a class of errors (omissions) that decomposition-only methods miss. The technical approach leverages structured knowledge graphs or Wikipedia to establish ground truth facts that should appear in a well-formed response.
How It Works
The system operates in two main phases. First, it constructs a reference fact set by querying external knowledge sources (Wikipedia, knowledge graphs) relevant to the topic or entity mentioned in the prompt, extracting atomic factual statements that represent 'what should be said'. Second, it performs bidirectional verification: (1) precision checks—parsing the LLM output into atomic claims and verifying each against the knowledge source, and (2) recall checks—comparing the reference facts against the generated text to identify which facts were omitted or inadequately covered. The framework produces two scores: a precision score indicating the proportion of stated facts that are correct, and a recall score indicating what fraction of the reference fact set appears in the generated response. Together these create an F1-like harmonic mean metric that penalizes responses that are either hallucinated or incomplete.
Production Impact
For engineers building LLM evaluation pipelines, this framework dramatically improves quality assessment by catching both hallucinations and critical omissions in a single evaluation pass. In a production RAG system or customer-facing Q&A bot, you could now automatically flag responses that are technically accurate but dangerously incomplete—for example, a medical chatbot that correctly states one drug interaction but omits three others. Implementation requires integrating structured knowledge sources (Wikipedia dumps, proprietary knowledge bases) and building claim extraction and matching pipelines, adding 50-200ms latency per evaluation depending on response length and knowledge source size. The trade-off is that you need curated reference knowledge for each domain and topic, which may require pre-computing fact sets for common queries, increasing system complexity but providing more reliable quality metrics than binary correctness checks.
Limitations and When Not to Use This
The framework assumes high-quality external knowledge sources are available and comprehensive for the domain—it will miss facts that don't appear in Wikipedia or your knowledge graph, potentially underestimating what should be known. Measuring recall against a fixed reference set also assumes there is a single canonical set of 'facts that must be included,' which breaks down for subjective or nuanced topics where reasonable responses differ on emphasis and scope. The paper doesn't fully address how to handle factual statements that are technically true but outdated or superseded (e.g., 'X was the record holder' when Y now holds it), which is common in long-form generation. Additionally, the computational cost of constructing reference fact sets at scale across diverse topics and the brittleness of claim matching (small paraphrases can break extraction pipelines) limit practical deployment to well-scoped domains.
Research Context
This work builds directly on the growing literature of LLM hallucination detection and fact verification, extending beyond decomposition-based methods like those using question-answering or NLI scoring. It responds to limitations in prior work like FEVER and ALIGN that focus on precision but largely ignore coverage, and aligns with broader concerns in the field about incomplete or evasive responses from LLMs. The paper contributes to the evaluation methodology track that includes BLEU, BERTScore, and other multi-dimensional metrics, pushing the community toward more comprehensive factuality assessment. This opens research directions in automated reference fact construction, scalable claim matching across domains, and integrating factuality evaluation into RLHF training loops to directly optimize for both precision and recall during model training.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::