Resources for Automated Evaluation of Assistive RAG Systems that Help Readers with News Trustworthiness Assessment
| Authors | Dake Zhang et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2602.24277 |
| Download | |
| Categories | cs.IR, cs.AI |
Abstract
Many readers today struggle to assess the trustworthiness of online news because reliable reporting coexists with misinformation. The TREC 2025 DRAGUN (Detection, Retrieval, and Augmented Generation for Understanding News) Track provided a venue for researchers to develop and evaluate assistive RAG systems that support readers' news trustworthiness assessment by producing reader-oriented, well-attributed reports. As the organizers of the DRAGUN track, we describe the resources that we have newly developed to allow for the reuse of the track's tasks. The track had two tasks: (Task 1) Question Generation, producing 10 ranked investigative questions; and (Task 2, the main task) Report Generation, producing a 250-word report grounded in the MS MARCO V2.1 Segmented Corpus. As part of the track's evaluation, we had TREC assessors create importance-weighted rubrics of questions with expected short answers for 30 different news articles. These rubrics represent the information that assessors believe is important for readers to assess an article's trustworthiness. The assessors then used their rubrics to manually judge the participating teams' submitted runs. To make these tasks and their rubrics reusable, we have created an automated process to judge runs not part of the original assessing. We show that our AutoJudge ranks existing runs well compared to the TREC human-assessed evaluation (Kendall's for Task 1 and for Task 2). These resources enable both the evaluation of RAG systems for assistive news trustworthiness assessment and, with the human evaluation as a benchmark, research on improving automated RAG evaluation.
Engineering Breakdown
Plain English
This paper describes the resources and evaluation framework developed for the TREC 2025 DRAGUN track, which focuses on building RAG systems that help readers assess news trustworthiness by generating well-sourced reports. The track consists of two tasks: generating 10 ranked investigative questions about a news article, and producing a 250-word grounded report using the MS MARCO V2.1 Segmented Corpus as the retrieval source. The authors released evaluation resources, datasets, and benchmarks to enable reproducible research on assistive systems that combat misinformation by providing readers with transparent, attribution-backed analysis. This addresses a real problem—many people lack tools to distinguish reliable reporting from false information—by creating standardized evaluation infrastructure that multiple teams can use to develop and compare RAG-based solutions.
Core Technical Contribution
The core contribution is the DRAGUN track itself and its associated evaluation resources—a structured benchmark and evaluation methodology for assessing assistive RAG systems designed specifically for news trustworthiness assessment. Unlike general-purpose RAG evaluation, this work introduces task-specific metrics and resources tailored to the unique demands of helping readers evaluate credibility: the need for investigative questions that surface relevant verification angles, and the requirement that generated reports be grounded in retrieved evidence with explicit attribution. The paper provides newly developed evaluation resources (datasets, ground truth annotations, evaluation protocols) that enable the research community to rigorously compare different approaches to building trustworthy, reader-oriented systems. This represents a shift from evaluating RAG systems solely on retrieval or generation quality to evaluating their effectiveness as assistive tools for a specific high-stakes application.
How It Works
The DRAGUN track operates as a two-stage pipeline. In Task 1, systems receive a news article and must generate 10 ranked investigative questions—queries that a reader might ask to verify or understand the article's claims (e.g., 'Who verified these statistics?' or 'What is the source's track record?'). These questions serve as guidance for the retrieval stage. In Task 2, the main task, systems must retrieve relevant evidence from the MS MARCO V2.1 Segmented Corpus and synthesize it into a 250-word report that addresses the article's key claims while explicitly attributing information to specific sources. The evaluation framework measures both the quality of question generation (relevance, comprehensiveness) and report generation (factuality, attribution accuracy, usefulness to readers assessing trustworthiness). Systems are evaluated against human-annotated ground truth, with metrics that reward well-grounded, transparent reasoning and penalize unsourced or contradictory claims. The pipeline treats question generation as a scaffolding mechanism that helps focus retrieval and report generation on the verification-relevant aspects of the news article.
Production Impact
For engineers building fact-checking or misinformation-detection systems, this work provides a concrete evaluation framework and dataset for validating that your RAG pipeline actually helps users make trustworthiness judgments—not just that it retrieves documents or generates fluent text. In production, you would adopt this approach by: (1) structuring your retrieval pipeline to support multi-hop reasoning (using generated questions to refine searches), (2) implementing explicit attribution tracking so every claim in your output is traceable to a source, and (3) evaluating your system against reader-oriented metrics rather than standard IR metrics. The main trade-off is latency and computational cost: generating ranked questions before retrieving and synthesizing reports adds a second forward-pass through your LLM, potentially doubling inference time per article. Integration complexity is moderate—you need to prepare your retrieval corpus in the MS MARCO segmented format and set up evaluation harnesses that check attribution accuracy and factual consistency, but the core LLM and retrieval components are standard. This approach is most impactful for high-stakes applications (news readers, policy analysts, fact-checkers) where transparency and grounding matter more than speed.
Limitations and When Not to Use This
This work does not address how to detect or handle adversarially manipulated or synthetic content in the retrieval corpus itself—if the source documents contain deliberate falsehoods, the RAG system will propagate them. The evaluation assumes access to a curated corpus (MS MARCO V2.1) with reasonable coverage of credible sources; in truly adversarial environments with rapidly evolving misinformation, coverage gaps could be substantial. The paper focuses on evaluation resources rather than novel architectural or algorithmic contributions, so it does not propose new techniques to improve RAG performance on this task—teams must still develop their own question generation and report synthesis models. Additionally, the 250-word report constraint and requirement for explicit attribution may not match all real-world applications (some users need longer deep dives; some systems operate in low-resource languages where large retrieval corpora don't exist), limiting generalizability. Finally, the work does not deeply explore how these systems perform on edge cases like satire, opinion pieces falsely presented as news, or coordinated misinformation campaigns where multiple sources reinforce the same false claim.
Research Context
This paper builds directly on the RAG literature, which has matured from basic retrieval-augmented generation to more structured, multi-stage pipelines (e.g., work on retrieval quality, prompt engineering for grounding). It also extends research on automated fact-checking and claim verification, adding the dimension of reader assistance—recognizing that systems should be designed not just to output facts but to help humans understand and evaluate credibility. The DRAGUN track sits within the broader TREC evaluation paradigm, which has a long history of creating community benchmarks (TREC QA, TREC Robust, etc.) to drive progress on shared problems. This work opens research directions in human-AI collaboration for misinformation resistance, in metrics that measure transparency and attribution quality, and in the construction of high-quality evaluation corpora for domain-specific RAG tasks. It also implicitly challenges the ML community to move beyond generic retrieval and generation benchmarks toward task-specific resources that capture real user needs.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::