MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations
| Authors | Sara Rosenthal et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.23184 |
| Download | |
| Categories | cs.CL |
Abstract
We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models. We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora. Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses. Our benchmark is available at https://github.com/IBM/mt-rag-benchmark
Engineering Breakdown
Plain English
MTRAG-UN is a benchmark dataset designed to expose weaknesses in multi-turn retrieval-augmented generation (RAG) systems, which are LLMs that fetch external documents to answer questions. The authors released 666 tasks spanning 2,800+ conversation turns across 6 domains, systematically testing how current retrieval and generation models handle unanswerable questions, underspecified queries, questions that don't stand alone without context, and unclear responses. Their experiments demonstrate that state-of-the-art retrieval and generation models struggle significantly on these realistic conversational challenges, with the benchmark providing a concrete evaluation framework to identify and measure these failure modes.
Core Technical Contribution
The core novelty is the systematic categorization and evaluation of failure modes in multi-turn RAG conversations that weren't comprehensively studied before. Rather than building a new algorithm, the authors engineered a benchmark specifically targeting four categories of conversational challenges (UNanswerable, UNderspecified, NONstandalone, and UNclear responses) that arise when RAG systems must maintain context and resolve references across multiple dialogue turns. This differs from prior single-turn RAG benchmarks because it captures the compositional complexity of real conversations where questions depend on prior context and retrievable information may not exist. The benchmark design itself—with 666 carefully curated tasks and accompanying domain-specific corpora—is the intellectual contribution, serving as a diagnostic tool to expose gaps in current models.
How It Works
The benchmark operates as a structured evaluation dataset where each conversation consists of multiple turns, with each turn containing a user query, ground-truth relevant documents, and reference answers. For each turn, the system (a retrieval model + generation model pipeline) must first retrieve relevant passages from a domain-specific corpus, then generate a response based on those passages and the conversation history. The benchmark measures performance across four specific challenge categories: questions with no valid answer in the corpus (UNanswerable), questions missing critical specificity (UNderspecified), questions requiring context from earlier turns to interpret (NONstandalone), and model responses that don't accurately convey information (UNclear). Evaluation metrics track whether systems can detect unanswerable cases, resolve underspecified references, propagate context from prior turns, and generate coherent multi-turn responses, providing fine-grained diagnostics beyond overall accuracy metrics.
Production Impact
For teams building production RAG systems, this benchmark provides concrete test coverage for failure modes that single-turn evaluations miss entirely. In a real conversational AI pipeline, many user queries depend on prior context (e.g., "What about the CEO?" requires understanding a previous mention), and MTRAG-UN helps quantify how your retrieval and generation models degrade on such inputs. This means you can now measure whether your system correctly refuses to answer when information isn't available, versus hallucinating or returning irrelevant passages. The practical integration requires updating your evaluation harness to include multi-turn conversations with explicit failure-mode labels, adding latency (since you're tracking conversation history), and potentially retraining retrieval models to recognize when queries are underspecified or context-dependent. The trade-off is that comprehensive coverage requires 666 labeled tasks and corpus maintenance, but the diagnostic granularity helps you prioritize which model improvements (retrieval vs. generation vs. context handling) will yield the largest production gains.
Limitations and When Not to Use This
The benchmark assumes that all domain-specific corpora are static and fully indexed—it doesn't address real-world scenarios where documents are added dynamically or when the corpus itself is incomplete before a conversation starts. The evaluation relies on binary or discrete labels for 'correctness' (unanswerable, underspecified, etc.), which may not capture nuanced cases where partial answers are sometimes acceptable in production. The benchmark covers only 6 domains, which may not represent specialized verticals like medical, legal, or financial RAG where domain-specific terminology and context-dependency create different failure patterns. Additionally, the paper doesn't deeply explore how retriever and generator errors compound across multi-turn conversations, nor does it provide baseline results on modern LLMs (like GPT-4) that may have improved contextual understanding since the benchmark's creation. Finally, the benchmark doesn't address scaling challenges: it's unclear how evaluation costs grow as conversation length increases or how the framework generalizes to long-context scenarios.
Research Context
This work builds on the growing recognition that single-turn RAG evaluation (like Natural Questions or SQuAD) fails to capture real conversational complexity, where users refine queries, reference earlier statements, and expect systems to maintain coherence. MTRAG-UN is in the lineage of conversational QA benchmarks like QuAC and CoQA, but extends them specifically to the retrieval-augmentation setting where systems must both retrieve and generate in dialogue. The benchmark directly addresses limitations identified in prior work showing that RAG systems fail on context-dependent questions and don't properly acknowledge information gaps. This opens a research direction toward improving both retrievers (to handle underspecification and context propagation) and generators (to avoid false confidence on unanswerable questions), as well as exploring end-to-end training objectives that jointly optimize for conversational coherence and retrieval accuracy.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::