An Agentic Approach to Generating XAI-Narratives
| Authors | Yifan He & David Martens |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.20003 |
| Download | |
| Categories | cs.CL |
Abstract
Explainable AI (XAI) research has experienced substantial growth in recent years. Existing XAI methods, however, have been criticized for being technical and expert-oriented, motivating the development of more interpretable and accessible explanations. In response, large language model (LLM)-generated XAI narratives have been proposed as a promising approach for translating post-hoc explanations into more accessible, natural-language explanations. In this work, we propose a multi-agent framework for XAI narrative generation and refinement. The framework comprises the Narrator, which generates and revises narratives based on feedback from multiple Critic Agents on faithfulness and coherence metrics, thereby enabling narrative improvement through iteration. We design five agentic systems (Basic Design, Critic Design, Critic-Rule Design, Coherent Design, and Coherent-Rule Design) and systematically evaluate their effectiveness across five LLMs on five tabular datasets. Results validate that the Basic Design, the Critic Design, and the Critic-Rule Design are effective in improving the faithfulness of narratives across all LLMs. Claude-4.5-Sonnet on Basic Design performs best, reducing the number of unfaithful narratives by 90% after three rounds of iteration. To address recurrent issues, we further introduce an ensemble strategy based on majority voting. This approach consistently enhances performance for four LLMs, except for DeepSeek-V3.2-Exp. These findings highlight the potential of agentic systems to produce faithful and coherent XAI narratives.
Engineering Breakdown
Plain English
This paper addresses a critical gap in XAI research: while existing explanation methods are technically sound, they're often too complex for non-experts to understand. The authors propose a multi-agent LLM framework where a Narrator agent generates natural-language explanations of AI model decisions, while multiple Critic Agents iteratively refine these narratives by evaluating faithfulness (accuracy to the original model) and coherence (readability and clarity). The framework uses feedback loops between the Narrator and Critics to progressively improve explanations, making black-box model predictions more interpretable for general audiences. This work bridges the gap between technical XAI methods and human-understandable communication.
Core Technical Contribution
The core novelty is applying multi-agent reasoning patterns to the XAI explanation problem, where specialized agents play distinct roles (generation vs. critique) to iteratively improve narrative quality. Rather than treating explanation as a one-shot text generation task, the authors model it as a collaborative refinement process with explicit evaluation criteria—faithfulness and coherence—that agents use to provide structured feedback. This agent-based architecture allows for modular evaluation of explanation quality and enables the system to balance competing objectives (technical accuracy vs. readability) through dialogue. The paper designs five distinct agentic systems, suggesting different configurations of agent roles and feedback mechanisms to optimize explanation generation.
How It Works
The system works as a two-phase iterative loop: first, the Narrator agent takes a post-hoc explanation (e.g., feature importance scores, attention weights, or decision trees) and generates a natural-language narrative describing what the model did and why. In the second phase, multiple Critic Agents independently evaluate the generated narrative—one assesses faithfulness by checking whether the narrative accurately represents the underlying explanation, while others evaluate coherence, readability, and alignment with the original decision. The Narrator receives aggregated feedback from all Critics and uses it to revise the narrative, addressing specific failure points identified by the critics. This loop repeats for multiple iterations until convergence or a stopping criterion is met. The framework is flexible enough to accommodate different types of post-hoc explanations (LIME, SHAP, attention maps) and can be instantiated with different LLM backbones and critic configurations.
Production Impact
For production systems, this approach solves a real UX problem: model cards and SHAP dashboards are often unusable by end-users who lack ML expertise. By generating polished natural-language explanations, teams can provide transparency to regulators, customers, and stakeholders without requiring them to interpret technical artifacts. Integration into a production pipeline would involve: (1) wrapping your existing XAI method (LIME/SHAP) to produce feature importance outputs, (2) feeding these into the agentic system to generate narratives, (3) caching generated explanations to avoid recomputation. The latency cost is moderate—LLM inference for Narrator plus multiple Critic evaluations typically adds 2-5 seconds per explanation. The main trade-off is increased API costs (multiple LLM calls per explanation) versus improved user trust and regulatory compliance. This is particularly valuable in high-stakes domains like healthcare, finance, and hiring where unexplainable decisions face legal and ethical scrutiny.
Limitations and When Not to Use This
The paper's approach depends entirely on the faithfulness of the underlying post-hoc explanation method—if LIME or SHAP is wrong, the narrative will elegantly communicate an incorrect explanation, potentially masking the problem. There's also a bootstrapping risk: the Critic agents are LLMs trained on text, so they may favor narrative smoothness over actual technical accuracy if those objectives conflict. The framework assumes that iterative refinement converges to a good explanation, but doesn't address cases where no faithful natural-language summary exists—some model decisions may be intrinsically high-dimensional and resist simple storytelling. Finally, the paper doesn't thoroughly evaluate how well domain experts validate these explanations against ground truth, or how end-users actually behave when given these narratives versus raw technical explanations.
Research Context
This work builds on the recent trend of using LLMs for explanation tasks (e.g., rationale generation, question answering explanations) but introduces a novel agent-based refinement loop. It's motivated by critiques of first-generation XAI methods (LIME, SHAP, attention) for being inaccessible to practitioners and end-users—a problem highlighted by the XAI interpretability literature and human-AI interaction research. The multi-agent framework draws from recent work on using LLM agents for complex reasoning (ReAct, Chain-of-Thought agents) and applies these patterns to the explanation generation domain. This opens research directions in: automated evaluation of explanation quality, trade-offs between faithfulness and simplicity, and user studies on how different narrative styles affect trust and decision-making.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::