Using Large Language Models and Knowledge Graphs to Improve the Interpretability of Machine Learning Models in Manufacturing
| Authors | Thomas Bayer et al. |
| Year | 2026 |
| Field | AI / Agents |
| arXiv | 2604.16280 |
| Download | |
| Categories | cs.AI |
Abstract
Explaining Machine Learning (ML) results in a transparent and user-friendly manner remains a challenging task of Explainable Artificial Intelligence (XAI). In this paper, we present a method to enhance the interpretability of ML models by using a Knowledge Graph (KG). We store domain-specific data along with ML results and their corresponding explanations, establishing a structured connection between domain knowledge and ML insights. To make these insights accessible to users, we designed a selective retrieval method in which relevant triplets are extracted from the KG and processed by a Large Language Model (LLM) to generate user-friendly explanations of ML results. We evaluated our method in a manufacturing environment using the XAI Question Bank. Beyond standard questions, we introduce more complex, tailored questions that highlight the strengths of our approach. We evaluated 33 questions, analyzing responses using quantitative metrics such as accuracy and consistency, as well as qualitative ones such as clarity and usefulness. Our contribution is both theoretical and practical: from a theoretical perspective, we present a novel approach for effectively enabling LLMs to dynamically access a KG in order to improve the explainability of ML results. From a practical perspective, we provide empirical evidence showing that such explanations can be successfully applied in real-world manufacturing environments, supporting better decision-making in manufacturing processes.
Engineering Breakdown
Plain English
This paper addresses a critical problem in explainable AI: making machine learning model predictions understandable to non-technical users. The authors propose a method that combines Knowledge Graphs (structured repositories of domain facts) with Large Language Models to generate natural-language explanations of ML results. Their system stores domain-specific data, ML predictions, and explanations in a KG, then uses selective retrieval to pull relevant information and feed it to an LLM for human-readable explanation generation. They validated the approach in a manufacturing environment using the XAI Question Bank, demonstrating that structured knowledge combined with language models can produce more interpretable and trustworthy AI outputs.
Core Technical Contribution
The core innovation is a hybrid architecture that treats the Knowledge Graph as a structured retrieval layer sitting between ML models and LLMs for explanation generation. Rather than having LLMs generate explanations from raw model outputs alone, the system explicitly grounds explanations in domain knowledge by extracting relevant KG triplets, which are then passed to the LLM as context. This selective retrieval mechanism ensures that explanations reference authoritative domain facts and established connections, reducing hallucination and improving consistency. The approach is novel because it decouples explanation generation (delegated to the LLM's language capabilities) from explanation grounding (delegated to the KG's structured knowledge), creating a clear separation of concerns that neither component alone can achieve.
How It Works
The system operates in three distinct stages: (1) An ML model makes a prediction or classification decision on input data, which is stored alongside its result. (2) A selective retrieval mechanism queries the Knowledge Graph to find relevant triplets—structured facts like (entity, relation, entity) tuples—that relate to the input data, the prediction, or the domain context. (3) These triplet sets are formatted and passed as retrieval context to an LLM, which uses them to synthesize a natural-language explanation that a non-technical user can understand. The KG acts as a structured memory that prevents the LLM from inventing explanations; the LLM acts as a summarization and communication layer that converts structured facts into readable prose. The selective retrieval is the critical bottleneck—it must identify which KG facts are relevant to each prediction, likely using similarity matching, relation types, or entity overlap between the input and the graph.
Production Impact
In production manufacturing systems, this approach enables compliance and trust by providing auditable, fact-grounded explanations for model decisions (e.g., why a quality check failed or why a machine was flagged for preventive maintenance). Instead of deploying black-box models that produce unexplainable predictions, teams can now surface both the decision and its justification to operators and auditors. The main trade-off is operational complexity: you must maintain a high-quality, up-to-date Knowledge Graph alongside your ML pipeline, which requires domain experts to curate relationships and facts. Latency impact is moderate—KG retrieval (typically milliseconds for graph queries) plus LLM inference (seconds) means explanations appear with a few-second delay, acceptable for most manufacturing use cases but not for real-time control. Integration requires careful schema design (how to represent ML results and domain facts in the same graph) and tooling to keep the KG synchronized with model retraining and domain changes.
Limitations and When Not to Use This
The paper does not address how to construct or maintain a high-quality Knowledge Graph at scale, which is a substantial engineering burden—domain experts must define entities, relationships, and attributes, and keep them current as business logic evolves. The selective retrieval mechanism is not thoroughly detailed in the abstract, so it's unclear how the system handles cases where no relevant KG facts exist, or where multiple conflicting facts apply; poor retrieval leads to irrelevant or contradictory explanations. The evaluation is limited to a manufacturing environment with the XAI Question Bank, so generalization to other domains (healthcare, finance, etc.) with different explanation needs remains unproven. The approach assumes the KG is correct and trustworthy, but if the graph contains errors or outdated information, the LLM will faithfully propagate those errors to users, creating a false sense of confidence in flawed explanations.
Research Context
This work builds on the established XAI literature (LIME, SHAP) and the emerging paradigm of retrieval-augmented generation (RAG), which uses external knowledge bases to improve LLM outputs and reduce hallucination. It extends prior work in Knowledge Graph applications by showing that KGs can serve as a bridge between opaque ML models and explainability, rather than just as a knowledge source for downstream tasks. The use of the XAI Question Bank as an evaluation benchmark connects the work to formal XAI evaluation research. The paper opens a research direction around knowledge-grounded explanation generation, suggesting that future work should focus on automatic KG construction from domain data, adaptive retrieval strategies that learn which facts are most useful for explanations, and evaluation methods that measure both explanation faithfulness (correctness relative to the model) and fidelity (whether the explanation truly reflects why the model decided as it did).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::