Causality Elicitation from Large Language Models
| Authors | Takashi Kameyama et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2603.04276 |
| Download | |
| Categories | cs.LG, cs.AI, cs.CL |
Abstract
Large language models (LLMs) are trained on enormous amounts of data and encode knowledge in their parameters. We propose a pipeline to elicit causal relationships from LLMs. Specifically, (i) we sample many documents from LLMs on a given topic, (ii) we extract an event list from from each document, (iii) we group events that appear across documents into canonical events, (iv) we construct a binary indicator vector for each document over canonical events, and (v) we estimate candidate causal graphs using causal discovery methods. Our approach does not guarantee real-world causality. Rather, it provides a framework for presenting the set of causal hypotheses that LLMs can plausibly assume, as an inspectable set of variables and candidate graphs.
Engineering Breakdown
Plain English
This paper proposes a systematic pipeline to extract causal relationships that large language models implicitly encode in their parameters. The approach works by sampling documents from an LLM on a topic, extracting events from each document, grouping them into canonical events across documents, building binary indicator vectors, and then applying causal discovery algorithms to infer candidate causal graphs. The key insight is that this doesn't claim to find ground-truth causality, but rather surfaces the set of causal hypotheses that an LLM could plausibly assume—making the model's implicit causal reasoning inspectable and auditable. This addresses a real problem: LLMs contain encoded knowledge about causal relationships, but those relationships are locked inside the model weights and difficult to extract or verify.
Core Technical Contribution
The core novelty is a five-step pipeline that bridges the gap between unstructured LLM outputs and structured causal graphs in a way that's scalable and interpretable. Unlike prior work that either tries to directly prompt LLMs for causal statements (which are often inconsistent) or applies causal discovery to arbitrary text corpora (which lack grounding), this approach leverages the LLM as a generative source while using event canonicalization to create a consistent feature space for causal inference. The key technical contribution is the canonicalization step—grouping event mentions across multiple documents into unified variables—which enables downstream causal discovery algorithms to work on stable, high-quality data. This transforms the problem from "how do we ask an LLM about causality" to "what causal structures are consistent with the distributions the LLM generates." More concretely, the contribution is treating LLM sampling as a black-box mechanism for generating documents and then applying standard causal discovery (e.g., constraint-based or score-based methods) to the resulting binary event matrices, thus making the latent causal assumptions of the LLM explicit and auditable.
How It Works
The pipeline operates in five sequential stages. First, you prompt the LLM repeatedly on a given topic (e.g., "describe a financial market crash") and collect many generated documents, capturing the model's implicit distribution over narratives. Second, from each document you extract a structured list of events (using either rule-based parsing, another LLM call, or semantic event extraction) so that each document becomes a sequence of discrete occurrences. Third, you canonicalize events across all documents—clustering mentions like "stock prices fell," "markets declined," and "shares dropped" into a single canonical event variable—effectively creating a unified vocabulary. Fourth, you build a binary indicator matrix where each row is a document and each column is a canonical event, with 1 indicating presence and 0 indicating absence, resulting in a dataset suitable for causal discovery. Fifth, you apply off-the-shelf causal discovery methods (constraint-based like PC, or score-based like GES) to this binary matrix to recover a directed acyclic graph (DAG) representing plausible causal relationships. The output is a set of candidate causal graphs ranked by some score (BIC, constraint satisfaction, etc.), which can be inspected and compared to domain knowledge.
Production Impact
For teams building systems that need to understand or audit what causal assumptions an LLM encodes, this pipeline provides a concrete, reproducible way to surface and validate those assumptions. In practice, you would use this to detect whether an LLM trained on particular data has learned spurious or biased causal relationships (e.g., between demographic attributes and outcomes), then flag those for human review or retraining. The computational cost is moderate: sampling hundreds or thousands of documents from an LLM is cheap relative to training, but the causal discovery step scales polynomially with the number of variables (canonical events), so you need to keep the event space manageable (hundreds rather than thousands). Integration is straightforward—it's a post-hoc analysis that doesn't require modifying the LLM itself, making it compatible with any existing deployed model. The main trade-off is that the quality of recovered causal graphs depends heavily on the canonicalization step; poor event extraction or grouping leads to noisy features and unreliable causal inferences.
Limitations and When Not to Use This
This approach makes strong assumptions that frequently break in practice: it assumes events can be cleanly extracted and canonicalized, that the LLM's generated documents are representative of its true causal beliefs, and that causal discovery algorithms can reliably recover structure from the resulting data. In reality, LLMs often generate text with implicit or vague causality that's hard to extract programmatically, and the causal discovery problem is known to be under-identified when you don't have interventional data—multiple DAGs can be observationally equivalent, so the recovered graphs may include spurious edges. The paper explicitly disclaims finding real-world causality; instead it finds causal patterns the LLM can plausibly generate, which may reflect training data biases rather than true causal mechanisms. Additionally, the method doesn't handle confounding well unless you have very clean event definitions, and it may miss causal relationships that the LLM knows but doesn't express frequently in its generative distribution. Follow-up work is needed to validate whether recovered graphs actually correspond to causal reasoning the model uses during generation versus artifacts of the sampling and canonicalization process.
Research Context
This work sits at the intersection of interpretability research on LLMs and the classical causal inference literature. It builds on prior efforts to probe LLMs for factual and causal knowledge (like prompting-based fact extraction or using LLMs as oracles), but advances them by applying formal causal discovery algorithms rather than relying on heuristic prompts. The approach connects to interpretability work on extracting structured knowledge from neural networks, as well as to recent work on causal representation learning that aims to recover causal variables from observational data. The paper implicitly acknowledges the open problem of grounding LLM knowledge in real-world causality—a tension at the heart of using language models as sources of causal information. This opens research directions in (1) validating whether recovered causal graphs align with ground-truth interventional experiments, (2) scaling the canonicalization step to handle larger event vocabularies, and (3) combining this with retrieval-augmented generation (RAG) to condition the LLM on specific sources and improve causal consistency.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::