AILS-NTUA at SemEval-2026 Task 12: Graph-Based Retrieval and Reflective Prompting for Abductive Event Reasoning
| Authors | Nikolas Karafyllis et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.04319 |
| Download | |
| Categories | cs.CL |
Abstract
We present a winning three-stage system for SemEval 2026 Task12: Abductive Event Reasoning that combines graph-based retrieval, LLM-driven abductive reasoning with prompt design optimized through reflective prompt evolution, and post-hoc consistency enforcement; our system ranks first on the evaluation-phase leaderboard with an accuracy score of 0.95. Cross-model error analysis across 14 models (7families) reveals three shared inductive biases: causal chain incompleteness, proximate cause preference, and salience bias, whose cross-family convergence (51% cause-count reduction) indicates systematic rather than model-specific failure modes in multi-label causal reasoning.
Engineering Breakdown
Plain English
This paper presents a three-stage system that won first place at SemEval 2026 Task 12 for abductive event reasoning, achieving 95% accuracy by combining graph-based retrieval, LLM-driven reasoning with optimized prompt design, and consistency enforcement. The key finding goes beyond the competition result: the authors analyzed failure modes across 14 different models from 7 model families and discovered three shared systematic biases that cause reasoning failures—incomplete causal chains, preference for proximate causes over root causes, and salience bias—that appear consistently across all model families. This convergence suggests these aren't random model-specific bugs but fundamental limitations in how current models approach multi-label causal reasoning. The 51% reduction in cause-count errors across model families indicates these biases are deeply rooted in how neural networks process causal information.
Core Technical Contribution
The technical novelty has two parts. First, the system architecture itself combines graph-based retrieval (to pull relevant context) with LLM-driven abductive reasoning and a novel reflective prompt evolution method that optimizes prompts through iterative refinement rather than manual engineering. Second and more significant, the paper's core contribution is the cross-model error analysis framework that identifies systematic inductive biases shared across diverse model architectures—this suggests that causal reasoning failures aren't model-specific bugs but convergent failure modes in neural architectures. Prior work typically analyzed individual model weaknesses; this paper reveals that seven different model families independently converge on the same three failure patterns, which is a meaningful discovery about the fundamental limits of current neural approaches to causal reasoning.
How It Works
The system operates in three stages. Stage one uses graph-based retrieval to construct a knowledge graph or retrieve relevant event chains from a corpus—this grounds the reasoning task in structured context rather than relying purely on the model's parametric knowledge. Stage two feeds this retrieved context into an LLM with prompts that are optimized through reflective prompt evolution, which means the system iteratively refines prompts by analyzing previous outputs and adjusting the instructions accordingly (rather than using fixed templates). Stage three applies post-hoc consistency enforcement, which takes the model's predictions and corrects them to ensure they form logically consistent causal chains—if the model predicts events A→B and B→C but not A→C, this stage fills in the missing link. The input is an event reasoning task; the output is a set of ranked abductive explanations that satisfy causal consistency constraints.
Production Impact
For engineers building causal reasoning or event understanding systems, this work provides both immediate and strategic value. Immediately, the three-stage architecture is implementable: you'd add a retrieval layer (likely using vector search or graph databases), wrap your LLM with a prompt refinement loop (using something like an additional critic model or rule-based feedback), and apply logical consistency post-processing (constraint satisfaction or topological sorting). The bigger impact is the cross-model error analysis framework—teams should adopt similar diagnostic approaches to understand whether their failures are model-specific or systematic. However, there are production trade-offs: the three-stage system increases latency (retrieval + LLM + consistency checking), requires maintaining both a retrieval corpus and a prompt refinement loop, and the 95% accuracy is task-specific (SemEval 2026 Task 12) so real-world performance on your domain's event reasoning may be lower. The reflective prompt evolution method requires extra inference passes, increasing compute cost, but the consistency enforcement is relatively cheap since it's post-hoc rule application.
Limitations and When Not to Use This
The paper's scope is limited to a specific task—SemEval 2026 Task 12 abductive event reasoning—so generalization to other causal reasoning domains (medical diagnosis, financial forecasting, root cause analysis) is unproven. The three identified biases (causal chain incompleteness, proximate cause preference, salience bias) are characterized empirically but the paper doesn't propose fundamental solutions to fix them—the system works around them with prompt engineering and consistency enforcement rather than addressing them at the architectural or training level. The cross-model analysis, while valuable, only covers 14 models from 7 families; it's unclear whether these same biases appear in newer architectures (multimodal models, mixture-of-experts systems, or future model classes) or whether they're specific to transformer-based language models. Finally, the paper doesn't address data contamination risk—if the retrieval corpus or training data overlapped with the SemEval test set, the 95% accuracy could be inflated, and the applicability to tasks with distribution shift remains unknown.
Research Context
This work builds on a long line of research in semantic understanding and event reasoning, particularly the SemEval shared task series which has driven progress in NLP evaluation. The paper directly improves on prior SemEval submissions by introducing the three-stage architectural pattern and the reflective prompt evolution technique, suggesting that LLM-based reasoning benefits from iterative prompt optimization. The cross-model error analysis contributes to a growing body of work on neural model interpretation and the discovery of shared failure modes (similar to recent work on mechanistic interpretability and circuit analysis), indicating that different model families may be learning similar spurious correlations or approximations. This opens a research direction: if multiple model families independently develop the same causal reasoning biases, future work should investigate whether these biases are inevitable given transformer architecture, or whether training procedures (different data, objectives, or regularization) could prevent them.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::