Evaluation of Deontic Conditional Reasoning in Large Language Models: The Case of Wason's Selection Task
| Authors | Hirohiko Abe et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.06416 |
| Download | |
| Categories | cs.CL |
Abstract
As large language models (LLMs) advance in linguistic competence, their reasoning abilities are gaining increasing attention. In humans, reasoning often performs well in domain specific settings, particularly in normative rather than purely formal contexts. Although prior studies have compared LLM and human reasoning, the domain specificity of LLM reasoning remains underexplored. In this study, we introduce a new Wason Selection Task dataset that explicitly encodes deontic modality to systematically distinguish deontic from descriptive conditionals, and use it to examine LLMs' conditional reasoning under deontic rules. We further analyze whether observed error patterns are better explained by confirmation bias (a tendency to seek rule-supporting evidence) or by matching bias (a tendency to ignore negation and select items that lexically match elements of the rule). Results show that, like humans, LLMs reason better with deontic rules and display matching-bias-like errors. Together, these findings suggest that the performance of LLMs varies systematically across rule types and that their error patterns can parallel well-known human biases in this paradigm.
Engineering Breakdown
Plain English
This paper investigates how large language models perform conditional reasoning, particularly under deontic rules (rules about obligations and permissions) versus purely descriptive logical rules. The authors created a new Wason Selection Task dataset that explicitly encodes deontic modality to systematically test whether LLMs reason differently in normative contexts compared to formal logic contexts. The study examines whether observed reasoning errors in LLMs are better explained by confirmation bias (seeking evidence that supports rules) or matching bias (matching literal content), which are two competing psychological explanations for human reasoning failures that haven't been thoroughly tested in LLMs.
Core Technical Contribution
The core novelty is a carefully designed dataset and evaluation framework that isolates deontic reasoning from descriptive reasoning in the Wason Selection Task, enabling systematic analysis of domain-specific reasoning patterns in LLMs. Rather than treating LLM reasoning as domain-agnostic, this work explicitly operationalizes deontic modality as a variable, allowing researchers to observe whether LLMs exhibit the same bias patterns (confirmation vs. matching) that explain human performance degradation. The methodological contribution is significant because prior work compared LLM reasoning to human reasoning broadly, but didn't systematically decompose whether reasoning failures stem from the same cognitive biases or different failure modes entirely.
How It Works
The Wason Selection Task is a classic test where participants see a rule (e.g., 'If a card shows a vowel, it has an even number on the back') and four cards, then select which cards to flip to verify the rule. The authors create two variants: deontic rules encoding obligations/permissions (e.g., 'If someone is drinking beer, they must be over 21') and descriptive rules encoding logical conditionals (e.g., 'If a letter is sealed, it has a 5-cent stamp'). For each rule type, they prompt LLMs to select which cards to flip and then analyze the selection patterns. By comparing error rates and selection distributions between rule types, they measure whether LLMs show domain-specific improvements (deontic rules are easier, matching human psychology) and whether error patterns correlate with confirmation bias (preferring cards that match the rule's consequent) or matching bias (preferring cards with mentioned content).
Production Impact
For engineers building reasoning-dependent systems (compliance checking, legal document analysis, policy verification), this research clarifies that LLMs may reason more reliably in normative/deontic domains than in purely formal logical domains. This means production systems should explicitly frame rules in deontic language (obligations, permissions, prohibitions) rather than abstract conditionals to improve LLM accuracy on conditional reasoning tasks. The cost is minimal—no architectural changes required—but it requires careful prompt engineering and understanding that out-of-domain descriptive conditionals will degrade reasoning performance. For safety-critical applications like contract verification or regulatory compliance, this suggests testing LLM reasoning on both formal and normative variants of the same rule to understand failure modes before deployment.
Limitations and When Not to Use This
The paper focuses on the Wason Selection Task, which is a laboratory-style reasoning test that may not reflect real-world reasoning patterns where LLMs see much richer context and broader information. The analysis relies on error pattern attribution (confirmation bias vs. matching bias), but the paper doesn't directly intervene to eliminate these biases, so it remains unclear whether observed patterns reflect fundamental limitations or merely reflect the model's training distribution. The approach assumes deontic modality is the key axis of variation, but real-world reasoning involves many other domain-specific factors (expert vocabulary, causal structure, consequence severity) that aren't isolated in the dataset. Scale and generalization remain open: it's unclear whether findings hold across different model sizes, architectures, or whether the dataset is large enough to detect subtle interactions between bias types.
Research Context
This work builds directly on decades of cognitive psychology research on the Wason Selection Task (Wason, 1966 onwards) and human reasoning biases, applying that framework to modern LLMs. It extends recent work comparing LLM reasoning to human reasoning by introducing domain specificity as a systematic variable, moving beyond coarse comparisons to mechanistic explanations. The research opens a new direction: understanding whether LLMs inherit human cognitive biases or develop different failure modes, which has implications for both interpretability (do LLMs' brains work like ours?) and safety (should we expect LLMs to fail in the same contexts humans do?). This connects to broader efforts to benchmark and improve LLM reasoning on formal logic, mathematics, and planning tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::