DeonticBench: A Benchmark for Reasoning over Rules
| Authors | Guangyao Dou et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2604.04443 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Reasoning with complex, context-specific rules remains challenging for large language models (LLMs). In legal and policy settings, this manifests as deontic reasoning: reasoning about obligations, permissions, and prohibitions under explicit rules. While many recent benchmarks emphasize short-context mathematical reasoning, fewer focus on long-context, high-stakes deontic reasoning. To address this gap, we introduce DEONTICBENCH, a benchmark of 6,232 tasks across U.S. federal taxes, airline baggage policies, U.S. immigration administration, and U.S. state housing law. These tasks can be approached in multiple ways, including direct reasoning in language or with the aid of symbolic computation. Besides free-form chain-of-thought reasoning, DEONTICBENCH enables an optional solver-based workflow in which models translate statutes and case facts into executable Prolog, leading to formal problem interpretations and an explicit program trace. We release reference Prolog programs for all instances. Across frontier LLMs and coding models, best hard-subset performance reaches only 44.4% on SARA Numeric and 46.6 macro-F1 on Housing. We further study training with supervised fine-tuning and reinforcement learning for symbolic program generation. Although training improves Prolog generation quality, current RL methods still fail to solve these tasks reliably. Overall, DEONTICBENCH provides a benchmark for studying context-grounded rule reasoning in real-world domains under both symbolic and non-symbolic settings.
Engineering Breakdown
Plain English
DeonticBench introduces a new benchmark with 6,232 tasks designed to evaluate how well large language models can reason about rules in high-stakes, real-world domains like tax law, airline policies, immigration, and housing regulations. The paper identifies a gap in existing benchmarks: while many focus on short-context mathematical reasoning, few tackle long-context deontic reasoning—the ability to understand obligations, permissions, and prohibitions embedded in complex rule systems. The benchmark supports multiple solution approaches, including chain-of-thought reasoning and symbolic computation-aided solving, enabling researchers to test both pure language understanding and hybrid reasoning strategies.
Core Technical Contribution
The core contribution is a large-scale, multi-domain benchmark that combines long-context rule reasoning with deontic logic evaluation across four legally and procedurally complex U.S. domains. Unlike prior benchmarks that treat reasoning as a single-modality problem, DeonticBench explicitly supports both free-form language reasoning and solver-based workflows, allowing researchers to study when pure LLM reasoning fails and when symbolic computation becomes necessary. The benchmark dataset construction likely involved extraction and formalization of real regulatory documents, creating a realistic distribution of rule interactions and edge cases that models encounter in production legal and administrative systems.
How It Works
The benchmark operates by presenting models with a rule set (extracted from actual regulations or policies) and a specific scenario or query, then requiring the model to determine whether an obligation, permission, or prohibition applies. Input consists of: (1) the regulatory or policy text containing deontic rules, (2) a fact pattern or specific case, and (3) a query asking about what is required, allowed, or forbidden. The model can either perform pure language-based chain-of-thought reasoning to trace through the rules logically, or optionally invoke a symbolic solver that formalizes the rule structure and performs deductive inference. The output is a categorical answer (obligation met/unmet, permission granted/denied, prohibition violated/respected) plus the reasoning path, which is then evaluated against ground-truth annotations derived from expert legal interpretation or verified interpretations of the actual regulations.
Production Impact
For teams building AI systems in legal tech, compliance automation, or policy analysis, DeonticBench provides a standardized way to validate that your models understand rule systems with the accuracy required for production. In practice, this means you can test whether an LLM can reliably answer queries about tax deductions, baggage allowances, visa eligibility, or lease restrictions—problems where a single wrong answer costs customers money or creates legal liability. Adopting this benchmark would shift your evaluation pipeline from generic language understanding metrics to domain-specific deontic reasoning checks; you'd likely implement a hybrid strategy where the LLM first attempts reasoning, but for high-stakes decisions, you route to symbolic solvers or human review for confirmation. The production trade-off is clear: pure LLM reasoning is fast and flexible but error-prone on edge cases; solver-based approaches are slower and require formal rule encoding but provide verifiable correctness guarantees.
Limitations and When Not to Use This
The benchmark is built on U.S. federal and state regulations, so its applicability to international legal systems, private contracts, or dynamic rules that change frequently is limited. The paper likely assumes that ground-truth annotations are stable and unambiguous, but in practice, legal interpretation often involves judgment calls, conflicting precedents, and domain expert disagreement—the benchmark may not capture this interpretive variance. Long-context reasoning over complex rule sets pushes the boundaries of current LLM context windows and reasoning depth, so the benchmark may become easier as model capabilities improve without necessarily translating to better real-world legal reasoning. The solver-based approach requires formal rule encoding, which is labor-intensive and may not scale to the full corpus of regulatory text; the paper does not address how to handle ambiguous, vague, or contradictory rules that appear in real legal documents.
Research Context
DeonticBench extends prior work on specialized reasoning benchmarks (like those for mathematical reasoning or commonsense QA) by focusing on a previously underexplored area: formal deontic logic in realistic, high-stakes contexts. It builds on research in legal AI and compliance automation, but provides the first large-scale public benchmark that combines long-context rule reasoning with symbolic computation support, enabling the community to move beyond toy problems toward practically deployable systems. The work opens research directions in hybrid reasoning (when to use LLM vs. symbolic solver), rule formalization at scale, and handling ambiguity in real regulatory text—all critical for broader adoption of AI in law and policy.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::