KWBench: Measuring Unprompted Problem Recognition in Knowledge Work
| Authors | Ankit Maloo |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2604.15760 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We introduce the first version of KWBench (Knowledge Work Bench), a benchmark for unprompted problem recognition in large language models: can an LLM identify a professional scenario before attempting to solve it. Existing frontier benchmarks have saturated, and most knowledge-work evaluations to date reduce to extraction or task completion against a specification. KWBench targets the step before that: recognizing the governing structure of the situation from raw inputs alone. The benchmark contains 223 tasks sourced from practitioners across acquisitions, contract negotiations, clinical pharmacy, organizational politics, fraud analysis, and incentive design. Each task encodes a formal game-theoretic pattern (principal-agent conflict, signaling, mechanism design failure, strategic omission, coalitional dynamics, strategic interdependence) and carries structured ground truth recording the expert reading of the situation and the anticipated failure modes. Models receive raw data and a task prompt with no indication of problem type. Scoring is a three-tier rubric gated by a mandatory conjunctive check. Mandatory criteria encode the predicted wrong paths. We evaluate 16 models. The best model passes on 27.9% of tasks. The top two models agree on only 31.7% of their passes. Among the top 8, 44 tasks are solved by exactly one model; routing across the top 8 covers 50.7% of the benchmark, nearly double the best single model. Conditional on passing, quality scores converge (approx 83% across models); unconditional scores do not. Same models articulate the relevant game-theoretic concept correctly when asked, then fail to apply it unprompted. We release KWBench to shift how frontier models are evaluated on knowledge work, scoring them on whether they recognize the right problem from the situation alone, not only on how well they execute once the problem has been framed for them.
Engineering Breakdown
Plain English
KWBench introduces the first benchmark specifically designed to measure whether large language models can recognize and identify the underlying structure of professional problems before attempting to solve them. The benchmark contains 223 real-world tasks sourced from practitioners in domains like acquisitions, contract negotiations, clinical pharmacy, fraud analysis, and organizational politics. Rather than testing whether an LLM can extract information or complete a task against a known specification, KWBench measures the harder problem: can the model recognize what type of problem it's facing from raw inputs alone? Each task encodes a formal game-theoretic pattern (principal-agent conflicts, signaling games, mechanism design failures), making this a measurement tool for a capability that existing frontier benchmarks don't evaluate.
Core Technical Contribution
The core novelty is identifying and benchmarking unprompted problem recognition as a distinct, measurable capability in LLMs—a step that exists logically before task execution but hasn't been formally evaluated before. Traditional knowledge-work benchmarks focus on task completion (given a problem type, solve it) or information extraction, but KWBench reverses the assumption: you give the model raw professional context and measure whether it correctly identifies the governing problem structure without explicit instructions. The benchmark operationalizes this by encoding 223 tasks around formal game-theoretic patterns, creating a systematic way to measure whether models understand the conceptual architecture of professional scenarios. This is technically novel because it requires the model to do unsupervised classification of problem types from realistic, messy inputs rather than constrained task specification.
How It Works
The mechanism operates in three stages: first, a task is presented to the model as raw professional context (e.g., a scenario description, contract excerpt, or clinical situation) with no explicit problem label or instruction. Second, the model must independently recognize and articulate what type of problem structure governs the scenario—does it involve principal-agent misalignment, information asymmetry requiring signaling, a broken mechanism design, or strategic omission by one party? Third, evaluation happens by comparing the model's unprompted identification against the ground-truth game-theoretic pattern that practitioners verified as the core governing structure. The input is unstructured or semi-structured professional text; the transformation is the model's internal reasoning process to recognize structural patterns; the output is the model's stated problem classification, which is then scored against expert annotations. Each task encodes one formal game-theoretic pattern, so success means the model identified that pattern without being told what category to look for.
Production Impact
For engineers building AI systems in knowledge work domains—M&A platforms, contract review, clinical decision support, fraud detection—this benchmark reveals a critical capability gap: your LLM pipeline likely assumes a user or upstream system has already classified the problem type, but in reality, professionals often present raw situations and expect the system to recognize what's happening. Adopting KWBench-style evaluation means instrumenting your system to test whether models can handle this unprompted recognition step, which would let you identify failure modes before deployment (e.g., a model that misclassifies a principal-agent problem as a simple extraction task). The practical trade-off is that problem recognition requires the model to engage in longer-context reasoning and potentially multi-step inference, which increases latency and token consumption compared to classification-only pipelines. Integration would involve adding an evaluation layer that extracts model-stated problem hypotheses, parses them against a game-theoretic taxonomy, and routes to appropriate downstream solvers—this adds orchestration complexity but enables more robust end-to-end systems.
Limitations and When Not to Use This
KWBench's scope is limited to 223 tasks, which while sourced from real practitioners, may not capture the full diversity of problem structures in unstructured professional domains; frontier models might overfit to the specific game-theoretic patterns encoded in this benchmark without generalizing to novel hybrid scenarios. The evaluation assumes that problem structure can be cleanly decomposed into discrete game-theoretic categories (principal-agent, signaling, mechanism design, etc.), but many real professional problems involve multiple overlapping structures, and the benchmark may penalize models for reasonable but non-canonical problem framings. The benchmark also doesn't measure whether correct problem recognition translates to better solution quality—a model might identify the right structure but still execute poorly, creating a gap between evaluation and actual business value. Additionally, the paper's scope to game-theoretic framings means it may not capture important non-game-theoretic problem recognition (e.g., domain-specific classification in clinical pharmacy that isn't naturally modeled as strategic interaction), limiting applicability to some knowledge work domains.
Research Context
This work builds on a growing recognition that LLM benchmarking has saturated at the frontier—models like GPT-4 achieve >90% on MMLU, reasoning benchmarks, and most task-completion measures—making it essential to identify harder, more realistic problem formulations. KWBench is motivated by prior work on task understanding and zero-shot classification, but it specifically targets knowledge work, a domain where prior benchmarks (like those in legal AI, medical AI, etc.) focus on task execution rather than problem recognition. The benchmark aligns with research in AI interpretability and reasoning that emphasizes structure recognition as a foundational capability, and it operationalizes problem recognition through formal game theory, connecting to decades of work in mechanism design and information economics. This opens a new research direction: evaluating LLMs not just on whether they can solve known problem types, but on whether they can independently recognize what category of problem they're facing, which is a precursor to robust AI systems for complex professional domains.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::