OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models
| Authors | Xiaomeng Hu et al. |
| Year | 2026 |
| HF Upvotes | 61 |
| arXiv | 2604.10866 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import processing), yet existing benchmarks can only evaluate agents in the few domains where public environments exist. We introduce OccuBench, a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, enabled by Language World Models (LWMs) that simulate domain-specific environments through LLM-driven tool response generation. Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity. OccuBench evaluates agents along two complementary dimensions: task completion across professional domains and environmental robustness under controlled fault injection (explicit errors, implicit data degradation, and mixed faults). We evaluate 15 frontier models across 8 model families and find that: (1) no single model dominates all industries, as each has a distinct occupational capability profile; (2) implicit faults (truncated data, missing fields) are harder than both explicit errors (timeouts, 500s) and mixed faults, because they lack overt error signals and require the agent to independently detect data degradation; (3) larger models, newer generations, and higher reasoning effort consistently improve performance. GPT-5.2 improves by 27.5 points from minimal to maximum reasoning effort; and (4) strong agents are not necessarily strong environment simulators. Simulator quality is critical for LWM-based evaluation reliability. OccuBench provides the first systematic cross-industry evaluation of AI agents on professional occupational tasks.
Engineering Breakdown
Plain English
OccuBench is a new benchmark for evaluating AI agents across 100 real-world professional tasks spanning 10 industries and 65 specialized domains—a massive expansion from existing benchmarks that only cover a handful of domains. The authors solve the problem of domain scarcity by introducing Language World Models (LWMs), which use LLMs to simulate domain-specific environments and generate realistic tool responses without requiring actual professional software systems. They built an automated pipeline that creates evaluation instances with guaranteed solvability, calibrated difficulty levels, and diversity grounded in real documents. This addresses a critical gap: existing agent benchmarks can't evaluate performance across the breadth of professional work that AI systems will eventually need to handle.
Core Technical Contribution
The core innovation is Language World Models (LWMs)—a paradigm where LLMs act as environment simulators, generating domain-realistic responses to tool calls without access to actual professional systems or databases. Rather than building 100+ separate sandboxed environments or APIs (which is impractical), the authors use LLMs with domain-specific prompting and document grounding to simulate how tools would respond in contexts like emergency triage systems, nuclear reactor monitoring, or customs processing. The second key contribution is a multi-agent synthesis pipeline that automatically generates evaluation instances with three guarantees: solutions exist (solvability), difficulty is calibrated to agent capability levels, and diversity comes from grounding scenarios in real professional documents. This is technically novel because it combines synthetic environment generation with automated instance synthesis, enabling benchmarks at scales previously impossible.
How It Works
The system operates in stages: first, domain experts or document corpora define the professional context and available tools for each of 65 specialized domains. Second, the LWM takes a user task description and generates realistic tool responses by conditioning on domain context—if an agent calls a medical database tool in the ED triage domain, the LWM returns medically plausible data consistent with the scenario. Third, the multi-agent synthesis pipeline constructs diverse problem instances by (a) sampling from real professional documents to ground scenarios, (b) generating multiple solution paths and checking solvability via agent simulation, and (c) adjusting task parameters to hit target difficulty levels. Agents are then evaluated on two dimensions: success on the primary task objective and performance on secondary evaluation criteria (efficiency, safety, reasoning quality). The pipeline iterates, using agent performance signals to recalibrate difficulty and diversity until instances meet quality thresholds.
Production Impact
For teams building production AI agents for professional domains, OccuBench provides the first comprehensive evaluation methodology that covers realistic breadth without requiring actual integration with legacy systems or regulated professional software. Engineers can use this to benchmark proprietary agents against standardized professional tasks before deployment—critical for high-stakes domains like medical triage or safety monitoring where black-box testing is insufficient. However, adoption requires understanding that LWM-generated environments, while realistic, may miss edge cases or adversarial scenarios that appear in production; they're a supplement to, not replacement for, domain-specific testing and regulatory validation. The computational cost is moderate (LWM inference for 100 scenarios × multiple agent attempts), but far cheaper than building real sandboxed environments. Integration complexity is low—it's a pure evaluation benchmark, not a training framework—so you can add it to existing agent pipelines as a testing gate.
Limitations and When Not to Use This
OccuBench's LWM-based environment simulation, while scalable, cannot guarantee coverage of rare but critical edge cases, failure modes, or adversarial scenarios that emerge in production professional systems—a benchmark task completing successfully doesn't mean the agent will handle unexpected system degradation, user error, or novel attack vectors. The benchmark assumes domain documents are available and representative, which may not hold for proprietary or rapidly-evolving professional domains, and it doesn't measure agent performance on tasks requiring real-time interaction, continuous monitoring, or feedback loops with actual users. Solvability guarantees apply only within the LWM's simulation; real-world task instances may have latent inconsistencies or ambiguities that the synthetic environment doesn't capture. The paper doesn't deeply address how to validate that LWM-generated tool responses are actually faithful to real professional systems—hallucination or subtle incorrectness in simulated responses could produce misleading benchmark scores.
Research Context
This work builds on the long line of AI benchmark construction (SuperGLUE, MMLU, etc.) but addresses a fundamental gap: most agent benchmarks require hand-built, domain-specific environments or APIs, limiting scale. It extends recent work on LLM-based world models and simulation, but applies that to the professional/occupational domain at an unprecedented scale (100 scenarios across 65 domains vs. prior work that targeted single domains). OccuBench also relates to research on synthetic data generation and automated instance synthesis—techniques used in training but less commonly applied to evaluation design. The benchmark opens research directions in multi-domain agent generalization, calibration of difficulty across domains, and measuring domain transfer in AI agents, which are critical questions as AI systems move from single-task to broad professional deployment.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::