ClawArena: Benchmarking AI Agents in Evolving Information Environments
| Authors | Haonian Ji et al. |
| Year | 2026 |
| HF Upvotes | 36 |
| arXiv | 2604.04202 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is scattered across heterogeneous sources that often contradict one another, new information can invalidate earlier conclusions, and user preferences surface through corrections rather than explicit instructions. Existing benchmarks largely assume static, single-authority settings and do not evaluate whether agents can keep up with this complexity. We introduce ClawArena, a benchmark for evaluating AI agents in evolving information environments. Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces across multi-channel sessions, workspace files, and staged updates. Evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization, whose interactions yield a 14-category question taxonomy. Two question formats, multi-choice (set-selection) and shell-based executable checks, test both reasoning and workspace grounding. The current release contains 64 scenarios across 8 professional domains, totaling 1{,}879 evaluation rounds and 365 dynamic updates. Experiments on five agent frameworks and five language models show that both model capability (15.4% range) and framework design (9.2%) substantially affect performance, that self-evolving skill frameworks can partially close model-capability gaps, and that belief revision difficulty is determined by update design strategy rather than the mere presence of updates. Code is available at https://github.com/aiming-lab/ClawArena.
Engineering Breakdown
Plain English
ClawArena is a benchmark designed to test whether AI agents can maintain accurate beliefs and make correct decisions when operating in real-world information environments where data constantly changes, contradicts itself, and comes from multiple sources. The paper identifies a critical gap in existing benchmarks—they typically assume static information from a single authoritative source, whereas deployed AI assistants must handle scattered evidence across emails, files, chat sessions, and user corrections that can invalidate previous conclusions. ClawArena solves this by providing scenarios with a hidden ground truth while exposing agents only to noisy, partial, and often contradictory information traces across multi-channel sessions and staged updates. This creates a more realistic evaluation framework that measures whether agents can incrementally update their beliefs as new information arrives and detect when earlier conclusions have become invalid.
Core Technical Contribution
The core technical novelty is the creation of a benchmark architecture that decouples the true state of the world from the agent's observation space, allowing controlled injection of noise, contradiction, and temporal dynamics into agent evaluation. Unlike existing static benchmarks that present information in isolation, ClawArena systematically generates multi-source information traces with explicit ground truth, enabling measurement of an agent's ability to resolve conflicting information, weight evidence from heterogeneous sources, and recognize when to overturn previous conclusions. The benchmark includes staged information updates that simulate how real-world evidence surfaces incrementally—through corrections, new documents, and user feedback rather than explicit instruction—requiring agents to continuously maintain consistent beliefs. This design enables rigorous quantification of belief consistency, information reconciliation ability, and temporal reasoning that prior benchmarks cannot measure.
How It Works
ClawArena operates by maintaining a complete hidden ground truth state for each scenario while systematically distributing partial, noisy, and sometimes contradictory observations across multiple information channels (emails, chat sessions, workspace files, user corrections). The agent receives these traces sequentially or in clustered updates, simulating how information naturally arrives in production systems. The benchmark measures three key dimensions: (1) whether the agent correctly identifies contradictions when they appear, (2) whether it properly updates its belief state when new evidence invalidates earlier conclusions, and (3) whether it recognizes the reliability and authority level of different information sources. Evaluation is performed by comparing the agent's maintained belief state against the hidden ground truth at multiple checkpoints throughout the scenario, capturing both steady-state accuracy and the agent's ability to transition between belief states as information evolves. The multi-channel structure forces agents to handle information fusion—combining evidence from emails, files, and direct user statements that may have different reliability levels.
Production Impact
For engineers building persistent AI assistants (customer service bots, enterprise knowledge workers, personal assistants), ClawArena provides a critical testing framework that existing benchmarks lack—the ability to validate that agents won't hallucinate, contradict themselves, or maintain stale beliefs as information changes. In production pipelines, you would use this benchmark before deployment to ensure your agent can handle real-world data conditions: conflicting information in email threads, file updates that contradict earlier statements, user corrections that override system conclusions. Adoption would require building belief-state tracking into your agent architecture (not just raw retrieval), implementing evidence reconciliation logic, and adding confidence scoring systems that weight information sources. The trade-offs are significant: agents need stateful memory management, more sophisticated reasoning to detect contradictions, and potentially higher latency due to evidence reconciliation computations; however, the alternative—shipping agents that confidently serve outdated or contradictory information—creates severe user trust and liability issues.
Limitations and When Not to Use This
ClawArena requires a complete hidden ground truth for each scenario, which limits the benchmark to domains where ground truth can be definitively established; this may not scale to open-ended domains like subjective preferences or emerging events where consensus itself is uncertain. The benchmark assumes information sources can be tagged with reliability levels and that contradictions are resolvable through careful reasoning, but real-world sources often have subtle reliability differences that aren't easily parameterizable, and some contradictions may be genuinely unresolvable. The paper doesn't address computational overhead—belief-state tracking and evidence reconciliation add complexity, and the benchmark doesn't report how much this slows down inference or what hardware requirements are needed for production deployment. Additionally, ClawArena evaluates agents at the belief-maintenance level but doesn't necessarily address downstream decision quality; an agent might maintain correct beliefs while still making poor decisions due to gaps in planning, reasoning, or user intent modeling.
Research Context
ClawArena builds on a growing recognition that static benchmarks (like those in the language model evaluation literature) inadequately capture the challenges of persistent, deployed AI agents. It extends prior work on information integration, multi-source fusion, and temporal reasoning by creating the first benchmark that systematically combines all three. The work connects to research on belief update in humans and rational agents, but operationalizes it for modern LLM-based systems that lack explicit belief representations. This opens a new research direction: moving beyond question-answering benchmarks toward benchmarks that evaluate agents as systems that must maintain consistent, evolving state over extended interactions with users and information sources.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::