ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
| Authors | Xiangyi Li et al. |
| Year | 2026 |
| HF Upvotes | 22 |
| arXiv | 2604.05172 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We decompose agent scaffolding into two independent levers (domain skills that inject API knowledge via progressive disclosure, and a meta prompt that coordinates behavior across services) and vary both to measure their separate and combined effects. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39-64% but exhibit unsafe action rates of 7-33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53-63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. We identify eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification.
Engineering Breakdown
Plain English
This paper introduces ClawsBench, a benchmark designed to evaluate large language model agents on realistic productivity tasks without the risk of causing actual damage to real services. The benchmark includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with complete state management and the ability to take deterministic snapshots and restore to previous states. It covers 44 structured tasks ranging from single-service operations to complex cross-service workflows and safety-critical scenarios. The key insight is that existing benchmarks use overly simplified environments that don't capture the real complexity and statefulness of multi-service workflows, making it difficult to safely evaluate and improve LLM agents before deploying them to production systems.
Core Technical Contribution
The core technical contribution is a stateful benchmark environment that faithfully reproduces the complexity of real productivity services while remaining isolated and deterministic. Unlike prior benchmarks that use simplified, stateless mocks or toy environments, ClawsBench implements full state management across multiple interconnected services with the ability to snapshot and restore to exact previous states—enabling reproducible evaluation and safe iteration. The paper also decomposes agent scaffolding into two independent levers (domain skills and execution capabilities), providing a framework for systematically improving agent performance. This dual-lever decomposition allows researchers to isolate whether performance gains come from better task understanding or better service interaction, which is a novel methodological contribution to agent evaluation.
How It Works
ClawsBench works by creating high-fidelity mock implementations of five Google Workspace services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) that accurately simulate the APIs, state transitions, and side effects of the real services. Each task is defined as a structured goal (e.g., 'schedule a meeting and send invites to attendees') along with an initial system state and expected outcome state. When an LLM agent attempts a task, it receives access to a stateful environment where each action (sending an email, creating a calendar event, updating a document) modifies the underlying state deterministically. The benchmark includes snapshot/restore functionality, allowing developers to revert the environment to any previous state after an agent makes a mistake, enabling safe iteration and repeated evaluation. The 44 tasks are categorized into three types: single-service tasks (one service involved), cross-service tasks (multiple services required), and safety-critical tasks (tasks where incorrect execution causes harm, like deleting important files). Agent scaffolding is decomposed into domain skills (understanding what action to take) and execution capabilities (correctly invoking service APIs), allowing independent evaluation of each component.
Production Impact
For engineers deploying LLM agents to production, ClawsBench solves the critical safety and evaluation problem: you can now thoroughly test agent behavior on realistic, complex workflows before risking actual data loss or service disruption. This enables a development cycle where teams iterate on agent prompts, reasoning strategies, and service integration logic against a faithful simulation before going live. The snapshot/restore capability means you can run thousands of evaluation trials without manual cleanup or state corruption, dramatically reducing the friction in benchmarking. However, there are significant production trade-offs: you must maintain mock implementations of your services that stay in sync with real API changes, the deterministic snapshot/restore system adds overhead and complexity, and the benchmark only covers Google Workspace services—you'll need to build similar mocks for other third-party services in your production stack. The decomposition of scaffolding into domain skills and execution is useful for identifying bottlenecks (e.g., 'our agent understands tasks well but fails at API calls'), but validating these insights on real services still requires careful gradual rollout procedures.
Limitations and When Not to Use This
ClawsBench inherently cannot capture emergent behaviors that only arise in real production environments with real users, concurrent operations, and actual data consistency challenges. The mock services, while high-fidelity, are still simplified simulations—they don't handle network latency, API rate limits, unexpected failures, or the edge cases that emerge from real user data patterns and service interactions. The benchmark is currently limited to Google Workspace services; teams using Salesforce, SAP, Jira, or other enterprise systems must build their own mocks, which is non-trivial and requires deep API knowledge. The safety-critical tasks are synthetically defined scenarios that may not cover all the ways agents can cause real harm in production (e.g., cascading failures across services, permission model violations, or data privacy leaks). Additionally, the paper doesn't address how agent performance on ClawsBench correlates with real-world performance—there's a missing validation study showing that agents that perform well on the benchmark actually perform well on live services.
Research Context
This paper advances the agent evaluation benchmarking space, building on prior work like WebShop, Mind2Web, and other task automation benchmarks that focus on single-service or single-website environments. It addresses a key limitation in that prior work: those benchmarks treat each service as stateless or independent, whereas real productivity workflows are inherently stateful and require coordination across services. ClawsBench is positioned as the first production-grade benchmark for multi-service agent evaluation, filling a gap between toy environments (which are too simple) and real services (which are too risky). The research direction it opens is towards benchmarks that balance fidelity, safety, and reproducibility for agent evaluation in complex real-world domains—suggesting that future work should extend this pattern to other domains (e-commerce, finance, healthcare) and explore how agent architectures and training procedures can be optimized specifically for stateful, multi-service environments.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::