Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
| Authors | Chenxin Li et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2604.28139 |
| Download | |
| Categories | cs.SE, cs.AI |
Abstract
LLM agents are expected to complete end-to-end units of work across software tools, business services, and local workspaces. Yet many agent benchmarks freeze a curated task set at release time and grade mainly the final response, making it difficult to evaluate agents against evolving workflow demand or verify whether a task was executed. We introduce Claw-Eval-Live, a live benchmark for workflow agents that separates a refreshable signal layer, updated across releases from public workflow-demand signals, from a reproducible, time-stamped release snapshot. Each release is constructed from public workflow-demand signals, with ClawHub Top-500 skills used in the current release, and materialized as controlled tasks with fixed fixtures, services, workspaces, and graders. For grading, Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts, using deterministic checks when evidence is sufficient and structured LLM judging only for semantic dimensions. The release contains 105 tasks spanning controlled business services and local workspace repair, and evaluates 13 frontier models under a shared public pass rule. Experiments reveal that reliable workflow automation remains far from solved: the leading model passes only 66.7% of tasks and no model reaches 70%. Failures are structured by task family and execution surface, with HR, management, and multi-system business workflows as persistent bottlenecks and local workspace repair comparatively easier but unsaturated. Leaderboard rank alone is insufficient because models with similar pass rates can diverge in overall completion, and task-level discrimination concentrates in a middle band of tasks. Claw-Eval-Live suggests that workflow-agent evaluation should be grounded twice, in fresh external demand and in verifiable agent action.
Engineering Breakdown
Plain English
Claw-Eval-Live addresses a critical gap in LLM agent evaluation: existing benchmarks freeze their task sets at release and only grade final outputs, making them unable to track how agent capabilities should evolve as real-world workflows change. The paper introduces a live benchmark framework that separates a refreshable signal layer (updated continuously from public workflow-demand data) from reproducible, timestamped release snapshots graded with fixed fixtures and services. Each release snapshot is constructed from the ClawHub Top-500 skills currently in use, materialized as controlled tasks with deterministic graders that can verify whether intermediate actions were actually executed, not just whether the final answer looks correct. This hybrid approach enables tracking agent performance against realistic, evolving task distributions while maintaining reproducibility for comparing results across time.
Core Technical Contribution
The key innovation is the dual-layer benchmark architecture that decouples signal freshness from reproducibility. Rather than releasing a frozen benchmark that becomes stale, Claw-Eval-Live maintains a continuously updated public signal layer (drawn from real workflow demand) and creates periodic fixed snapshots for benchmarking—similar to how train/test splits work in ML, but applied to benchmark evolution itself. The framework also introduces materialized, instrumented task execution that can grade intermediate steps and verify tool invocations against actual service responses, moving beyond terminal-response evaluation. This is architecturally different from prior work like SWE-bench or AgentBench, which either snapshot real repositories or curate static task sets; Claw-Eval-Live provides a systematic way to keep benchmarks synchronized with real-world demand while maintaining scientific reproducibility.
How It Works
The system operates in two parallel layers: (1) a public workflow-demand signal layer that continuously ingests data on which skills are actually used in production, indexed by frequency and recency, and (2) release snapshots created at scheduled intervals by sampling from the current top-500 most-used skills. For each task in a release, the benchmark materializes a controlled execution environment with fixed service fixtures, workspace state, and grading harnesses. When an agent attempts a task, it receives real or simulated API calls to these fixtures, and the grading layer can verify not just the final output but the sequence of actions taken—for example, whether the agent actually called the correct API with the correct parameters, or tried multiple approaches before arriving at the answer. Each release is timestamped and frozen, so results from 2026-Q1 remain comparable to 2026-Q2, while the underlying signal layer may have shifted, allowing researchers to study how agent generalization tracks real-world workflow evolution.
Production Impact
For teams building LLM agents in production, this benchmark provides a more realistic evaluation framework that directly reflects the tasks your users actually perform rather than a curated toy problem set. Instead of achieving 95% on a static benchmark that becomes irrelevant in six months, you can now track whether your agent maintains performance against the distribution of skills actually used in your customer base, and benchmark updates will surface when your agent's performance degrades on newly popular workflows. The intermediate-step grading is critical for production safety: it catches agents that produce correct final outputs through hallucinated actions, which current benchmarks would pass but real services would reject. Integration requires instrumenting your agent execution layer to expose action traces and connecting to the live signal provider, adding moderate engineering overhead but enabling continuous, realistic evaluation. The trade-off is that you must accept the benchmark's materialized service fixtures as representative of real behavior—mismatches in fixture fidelity (e.g., API behavior that differs from production) will give misleading results.
Limitations and When Not to Use This
The paper's approach assumes that public workflow-demand signals accurately reflect what your specific user base needs; if your agents serve a different task distribution than the ClawHub Top-500, the benchmark may not predict real-world performance. Materialized service fixtures cannot capture the full complexity of real APIs—error modes, rate limits, state mutations, and edge cases will be simplified, potentially allowing agents to succeed on Claw-Eval-Live but fail in production. The framework also does not address how to handle tasks with multiple valid solution paths or subjective correctness criteria (e.g., 'improve this email'); the grading logic appears to assume deterministic, verifiable task completion. Finally, the paper is incomplete in the abstract—it cuts off mid-sentence during the grading discussion—so key details about how grading works, how the Top-500 is selected and updated, and what the computational cost of maintaining this infrastructure is remain undescribed.
Research Context
This work builds on a recent surge in agentic AI benchmarking (SWE-bench for code agents, WebArena for web browsing, WorkArena for spreadsheets) but identifies a fundamental limitation they all share: static task sets diverge from real-world demand over time. Claw-Eval-Live extends the live-benchmark idea from domains like ImageNet-ReaL or Hugging Face's leaderboards to workflow agents, where task evolution is driven by public adoption signals rather than researcher intuition. The separation of signal freshness from reproducibility is conceptually related to data versioning and MLOps practices (DVC, Quilt), but applied at the evaluation level rather than the training level. The work opens a research direction toward continuous benchmark maintenance for agentic systems, which will likely become standard practice as LLM agents move into production and the pace of capability changes accelerates.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::