A Systematic Security Evaluation of OpenClaw and Its Variants
| Authors | Yuhang Wang et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2604.03131 |
| Download | |
| Categories | cs.CR, cs.AI |
Abstract
Tool-augmented AI agents substantially extend the practical capabilities of large language models, but they also introduce security risks that cannot be identified through model-only evaluation. In this paper, we present a systematic security assessment of six representative OpenClaw-series agent frameworks, namely OpenClaw, AutoClaw, QClaw, KimiClaw, MaxClaw, and ArkClaw, under multiple backbone models. To support this study, we construct a benchmark of 205 test cases covering representative attack behaviors across the full agent execution lifecycle, enabling unified evaluation of risk exposure at both the framework and model levels. Our results show that all evaluated agents exhibit substantial security vulnerabilities, and that agentized systems are significantly riskier than their underlying models used in isolation. In particular, reconnaissance and discovery behaviors emerge as the most common weaknesses, while different frameworks expose distinct high-risk profiles, including credential leakage, lateral movement, privilege escalation, and resource development. These findings indicate that the security of modern agent systems is shaped not only by the safety properties of the backbone model, but also by the coupling among model capability, tool use, multi-step planning, and runtime orchestration. We further show that once an agent is granted execution capability and persistent runtime context, weaknesses arising in early stages can be amplified into concrete system-level failures. Overall, our study highlights the need to move beyond prompt-level safeguards toward lifecycle-wide security governance for intelligent agent frameworks.
Engineering Breakdown
Plain English
This paper presents a systematic security evaluation of six OpenClaw-series agent frameworks (OpenClaw, AutoClaw, QClaw, KimiClaw, MaxClaw, and ArkClaw) that augment large language models with tool-use capabilities. The authors constructed a benchmark of 205 test cases covering representative attack behaviors across the full agent execution lifecycle to identify vulnerabilities that wouldn't be caught by evaluating models in isolation. The core finding is stark: all evaluated agents exhibit substantial security vulnerabilities, and agentized systems are significantly riskier than unaugmented language models. This work demonstrates that the integration of tool-calling interfaces, external APIs, and multi-step reasoning creates a substantially larger attack surface than previously understood.
Core Technical Contribution
The primary novelty is the first systematic, unified security benchmark for tool-augmented agent frameworks, moving beyond model-only evaluation to measure risk exposure at both the framework and model levels simultaneously. Rather than proposing a new defense mechanism, the authors' contribution is methodological: they constructed 205 carefully designed test cases that exercise the full agent lifecycle (planning, tool invocation, output handling, state management) rather than just the underlying LLM. This is fundamentally different from prior work that evaluated LLM safety in isolation, because agentized systems introduce new attack vectors through tool interaction, API chaining, and framework-level execution logic. The benchmark covers multiple backbone models and multiple framework implementations, enabling comparative analysis that reveals which vulnerabilities are model-dependent versus framework-specific.
How It Works
The evaluation methodology operates across three layers: (1) the LLM backbone, (2) the agent framework orchestration logic, and (3) the tool execution environment. For each of the six frameworks, the authors execute the 205 test cases and observe whether the agent can be manipulated into unsafe tool invocations, unauthorized API calls, information leakage through tool outputs, or incorrect state management during multi-step reasoning. The test cases are categorized by attack type (e.g., prompt injection, tool argument poisoning, unauthorized privilege escalation, output extraction) and execution phase (planning phase attacks, tool invocation attacks, feedback processing attacks). The benchmark records both whether an attack succeeds and at which layer it succeeds—allowing researchers to distinguish between model vulnerabilities and framework-level design flaws. This holistic instrumentation enables the paper to measure not just that attacks work, but why they work and what component failed to defend against them.
Production Impact
For teams deploying LLM agents in production, this paper provides a concrete risk assessment framework and benchmark that should inform architecture decisions before deployment rather than after incidents occur. Specifically, engineers should understand that tool-augmented agents require defense layers beyond standard LLM safety measures—including sandboxed tool execution, strict API permission boundaries, input validation at tool invocation time, and explicit state isolation between tool calls. The 205-case benchmark can be adapted to evaluate custom agent frameworks before production rollout, helping teams identify vulnerability classes early in development. The trade-off is clear: agents are more capable (they can interact with external systems) but substantially riskier, so adopting this evaluation methodology requires building robust guardrails and acceptance testing infrastructure. Organizations deploying agents should expect to invest in security evaluation effort proportional to the number of tools and APIs the agent can access, as each new integration expands the attack surface.
Limitations and When Not to Use This
The paper evaluates only six specific OpenClaw variants, leaving open the question of how findings generalize to other agent frameworks (ReAct, multi-agent systems, hierarchical agents) or to custom in-house implementations that may have different architectures. The threat model assumes adversaries can craft malicious inputs or poison tool outputs, but does not address supply-chain attacks (compromised model weights), sophisticated gradient-based attacks, or adversaries with internal access to the agent infrastructure. The benchmark is static—205 fixed test cases—so it may not capture novel attack strategies discovered after publication or attacks that require adaptive, multi-turn exploitation. Additionally, the paper does not propose concrete mitigations or defenses, so readers get a clear picture of the problem landscape but limited guidance on how to fix each vulnerability class; follow-up work on agent-specific defense mechanisms would be valuable.
Research Context
This work builds on a growing body of AI safety research that has extended from model-only evaluation (testing LLMs in isolation) to systems-level evaluation (testing models embedded in larger applications). It complements prior work on adversarial robustness of language models and prompt injection attacks, but extends the threat model to include framework-level logic and tool-execution semantics. The paper contributes to the agent security research direction that includes work on tool use in language models, multi-agent coordination safety, and autonomous system security. The benchmark itself is a research artifact that will likely become a standard evaluation tool in the agent security community, similar to how other security benchmarks (e.g., for adversarial examples) have become infrastructure for the field.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::