Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw
| Authors | Zijun Wang et al. |
| Year | 2026 |
| HF Upvotes | 22 |
| arXiv | 2604.04759 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
OpenClaw, the most widely deployed personal AI agent in early 2026, operates with full local system access and integrates with sensitive services such as Gmail, Stripe, and the filesystem. While these broad privileges enable high levels of automation and powerful personalization, they also expose a substantial attack surface that existing sandboxed evaluations fail to capture. To address this gap, we present the first real-world safety evaluation of OpenClaw and introduce the CIK taxonomy, which unifies an agent's persistent state into three dimensions, i.e., Capability, Identity, and Knowledge, for safety analysis. Our evaluations cover 12 attack scenarios on a live OpenClaw instance across four backbone models (Claude Sonnet 4.5, Opus 4.6, Gemini 3.1 Pro, and GPT-5.4). The results show that poisoning any single CIK dimension increases the average attack success rate from 24.6% to 64-74%, with even the most robust model exhibiting more than a threefold increase over its baseline vulnerability. We further assess three CIK-aligned defense strategies alongside a file-protection mechanism; however, the strongest defense still yields a 63.8% success rate under Capability-targeted attacks, while file protection blocks 97% of malicious injections but also prevents legitimate updates. Taken together, these findings show that the vulnerabilities are inherent to the agent architecture, necessitating more systematic safeguards to secure personal AI agents. Our project page is https://ucsc-vlaa.github.io/CIK-Bench.
Engineering Breakdown
Plain English
This paper presents the first real-world safety evaluation of OpenClaw, a widely deployed personal AI agent that runs with full local system access and integrations to services like Gmail, Stripe, and the filesystem. The researchers discovered that existing sandboxed evaluations miss critical attack surfaces in production deployments. They introduced the CIK taxonomy—a framework that categorizes an agent's persistent state into three dimensions (Capability, Identity, Knowledge)—and tested 12 attack scenarios across four major LLM backbones (Claude Sonnet 4.5, Opus 4.6, Gemini 3.1 Pro, GPT-5.4). The evaluation revealed significant vulnerabilities that traditional test environments fail to capture, establishing a new benchmark for assessing agent safety in real-world conditions.
Core Technical Contribution
The core novelty is the CIK (Capability-Identity-Knowledge) taxonomy, which provides a systematic framework for decomposing and analyzing the persistent state space of AI agents operating with privileged system access. Unlike prior work that evaluates agents in controlled sandboxes, this paper introduces a methodology for real-world safety evaluation against active threat scenarios on live agent instances. The three-dimensional taxonomy unifies disparate security concerns—what an agent can do (Capability), who it believes it is (Identity), and what information it retains (Knowledge)—into a coherent model for safety analysis. This is the first rigorous attempt to map the specific attack surface created by agent-level privilege escalation and service integrations in production deployments.
How It Works
The researchers operate a live OpenClaw instance and systematically craft 12 attack scenarios targeting different combinations of the CIK dimensions. For each scenario, they measure how effectively the underlying LLM backbone (tested across four different models) can be manipulated to misuse its privileges—such as executing unauthorized filesystem operations, making unintended financial transactions via Stripe integration, or exfiltrating data from email. The evaluation protocol presents adversarial prompts and scenarios to the agent and observes whether it maintains safety constraints or violates them by leveraging its persistent state (stored credentials, session tokens, user context). The CIK taxonomy provides the structure: attacks targeting Capability test privilege misuse; Identity attacks test impersonation or context confusion; Knowledge attacks test whether sensitive state can be leaked or modified. Results are aggregated across the four model backbones to identify both model-specific vulnerabilities and universal failure modes in the agent deployment architecture.
Production Impact
For teams deploying personal AI agents with system-level access, this paper makes clear that sandbox-based safety testing is insufficient and that real-world evaluations must account for persistent state, integrated services, and accumulated credentials. Production deployments should implement runtime constraints beyond model-level safeguards—such as privileged action approval workflows, capability-scoping per service integration, and audit logging for sensitive operations. The CIK framework provides a concrete taxonomy that engineering teams can use to inventory their agent's attack surface: map all capabilities the agent holds (file operations, API calls, etc.), all identity contexts it operates under (user accounts, service credentials), and all knowledge it persists (conversation history, cached data). The trade-off is significant operational overhead—real-world safety validation requires maintaining live instances, simulating multi-model deployments, and continuously refreshing threat scenarios as new LLM versions emerge. However, the alternative—deploying agents without understanding their real-world failure modes—carries substantially higher risk of financial loss, data breach, or unauthorized system modification.
Limitations and When Not to Use This
The paper evaluates only four LLM backbones as of early 2026; as new models emerge with different architectures and training procedures, the specific vulnerability patterns may shift, requiring re-evaluation. The 12 attack scenarios, while representative, may not exhaust the full space of adversarial inputs an intelligent attacker could craft—particularly scenarios involving multi-step prompt injection, timing-based attacks, or social engineering that chains with agent behavior. The evaluation assumes the attacker has access to interact directly with the agent (user-level threat model) but does not deeply explore supply-chain attacks against the agent framework itself or compromises at the OS kernel level. Finally, the paper does not provide a complete remediation strategy or demonstrate that the vulnerabilities can be fully mitigated at the model level; it remains unclear whether the root cause is insufficient model alignment, inadequate capability-scoping architecture, or fundamental limitations of current LLM safety training methods.
Research Context
This work builds on a decade of AI safety research that has shifted from offline red-teaming in controlled labs toward real-world evaluation of deployed systems—extending prior work on prompt injection attacks, jailbreaking, and adversarial robustness into the agent deployment domain. It directly responds to the gap between academic safety benchmarks (which test isolated model behavior) and practical safety concerns in systems like autonomous code execution engines and financial transaction agents. The CIK taxonomy is inspired by classical security frameworks (capability-based security, identity management, information flow control) adapted for the specific constraints and affordances of LLM agents. This paper establishes a new evaluation standard that will likely influence future benchmarking efforts for agent-based systems, similar to how prior work on model robustness shifted evaluation practices across the field.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::