Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind
| Authors | Hanqi Xiao et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.11666 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
As large language models (LLMs) become the engine behind conversational systems, their ability to reason about the intentions and states of their dialogue partners (i.e., form and use a theory-of-mind, or ToM) becomes increasingly critical for safe interaction with potentially adversarial partners. We propose a novel privacy-themed ToM challenge, ToM for Steering Beliefs (ToM-SB), in which a defender must act as a Double Agent to steer the beliefs of an attacker with partial prior knowledge within a shared universe. To succeed on ToM-SB, the defender must engage with and form a ToM of the attacker, with a goal of fooling the attacker into believing they have succeeded in extracting sensitive information. We find that strong frontier models like Gemini3-Pro and GPT-5.4 struggle on ToM-SB, often failing to fool attackers in hard scenarios with partial attacker prior knowledge, even when prompted to reason about the attacker's beliefs (ToM prompting). To close this gap, we train models on ToM-SB to act as AI Double Agents using reinforcement learning, testing both fooling and ToM rewards. Notably, we find a bidirectionally emergent relationship between ToM and attacker-fooling: rewarding fooling success alone improves ToM, and rewarding ToM alone improves fooling. Across four attackers with different strengths, six defender methods, and both in-distribution and out-of-distribution (OOD) evaluation, we find that gains in ToM and attacker-fooling are well-correlated, highlighting belief modeling as a key driver of success on ToM-SB. AI Double Agents that combine both ToM and fooling rewards yield the strongest fooling and ToM performance, outperforming Gemini3-Pro and GPT-5.4 with ToM prompting on hard scenarios. We also show that ToM-SB and AI Double Agents can be extended to stronger attackers, demonstrating generalization to OOD settings and the upgradability of our task.
Engineering Breakdown
Plain English
This paper introduces ToM-SB (Theory-of-Mind for Steering Beliefs), a new adversarial challenge that tests whether large language models can reason about an attacker's mental state and beliefs to defend sensitive information. The setup is a game where a defender acts as a double agent trying to fool an attacker into thinking they've successfully extracted secrets, while the attacker has partial prior knowledge. The authors tested frontier models like Gemini 3-Pro and GPT-5.4 and found they struggle significantly with this task, suggesting that even the most capable LLMs lack robust theory-of-mind reasoning needed for adversarial interactions.
Core Technical Contribution
The core novelty is framing adversarial defense as a recursive theory-of-mind problem rather than a static information-hiding problem. Instead of simply blocking access or encrypting data, the defender must model the attacker's beliefs, knowledge state, and reasoning process, then strategically provide false information that aligns with the attacker's prior beliefs to make deception convincing. This is fundamentally different from prior security approaches because it requires the model to reason about what the opponent believes they know and how new information will update those beliefs, creating a multi-step inference chain. The paper operationalizes this as a structured game where success requires both defensive deception and accurate modeling of adversarial reasoning patterns.
How It Works
The ToM-SB challenge operates as a two-player adversarial game within a shared fictional universe where both players have incomplete information. The attacker has a goal (extract secret X) and prior knowledge about what the defender might do, while the defender knows what the attacker believes and can observe their queries. The defender's task is to craft responses that are consistent with the attacker's prior beliefs but steer them away from the true secret—this requires the model to form an explicit mental model of the attacker's reasoning process and predict how each response will update their beliefs. The evaluation measures success by whether the attacker can extract the actual secret after interacting with the defender, with the defender's goal being to make the attacker confident they succeeded while actually feeding them false information. The technical mechanism relies on the LLM's ability to reason about nested beliefs (defender knows that attacker believes X) and to maintain consistency across a multi-turn dialogue while executing a deceptive strategy.
Production Impact
For production systems, this work reveals a critical gap in LLM safety for adversarial scenarios where passive information access controls (rate limiting, filtering) are insufficient. Systems like conversational AI assistants in high-stakes domains (finance, healthcare, security) need to defend against sophisticated attackers who can reason about the system's behavior and may not explicitly ask for secrets—they may try to infer them through indirect questioning. Implementing ToM-based defenses would require fine-tuning models on adversarial reasoning examples and building explicit belief-tracking components into the dialogue pipeline, which adds computational overhead and complexity to inference (roughly 2-3x latency increase due to multi-turn planning and consistency checking). The practical trade-off is that systems would become significantly more robust against advanced persistent attacks but less responsive for benign users due to increased inference time and the need for adversarial reasoning on every query.
Limitations and When Not to Use This
The paper assumes a fully observable game state where the defender knows the attacker's prior beliefs, which is unrealistic in real-world scenarios where an attacker's knowledge is hidden or mixed (they may have correct and incorrect beliefs simultaneously). The evaluation is conducted in synthetic, controlled environments with explicit universe definitions, which may not transfer to open-domain conversations where the space of possible prior beliefs is unbounded. Current frontier models' poor performance suggests the approach may require significant amounts of adversarial training data that doesn't yet exist at scale, and the paper doesn't demonstrate whether scaling data or model size actually improves ToM-SB performance. Additionally, the work assumes dialogue-based interaction and may not apply to non-interactive attack vectors (prompt injection, jailbreaking through system prompts) that bypass the defender's ability to model and influence the attacker.
Research Context
This work builds on a growing literature in LLM interpretability and adversarial robustness, extending recent work on theory-of-mind benchmarks for language models (like ToMi and SOCIALIQA) into adversarial and deceptive reasoning domains. It relates to research on jailbreaking and prompt injection attacks but inverts the problem—instead of defending against known attack patterns, it asks whether models can proactively defend by reasoning about attacker psychology. The paper contributes to the evaluation landscape for safety-critical LLMs by proposing a new benchmark that tests a specific but important capability (adversarial theory-of-mind), similar to how adversarial robustness benchmarks revealed vulnerabilities in computer vision. This opens a research direction into whether LLMs can be trained to reason defensively about multi-step adversarial interactions, potentially through adversarial fine-tuning or constitutional approaches that encourage models to model and counter opponent reasoning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::