Many-Tier Instruction Hierarchy in LLM Agents
| Authors | Jingyu Zhang et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.09443 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
Large language model agents receive instructions from many sources-system messages, user prompts, tool outputs, and more-each carrying different levels of trust and authority. When these instructions conflict, models must reliably follow the highest-privilege instruction to remain safe and effective. The dominant paradigm, instruction hierarchy (IH), assumes a fixed, small set of privilege levels (typically fewer than five) defined by rigid role labels (e.g., system > user). This is inadequate for real-world agentic settings, where conflicts can arise across far more sources and contexts. In this work, we propose Many-Tier Instruction Hierarchy (ManyIH), a paradigm for resolving instruction conflicts among instructions with arbitrarily many privilege levels. We introduce ManyIH-Bench, the first benchmark for ManyIH. ManyIH-Bench requires models to navigate up to 12 levels of conflicting instructions with varying privileges, comprising 853 agentic tasks (427 coding and 426 instruction-following). ManyIH-Bench composes constraints developed by LLMs and verified by humans to create realistic and difficult test cases spanning 46 real-world agents. Our experiments show that even the current frontier models perform poorly (~40% accuracy) when instruction conflict scales. This work underscores the urgent need for methods that explicitly target fine-grained, scalable instruction conflict resolution in agentic settings.
Engineering Breakdown
Plain English
This paper addresses a critical safety problem in LLM agents: when instructions come from multiple sources (system prompts, user inputs, tool outputs, etc.) and conflict, the model must reliably follow the highest-authority instruction. Current approaches use rigid instruction hierarchies with only 2-5 fixed privilege levels (e.g., system > user > tool), which is inadequate for real-world agentic systems where instruction sources and trust levels are far more granular and context-dependent. The authors propose Many-Tier Instruction Hierarchy (ManyIH), a new paradigm that handles arbitrarily many privilege levels, and introduce ManyIH-Bench to evaluate models' ability to resolve these complex instruction conflicts across diverse scenarios.
Core Technical Contribution
The core innovation is moving from fixed, small-cardinality instruction hierarchies (typically 2-5 levels) to a flexible framework that can represent and resolve conflicts among arbitrarily many privilege levels. Rather than hardcoding role-based rules ("system always wins"), ManyIH likely uses learned or specified privilege orderings that can be dynamically configured based on context, source reliability, and application requirements. This is a fundamental shift in how agents model authority and trust in instruction resolution—instead of a global static ordering, it enables domain-specific, fine-grained privilege models that better reflect real deployment scenarios where instruction sources vary widely in trustworthiness.
How It Works
The system accepts instructions from multiple sources, each tagged with metadata about its origin (system message, user prompt, tool output, etc.) and potentially carrying explicit or implicit privilege indicators. When instruction conflicts are detected—e.g., a user asks the agent to ignore safety constraints while the system prompt forbids this—the ManyIH framework compares the privilege levels of competing instructions using a learned or predefined ordering function. Rather than a simple linear ranking, ManyIH likely models privilege as a multi-dimensional or context-dependent property, allowing the agent to reason about partial orderings and edge cases. The framework outputs a resolved instruction that respects the highest-privilege directive, along with confidence scores or explanations of the resolution process. ManyIH-Bench provides diverse test cases where instruction sources conflict across many different privilege tiers, enabling evaluation of both accuracy in hierarchy resolution and robustness to adversarial instruction injection.
Production Impact
For teams building agentic systems, ManyIH directly improves safety and controllability by providing a principled way to manage instruction conflicts beyond simple role-based rules. In practice, this means you can deploy agents in environments where instructions come from heterogeneous sources (user, API, database, real-time monitoring systems) without having to manually engineer a fixed hierarchy—instead, you specify privilege rules declaratively and let the model reason about precedence. The trade-off is added complexity: you need to annotate instructions with privilege metadata, define or train a privilege ordering function, and potentially add guardrails to ensure the model doesn't learn exploitable privilege orderings during fine-tuning. Latency impact is minimal if privilege resolution is deterministic, but if it requires model inference to rank instructions, you incur additional forward-pass cost. Integration requires refactoring instruction pipelines to tag sources explicitly and validation logic to catch conflicts before they reach the agent.
Limitations and When Not to Use This
The paper's evaluation depends heavily on ManyIH-Bench, which may not capture all real-world instruction conflict scenarios—adversarial users may discover edge cases where the privilege model breaks down or can be manipulated. The approach assumes instructions can be cleanly tagged with their source and privilege level, but in practice many instructions are implicit or multi-sourced (e.g., a user requesting behavior that contradicts a learned constraint), making privilege assignment ambiguous. The paper likely does not fully address cases where privilege itself should be context-dependent or time-varying (e.g., a user's authority might depend on their role, the task, or recent validation), which requires either more sophisticated modeling or manual policy specification. Additionally, there is no clear analysis of robustness to adversarial instruction crafting where attackers try to confuse the privilege model or exploit misalignment between intended and learned hierarchies.
Research Context
This work builds on a growing body of research on instruction-following safety in LLMs, including studies on prompt injection, jailbreaking, and instruction conflict resolution. It extends beyond prior work like Constitutional AI and RLHF-based safety training by focusing specifically on the mechanical problem of resolving conflicting instructions rather than preventing unsafe instructions from being generated. The Many-Tier framing is a response to real-world deployment challenges where the simplistic system-vs-user binary breaks down, and sets up a new evaluation frontier (ManyIH-Bench) that will likely become a standard benchmark for measuring instruction robustness. This opens research directions in learning privilege orderings from data, reasoning about partial orderings of instructions, and designing agents that can explain their instruction resolution decisions to users.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::