Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation
| Authors | Garvin Kruthof |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.28031 |
| Download | |
| Categories | cs.CL |
Abstract
When researchers iteratively refine ideas with large language models, do the models preserve fidelity to the original objective? We introduce DriftBench, a benchmark for evaluating constraint adherence in multi-turn LLM-assisted scientific ideation. Across 2,146 scored benchmark runs spanning seven models from five providers (including two open-weight), four interaction conditions, and 38 research briefs from 24 scientific domains, we find that iterative pressure reliably increases structural complexity and often reduces adherence to original constraints. A restatement probe reveals a dissociation between declarative recall and behavioral adherence, as models accurately restate constraints they simultaneously violate. The knows-but-violates (KBV) rate, measuring constraint non-compliance despite preserved recall, ranges from 8% to 99% across models. Structured checkpointing partially reduces KBV rates but does not close the dissociation, and complexity inflation persists. Human validation against blind raters confirms that the LLM judge under-detects constraint violations, making reported constraint adherence scores conservative. Sensitivity analyses confirm the findings are robust to temperature (0.7 vs.\ 1.0) and pressure type (novelty vs.\ rigor). We release all briefs, prompts, rubrics, transcripts, and scores as an open benchmark.
Engineering Breakdown
Plain English
This paper introduces DriftBench, a benchmark that measures whether large language models maintain fidelity to their original objectives when researchers iteratively refine ideas with them over multiple turns. The authors evaluated 2,146 benchmark runs across seven models from five providers (including two open-weight models) using 38 research briefs from 24 scientific domains, finding that iterative refinement reliably increases structural complexity but often degrades adherence to original constraints. A key finding is the 'knows-but-violates' (KBV) phenomenon: models can accurately restate constraints in direct questions but simultaneously violate those same constraints during collaborative refinement, revealing a dissociation between declarative recall and actual behavioral compliance.
Core Technical Contribution
The core contribution is the identification and quantification of constraint drift in multi-turn LLM-assisted ideation—specifically the KBV rate metric that measures when models violate constraints they can explicitly recall. This is the first systematic benchmark designed to evaluate this failure mode across different models and interaction conditions, moving beyond single-turn evaluations to measure realistic collaborative scenarios where researchers iteratively refine ideas with LLMs. The restatement probe technique is novel: it separates declarative knowledge (can the model repeat the constraint?) from behavioral adherence (does the model follow it?), exposing a previously undocumented gap in LLM reasoning that persists even when models 'know' what they should do.
How It Works
DriftBench operates as a multi-turn evaluation framework where a research brief containing explicit constraints is presented to an LLM, followed by iterative refinement prompts that pressure the model to elaborate, improve, or extend the initial response. At each turn, the benchmark measures both the structural complexity of outputs and adherence to the original constraints through automated scoring. The restatement probe is inserted mid-interaction: the model is asked to explicitly restate the constraints, then immediately asked to continue refining the work—creating a direct comparison point between what the model claims to remember and what it actually does. Across the interaction conditions (presumably varying dialogue styles, constraint framing, or pressure patterns), the authors collect a dataset of constraint violations and accuracy metrics, then aggregate results by model, domain, and interaction type to identify systematic patterns of drift.
Production Impact
This work directly impacts any production system using LLMs for collaborative knowledge work—research assistance, code generation, proposal writing, or domain-specific ideation. The KBV phenomenon suggests that guardrails, constraints, or system prompts may provide a false sense of safety: engineers cannot assume that because a model can restate a constraint, it will follow it under iterative pressure. Teams building LLM-assisted products should expect to see gradual divergence from user intent during multi-turn workflows and should implement explicit constraint verification mechanisms (e.g., periodic constraint re-checks, structured output validation, or intervention alerts when drift is detected). The trade-off is additional computational overhead per interaction and more complex logging/monitoring infrastructure, but the cost of undetected constraint violation in scientific, medical, or high-stakes domains is much higher.
Limitations and When Not to Use This
The paper evaluates only seven models, and it's unclear how well the findings generalize to newer architectures or proprietary frontier models not included in the study. The benchmark focuses on research ideation, so the transferability to other domains (customer service, content moderation, creative generation where constraint drift might be desirable) is limited. The restatement probe assumes that explicit restating is a reliable proxy for knowledge, but doesn't address whether fine-tuning, instruction engineering, or architectural changes could actually prevent the KBV phenomenon rather than merely detect it. Finally, the paper doesn't provide actionable remediation strategies—it diagnoses the problem but doesn't test solutions like dynamic constraint reinforcement, explicit penalty mechanisms, or multi-turn constraint verification during inference.
Research Context
This work builds on the growing body of research into LLM reliability, consistency, and alignment under realistic use conditions, moving beyond single-prompt benchmarks to multi-turn interactive scenarios that reflect actual product usage. It contributes to the safety and interpretability literature by identifying a specific failure mode (dissociation between recall and behavior) that sits between traditional prompt injection attacks and genuine reasoning failures. DriftBench itself becomes a reference benchmark in the evaluation space, likely to be used by other researchers testing new prompting techniques, fine-tuning methods, or architectural modifications aimed at improving constraint adherence. The work also opens research directions in understanding why this dissociation exists—whether it's a fundamental limitation of transformer attention patterns, a training data artifact, or an optimization issue addressable through better objectives.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::