Do LLMs Benefit From Their Own Words?
| Authors | Jenny Y. Huang et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.24287 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
Multi-turn interactions with large language models typically retain the assistant's own past responses in the conversation history. In this work, we revisit this design choice by asking whether large language models benefit from conditioning on their own prior responses. Using in-the-wild, multi-turn conversations, we compare standard (full-context) prompting with a user-turn-only prompting approach that omits all previous assistant responses, across three open reasoning models and one state-of-the-art model. To our surprise, we find that removing prior assistant responses does not affect response quality on a large fraction of turns. Omitting assistant-side history can reduce cumulative context lengths by up to 10x. To explain this result, we find that multi-turn conversations consist of a substantial proportion (36.4%) of self-contained prompts, and that many follow-up prompts provide sufficient instruction to be answered using only the current user turn and prior user turns. When analyzing cases where user-turn-only prompting substantially outperforms full context, we identify instances of context pollution, in which models over-condition on their previous responses, introducing errors, hallucinations, or stylistic artifacts that propagate across turns. Motivated by these findings, we design a context-filtering approach that selectively omits assistant-side context. Our findings suggest that selectively omitting assistant history can improve response quality while reducing memory consumption.
Engineering Breakdown
Plain English
This paper challenges a fundamental design choice in multi-turn LLM conversations: whether models actually need to see their own previous responses in the conversation history. The authors tested this by comparing standard full-context prompting against a stripped-down approach that removes all assistant responses, keeping only user turns, across multiple state-of-the-art models using real in-the-wild conversations. They found that removing assistant-side history had minimal impact on response quality for a significant portion of turns while reducing cumulative context length by up to 10x. This discovery has immediate practical implications for reducing computational costs in deployed conversational systems.
Core Technical Contribution
The core insight is that LLMs do not meaningfully depend on their own historical outputs for many conversation turns, contradicting the standard assumption that full context is necessary for coherent multi-turn interaction. The authors' technical contribution is a systematic evaluation methodology using real conversation data to isolate the effect of assistant history versus user history, showing that context can be dramatically pruned without quality degradation. They provide empirical evidence across multiple model scales and architectures that the redundancy in conversation history is substantial, enabling a simple yet effective optimization. This reframes context window management as an efficiency problem with a surprisingly simple solution.
How It Works
The experimental approach is straightforward but rigorous: the authors take existing multi-turn conversations and create two versions—one with full context (user turns + assistant responses) and one with only user turns. They then prompt the model to generate the next response given the current user input, comparing output quality metrics between the two conditions. To explain why assistant history doesn't help, they analyze the structural properties of multi-turn conversations—examining factors like conversation length, turn distribution, and semantic overlap between consecutive exchanges. The key mechanism they identify is that users tend to ask self-contained questions or rephrase key information, reducing the model's dependence on prior model outputs for maintaining coherence.
Production Impact
This finding directly translates to significant cost savings in deployed conversational systems. By removing assistant history, engineers can reduce the context window required per request by up to 10x, which directly cuts GPU memory requirements, reduces inference latency, and lowers API costs for models charged per-token. In a system handling millions of conversations daily, this translates to material infrastructure savings—fewer GPUs needed, faster token generation, lower memory bandwidth bottlenecks. The approach is trivial to implement (simply filter conversation history before encoding) but requires careful A/B testing in your specific domain since conversation patterns vary—technical support may behave differently than creative writing tasks. Teams should validate on their actual conversation distribution before deploying, but this represents a near-zero-cost optimization to try first before expensive interventions like model distillation or quantization.
Limitations and When Not to Use This
The paper's findings may not generalize across all conversation types—technical support, coding assistance, or long-form reasoning tasks where the model's reasoning scaffolding matters might show different patterns than the general conversations analyzed. The research doesn't deeply explore failure modes or identify which specific conversation characteristics make assistant history critical, leaving practitioners without clear guidance on when this optimization is safe. The work assumes consistent user behavior (clear, self-contained turns) which breaks down in adversarial scenarios or when users deliberately reference model-generated content from previous turns. Additionally, the paper doesn't address the interaction with specific system prompts, few-shot examples, or fine-tuned models that might have learned to depend on their own outputs during training.
Research Context
This work sits at the intersection of prompt engineering optimization and context window management, building on prior work exploring what information in prompts actually influences model outputs (e.g., research on prompt tokens versus reasoning tokens). It relates to broader efficiency research in LLMs like context compression, retrieval-augmented generation, and token pruning, but uniquely focuses on the structural redundancy in conversation history rather than information-theoretic approaches. The findings align with emerging research questioning assumptions about what context matters in in-context learning, contributing to a more nuanced understanding of effective prompting. This opens research directions into automatic conversation history distillation, adaptive context selection strategies, and domain-specific conversation patterns that determine when full context is truly necessary.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::