MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games
| Authors | Jacob Eisenstein et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.24188 |
| Download | |
| Categories | cs.CL, cs.LG |
Abstract
We present a scalable methodology for evaluating language models in multi-turn interactions, using a suite of collaborative games that require effective communication about private information. This enables an interactive scaling analysis, in which a fixed token budget is divided over a variable number of turns. We find that in many cases, language models are unable to use interactive collaboration to improve over the non-interactive baseline scenario in which one agent attempts to summarize its information and the other agent immediately acts -- despite substantial headroom. This suggests that state-of-the-art models still suffer from significant weaknesses in planning and executing multi-turn collaborative conversations. We analyze the linguistic features of these dialogues, assessing the roles of sycophancy, information density, and discourse coherence. While there is no single linguistic explanation for the collaborative weaknesses of contemporary language models, we note that humans achieve comparable task success at superior token efficiency by producing dialogues that are more coherent than those produced by most language models. The proactive management of private information is a defining feature of real-world communication, and we hope that MT-PingEval will drive further work towards improving this capability.
Engineering Breakdown
Plain English
This paper introduces MT-PingEval, a methodology for evaluating how well language models collaborate across multiple turns of conversation while managing private information that only one agent knows. The researchers created a suite of collaborative games where two agents must communicate effectively to solve tasks, then measured performance when they divided a fixed computational budget across varying numbers of conversation turns. They found a striking result: despite having substantial room for improvement, state-of-the-art models fail to leverage multi-turn interaction effectively—they perform only marginally better than a simple baseline where one agent dumps all information at once and the other immediately acts. This reveals fundamental weaknesses in how current LLMs plan and execute collaborative dialogues over multiple exchanges.
Core Technical Contribution
The core contribution is a scalable, interactive evaluation framework that systematically measures language model performance under constrained token budgets distributed across variable conversation lengths. Rather than just testing if models can talk more, the authors isolate the strategic value of multi-turn collaboration by fixing total compute and measuring whether agents improve performance by spreading communication across multiple turns—a more realistic constraint than unlimited dialogue. The private information games are the key innovation: they force agents to communicate strategically about asymmetric knowledge, making it impossible to succeed without genuine coordination. This framework reveals that current models lack the planning and execution capabilities to exploit multi-turn interaction, which prior benchmarks didn't surface because they either allowed unlimited dialogue or didn't measure efficiency-constrained performance.
How It Works
The system works as follows: two agents are given a collaborative task where each has private information the other needs. In the interactive condition, they have multiple turns to exchange messages, with a fixed total token budget divided across turns (e.g., 1000 tokens split into 2, 5, or 10 turns). In the baseline condition, one agent gets a single turn to communicate everything it knows, then the other agent acts immediately with the same total budget. The evaluation measures task success rate and efficiency metrics (tokens used per turn, information transfer rate, goal achievement). The authors analyze dialogue transcripts to identify linguistic patterns—tracking whether agents ask clarifying questions, build shared context, or just dump information—to understand why multi-turn collaboration often fails. By varying the number of turns and measuring performance against the non-interactive baseline, they can quantify the strategic value (or lack thereof) that additional rounds of communication actually provide.
Production Impact
For engineers building multi-turn conversational AI systems, this work highlights a critical blind spot: your models may not be getting genuine benefit from multiple conversation rounds, even though the code appears to support it. If you're deploying collaborative agents (customer service systems working with human agents, AI assistants coordinating with other AI systems, or humans and AI planning together), you need MT-PingEval-style evaluation to verify that your multi-turn setup actually improves performance over simpler alternatives—otherwise you're burning latency and tokens for no gain. The practical implication is that before scaling up dialogue systems, you should establish a private-information coordination baseline and measure whether additional turns improve over it; this might reveal that you need architectural changes (better planning mechanisms, explicit context management, or turn-taking protocols) rather than just longer conversations. The token-budget framing is particularly relevant for production: it forces you to think about the real trade-off between conversation depth and cost, rather than assuming more turns are always better.
Limitations and When Not to Use This
The paper focuses on synthetic collaborative games with clear success criteria, which may not capture the full complexity of real-world multi-turn dialogue where success is ambiguous or negotiated. The evaluation assumes that private information games are representative of useful collaboration patterns, but many real applications involve different interaction structures (hierarchical decision-making, consensus-building, adversarial negotiation) that might benefit differently from multi-turn interaction. The analysis doesn't explore whether architectural changes to the models themselves (e.g., explicit planning layers, hierarchical attention, or memory mechanisms) could unlock better multi-turn performance, so it remains unclear whether the failures are fundamental to current LLM architectures or just training/prompting issues. Additionally, the paper doesn't address scaling—it's unclear whether larger models or different training procedures (e.g., dialogue-specific fine-tuning or RLHF with multi-turn objectives) would change the conclusions, which limits actionability for teams with different model scales or training capabilities.
Research Context
This work builds on the long tradition of evaluating AI agents through game-theoretic frameworks and collaborative tasks (prior work on dialogue evaluation, reference games, and multi-agent coordination), but extends it with a computational efficiency lens that's novel for language models. It relates to recent research on scaling laws and token efficiency, asking not just whether more compute helps but whether the structure of that compute (number of turns) matters for collaboration. The paper advances the evaluation methodology landscape for LLMs, similar to how benchmarks like MMLU and HumanEval measure different capabilities, by introducing a new dimension: interactive scaling under token constraints. This opens a research direction into multi-turn planning and communication in LLMs, potentially influencing how future models are trained to handle sequential collaboration and whether future architectures should include explicit mechanisms for long-horizon planning across dialogue turns.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::