Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces
| Authors | Jiawei Chen et al. |
| Year | 2026 |
| HF Upvotes | 12 |
| arXiv | 2604.08362 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
The emergence of Large Language Models (LLMs) has illuminated the potential for a general-purpose user simulator. However, existing benchmarks remain constrained to isolated scenarios, narrow action spaces, or synthetic data, failing to capture the holistic nature of authentic human behavior. To bridge this gap, we introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data, integrating long-horizon, cross-scenario, and heterogeneous behavioral patterns into a unified framework. Based on this benchmark, we first provide empirical evidence that previous datasets with isolated scenarios suffer from tunnel vision, whereas real-world decision-making relies on long-term, cross-scenario causal chains. Extensive evaluations of state-of-the-art LLMs reveal that current models struggle to accurately simulate these complex behaviors, with performance plateauing even as context windows expand. Crucially, a systematic comparison between simulated and authentic behaviors uncovers a fundamental structural bias: LLMs tend to converge toward a positive average person, exhibiting hyper-activity, persona homogenization, and a Utopian bias. This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
Engineering Breakdown
Plain English
This paper introduces OmniBehavior, a new benchmark for evaluating how well large language models can simulate realistic user behavior. Unlike existing benchmarks that test users in isolated scenarios with limited actions, OmniBehavior is built entirely from real-world data and captures long-horizon decision-making where users' choices depend on past events and span multiple different contexts. The authors demonstrate that previous isolated-scenario datasets fail to capture how real humans actually make decisions, which involve complex causal chains across time and situations. Evaluations show that state-of-the-art LLMs struggle with this more realistic, connected behavioral simulation.
Core Technical Contribution
The core novelty is the first large-scale user simulation benchmark constructed from authentic real-world behavioral data rather than synthetic or constrained scenarios. The authors identified and formalized a critical gap in existing work: that isolated-scenario datasets create 'tunnel vision' and miss the causal dependencies that drive authentic human decision-making across time and context. The benchmark explicitly integrates three dimensions that prior work ignored: long-horizon sequences (decisions depend on history), cross-scenario patterns (behavior transfers across different contexts), and heterogeneous actions (diverse, realistic choice spaces). This enables rigorous evaluation of whether LLMs can actually model human behavior in the messy, interconnected way it happens in practice.
How It Works
OmniBehavior constructs a unified simulation framework by extracting behavioral patterns from large collections of real-world interaction logs across multiple domains and contexts. The benchmark represents user behavior as sequences of decisions where each action depends on prior history and context from other scenarios—the key insight being that humans don't compartmentalize decisions in isolation. The evaluation methodology prompts LLMs to predict or generate user actions given a context window of historical behavior and task descriptions, measuring accuracy against ground-truth behavioral outcomes from the real data. The framework explicitly models causal chains by including dependencies where decisions in one scenario influence or relate to decisions in another, forcing models to reason about longer-term behavioral consistency. Heterogeneous action spaces are preserved from real-world data rather than simplified, meaning the model must handle the full complexity of what users actually do. Baseline evaluations run state-of-the-art LLMs against this benchmark to quantify performance gaps and identify which behavioral patterns (long-horizon, cross-scenario, heterogeneous) are hardest to simulate.
Production Impact
For engineers building user simulation systems—recommendation engines, testing platforms, user modeling services—this benchmark provides a rigorous, real-world evaluation standard that exposes failures in deployed models that isolated benchmarks would miss. If you adopt OmniBehavior evaluation, you'll likely discover that your LLM-based user simulators perform worse than in-house tests suggested, because real behavior involves memory effects and context dependencies that synthetic data doesn't capture. Production systems using this work would need to retrain or fine-tune models on cross-scenario behavioral data and implement mechanisms to track long-horizon context (likely increasing inference latency and memory requirements). The trade-off is clear: more expensive evaluation and potentially higher compute costs to support longer context windows, but in return you get simulators that actually work on real user behavior rather than toy problems. For A/B testing platforms, fraud detection, and personalization systems, the ability to accurately simulate authentic user sequences dramatically improves the quality of offline evaluation before live deployment.
Limitations and When Not to Use This
The paper doesn't fully address how to handle distributional shift—real-world data reflects specific user populations at specific times, and models trained on this data may fail when user behavior changes or in new domains. The work assumes that LLMs can be meaningfully evaluated on user simulation tasks, but it doesn't solve the fundamental challenge of grounding language model reasoning in physical or interaction outcomes; a model might generate plausible text that describes unrealistic behavior. Computational cost is underexplored: evaluating LLMs on long-horizon sequences with rich context likely requires substantial inference compute, and the paper doesn't quantify latency or memory overhead compared to simpler simulation baselines. The benchmark is also constrained to domains and scenarios represented in the real-world data collected; generalization to completely new domains or user populations remains an open question. Finally, the paper lacks analysis of failure modes—when and why do LLMs break down?—which is critical for practitioners deciding whether to trust these simulators in high-stakes applications.
Research Context
This work advances the broader research agenda around using LLMs as general-purpose simulators, building on prior work that explored LLM-based interaction models but in restricted settings. It directly critiques and moves beyond existing benchmarks like dialogue datasets or task-specific interaction simulators that operate in narrow, isolated scenarios. The contribution parallels recent efforts in long-horizon planning and reasoning with LLMs, showing that user simulation requires similar multi-step causal reasoning. OmniBehavior opens a research direction toward more realistic, integrated evaluation of behavioral models and positions real-world data-driven benchmarking as the standard for this space, likely influencing how future user simulation and agent modeling work is evaluated.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::