GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
| Authors | Mingyu Ouyang et al. |
| Year | 2026 |
| HF Upvotes | 13 |
| arXiv | 2604.07429 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse feedback, and irreversible mistakes. Video games offer an ideal testbed with rich visual observations and closed-loop interaction, demanding fine-grained perception, long-horizon planning, and precise control. However, systematically evaluating these capabilities is currently hindered by heterogeneous action interfaces and heuristic verification. To this end, we introduce GameWorld, a benchmark designed for standardized and verifiable evaluation of MLLMs as generalist game agents in browser environments. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space via deterministic Semantic Action Parsing. GameWorld contains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games. Extensive experiments of repeated full-benchmark reruns demonstrate the robustness of the benchmark, while further studies on real-time interaction, context-memory sensitivity, and action validity expose more challenges ahead for game agents. Together, by offering a standardized, verifiable, and reproducible evaluation framework, GameWorld lays a robust foundation for advancing research on multimodal game agents and beyond. The project page is at https://gameworld-bench.github.io.
Engineering Breakdown
Plain English
This paper introduces GameWorld, a standardized benchmark for evaluating multimodal large language models (MLLMs) as game-playing agents in browser environments. The authors address a critical gap in MLLM evaluation by creating a platform with heterogeneous action interfaces—both computer-use agents that emit raw keyboard/mouse controls and generalist multimodal agents—enabling systematic assessment of fine-grained perception, long-horizon planning, and precise control. Current MLLM agents struggle with latency, sparse feedback, and irreversible mistakes, which video games expose naturally through rich visual observations and closed-loop interaction loops. GameWorld provides verifiable evaluation mechanisms rather than heuristic approaches, creating a controlled testbed where performance on game tasks directly measures embodied AI capabilities.
Core Technical Contribution
The paper's core innovation is GameWorld itself—a benchmark architecture that standardizes evaluation of MLLMs across heterogeneous action spaces and game environments in browser settings. Unlike prior ad-hoc evaluations, GameWorld introduces dual agent interfaces: computer-use agents that directly control keyboards and mice (measuring low-level motor control), and generalist multimodal agents (measuring higher-level reasoning). The key technical contribution is the verifiable evaluation framework—games provide objective ground truth on task completion and failure modes, eliminating subjective heuristics used in previous MLLM assessments. This represents a systematic shift from evaluating language models on static benchmarks to evaluating embodied agents on interactive, reversible, and measurable tasks.
How It Works
GameWorld functions as a distributed evaluation platform where MLLMs receive visual observations (screenshot frames from browser games) as input and must generate appropriate actions (keyboard keys, mouse coordinates, click commands) as output. The system maintains a closed-loop interaction pipeline: the MLLM processes the current game state via vision encoding, predicts the next action through the language model backbone, executes that action in the game engine, receives visual feedback, and repeats. For computer-use agents, raw pixel-level coordinates and keystrokes are emitted directly, requiring the model to ground spatial reasoning in screen coordinates. For generalist agents, actions may be higher-level API calls or structured commands that the benchmark translates to controls. The framework logs all interactions, enabling post-hoc analysis of success rates, error patterns, and latency metrics; game state checksums and replay mechanisms ensure verifiability and reproducibility across multiple evaluation runs.
Production Impact
Engineers building embodied AI systems can adopt GameWorld's evaluation methodology to benchmark MLLM agents before deploying them to real-world robotics or control tasks, since game environments provide safe, reversible feedback at massive scale. The dual agent interface design directly applies to production systems: computer-use agents map to RPA (robotic process automation) and desktop automation tasks, while generalist agents scale to higher-level planning in structured environments like warehouse robotics or autonomous navigation. The standardized action space and verifiable ground truth reduce evaluation friction—teams no longer need custom evaluation harnesses for each new MLLM checkpoint. Key trade-offs include the computational overhead of rendering browser games and running inference loops (likely 500ms–5s latency per action depending on model size), and the gap between game environments and real-world visual variability, domain-specific physics, and irreversible consequences. For teams deploying MLLMs in critical applications (surgical robotics, financial trading), the verifiable feedback and long-horizon planning evaluation in GameWorld would become a standard pre-deployment gate.
Limitations and When Not to Use This
The paper's scope is limited to browser-based games, which have simplified physics, predictable visual designs, and often discrete action spaces—real-world environments exhibit continuous dynamics, occlusions, and visual noise that game environments do not capture. The verifiability advantage of games (clear win/loss conditions) does not transfer to open-ended tasks like manipulation in unstructured homes or navigation in wilderness; the assumption that game performance predicts real-world embodied capability remains unvalidated. The paper does not address how to bridge the sim-to-real gap or calibrate game-based evaluation metrics to real-world success rates, leaving practitioners uncertain about deployment readiness. Additionally, the computational cost of running MLLM inference loops in real-time (especially for larger models like GPT-4V) may not reflect actual latency constraints in robotics systems with 10–100ms control cycles, potentially masking latency problems that only surface in real deployment.
Research Context
GameWorld builds on a growing recognition that static benchmark evaluation (MMLU, COCO captioning) fails to measure embodied AI capabilities like planning, error recovery, and grounded reasoning. The work follows in the tradition of interactive evaluation environments like TextWorld and ALFWorld, but extends them to multimodal agents and modern MLLMs at scale. It directly addresses limitations of prior MLLM evaluations (e.g., GPT-4V on object detection or spatial reasoning tasks) by moving from image-level understanding to task-level performance in dynamic environments. This positions GameWorld as a bridge between pure language-model benchmarks and full robotic simulation platforms (like Habitat or IsaacGym), offering a practical middle ground for rapid iteration on MLLM agents with verifiable metrics.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::