WorldMark: A Unified Benchmark Suite for Interactive Video World Models
| Authors | Xiaojie Xu et al. |
| Year | 2026 |
| HF Upvotes | 35 |
| arXiv | 2604.21686 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Interactive video generation models such as Genie, YUME, HY-World, and Matrix-Game are advancing rapidly, yet every model is evaluated on its own benchmark with private scenes and trajectories, making fair cross-model comparison impossible. Existing public benchmarks offer useful metrics such as trajectory error, aesthetic scores, and VLM-based judgments, but none supplies the standardized test conditions -- identical scenes, identical action sequences, and a unified control interface -- needed to make those metrics comparable across models with heterogeneous inputs. We introduce WorldMark, the first benchmark that provides such a common playing field for interactive Image-to-Video world models. WorldMark contributes: (1) a unified action-mapping layer that translates a shared WASD-style action vocabulary into each model's native control format, enabling apples-to-apples comparison across six major models on identical scenes and trajectories; (2) a hierarchical test suite of 500 evaluation cases covering first- and third-person viewpoints, photorealistic and stylized scenes, and three difficulty tiers from Easy to Hard spanning 20-60s; and (3) a modular evaluation toolkit for Visual Quality, Control Alignment, and World Consistency, designed so that researchers can reuse our standardized inputs while plugging in their own metrics as the field evolves. We will release all data, evaluation code, and model outputs to facilitate future research. Beyond offline metrics, we launch World Model Arena (warena.ai), an online platform where anyone can pit leading world models against each other in side-by-side battles and watch the live leaderboard.
Engineering Breakdown
Plain English
WorldMark is the first unified benchmark for comparing interactive video generation models like Genie, YUME, and HY-World, which currently cannot be fairly evaluated against each other because each model uses private test scenes and custom control schemes. The paper identifies a critical gap: existing metrics (trajectory error, aesthetic scores, VLM judgments) exist but are incomparable across models due to heterogeneous input interfaces and non-standardized test conditions. WorldMark solves this by providing identical scenes, identical action sequences, and a unified control layer (a WASD-style action-mapping interface) that all models can use. This enables apples-to-apples comparison for the first time, establishing common ground for measuring progress in interactive world models.
Core Technical Contribution
The core innovation is a unified action-mapping abstraction layer that translates a standardized WASD keyboard input format into model-specific control representations, decoupling the benchmark's test conditions from each model's heterogeneous input requirements. Rather than forcing models to accept the same input format (which is impractical since models were designed with different APIs), WorldMark implements an adapter pattern that normalizes outputs and preserves semantics across diverse architectures. This is paired with a shared dataset of curated scenes and action trajectories that serve as the common test bed. The technical contribution is architectural rather than algorithmic—it's about standardizing the evaluation substrate, not inventing new model components, which makes it practical and broadly applicable.
How It Works
The system works in three stages: (1) Input normalization—a user or test harness provides WASD commands and discrete actions (jump, interact, etc.) to WorldMark's unified interface; (2) Model-specific translation—the action-mapping layer converts these canonical inputs into each model's native format (e.g., tokenized action sequences for Genie, latent action vectors for Matrix-Game, proprietary command structures for others); (3) Standardized evaluation—all models consume identical scenes (rendered from a common environment set) and execute identical action sequences, producing video outputs that are then scored on trajectory error (spatial accuracy of agent position over time), aesthetic metrics (visual quality, consistency), and VLM-based reasoning judgments (semantic correctness of predicted world state). The key architectural insight is that the action-mapping layer acts as a compatibility shim, allowing heterogeneous models to share a common test harness without modification.
Production Impact
For teams building interactive world models or comparing different video generation approaches, WorldMark eliminates the need to build custom evaluation pipelines for each baseline you want to compare. You can drop in any model that exposes a video output interface, map its action API once (typically a few hours of engineering), and immediately get standardized metrics against all other benchmarked models. This dramatically reduces the friction and cost of model selection and ablation studies—instead of spending weeks validating internal metrics, teams can use published WorldMark scores as a common currency for comparison. The main trade-off is upfront integration work (implementing the action mapper for your model's specific API) and potential semantic loss if your model's control scheme is fundamentally misaligned with the WASD abstraction (though the paper likely demonstrates this is rare). Compute cost is negligible—it's a thin adapter layer—but you must ensure your model can run in the shared benchmark environment (memory, latency constraints).
Limitations and When Not to Use This
WorldMark assumes all models can operate in a shared visual environment and execute sequences of discrete timesteps, which may not hold for models with continuous action spaces or models optimized for specific game engines or proprietary environments. The WASD+discrete-actions abstraction is designed for agent-centric games but may not naturally represent more complex control schemes (e.g., menu selection, text input, continuous analog sticks), limiting scope to a subset of interactive video domains. The paper doesn't address generalization—models trained on different datasets or with different visual conventions may perform inconsistently on WorldMark's curated scenes due to domain shift, which the benchmark cannot eliminate but only measure. Long-term trajectory evaluation is also unresolved: the benchmark likely uses relatively short action sequences, but real world models need to predict hours of coherent video, a challenge that benchmark protocols don't typically capture.
Research Context
This paper addresses a well-known problem in generative modeling: the evaluation crisis, where every research group publishes results on private benchmarks, making it impossible to track real progress. WorldMark follows the tradition of unified benchmarks like ImageNet (vision), GLUE (NLP), and ALE (RL), which historically have accelerated research by creating common evaluation ground and enabling meta-analysis. The work is timely because interactive video generation is now mature enough (Genie, YUME, HY-World are all recent) that the lack of unified benchmarking is actively hindering comparison and reproducibility. It opens a research direction on standardizing evaluation for multimodal generative models with heterogeneous control interfaces, which will become increasingly important as foundation models proliferate with incompatible APIs.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::