MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints
| Authors | Yu Qi et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.20194 |
| Download | |
| Categories | cs.CV |
Abstract
Video generative models show emerging reasoning behaviors. It is essential to ensure that generated events remain causally consistent across frames for reliable deployment, a property we define as reasoning coherence. To bridge the gap in literature for missing reasoning coherence evaluation, we propose MME-CoF-Pro, a comprehensive video reasoning benchmark to assess reasoning coherence in video models. Specifically, MME-CoF-Pro contains 303 samples across 16 categories, ranging from visual logical to scientific reasoning. It introduces Reasoning Score as evaluation metric for assessing process-level necessary intermediate reasoning steps, and includes three evaluation settings, (a) no hint (b) text hint and (c) visual hint, enabling a controlled investigation into the underlying mechanisms of reasoning hint guidance. Evaluation results in 7 open and closed-source video models reveals insights including: (1) Video generative models exhibit weak reasoning coherence, decoupled from generation quality. (2) Text hints boost apparent correctness but often cause inconsistency and hallucinated reasoning (3) Visual hints benefit structured perceptual tasks but struggle with fine-grained perception. Website: https://video-reasoning-coherence.github.io/
Engineering Breakdown
Plain English
This paper addresses a critical gap in evaluating video generation models by proposing MME-CoF-Pro, a benchmark with 303 samples across 16 reasoning categories designed to measure whether generated videos maintain causal consistency and logical coherence across frames. The authors introduce a Reasoning Score metric that assesses whether intermediate reasoning steps are necessary and correct during video generation, covering tasks from visual logic puzzles to scientific reasoning problems. The benchmark includes three evaluation modes—no hint, text hint, and visual hint—allowing researchers to systematically understand how different forms of guidance affect a model's ability to reason coherently. This addresses a real deployment problem: current video models can generate visually plausible frames that are causally incoherent or violate the laws of physics, making them unreliable for applications requiring reliable reasoning.
Core Technical Contribution
The core contribution is the definition and operationalization of 'reasoning coherence' as a measurable property of video generation models, paired with MME-CoF-Pro benchmark and the Reasoning Score evaluation metric. Unlike prior work that evaluates video quality through visual fidelity or temporal consistency, this paper specifically targets whether generated sequences maintain logical causality and follow multi-step reasoning patterns that humans would expect. The Reasoning Score moves beyond binary correctness to assess process-level intermediate steps, enabling fine-grained diagnosis of where models fail in reasoning chains. The three-setting evaluation design (no hint / text hint / visual hint) is novel in allowing controlled ablation of what information models need to maintain coherence, providing mechanistic insights into model reasoning capabilities.
How It Works
The MME-CoF-Pro benchmark operates as follows: it presents video generation models with reasoning tasks spanning 16 categories (visual logic, physics, mathematics, scientific reasoning, etc.), each with a question or constraint that must be satisfied across the generated video frames. For each task, the model generates frames that should demonstrate step-by-step logical progression—for example, if asked to generate a sequence showing water freezing, the frames must show temperature drop, phase transition, and final solid state in proper order. The Reasoning Score evaluates generated outputs by: (1) checking whether the final outcome is correct, (2) verifying that necessary intermediate reasoning steps appear in the sequence, and (3) assessing whether the causal relationships between frames are valid. The three evaluation settings control information available to the model: the no-hint setting requires pure generative reasoning, text-hint provides natural language descriptions of the reasoning steps needed, and visual-hint provides intermediate visual examples, allowing researchers to identify whether failures stem from reasoning ability versus visual execution capability.
Production Impact
For engineers building video generation systems, this benchmark enables quality gates for causal coherence before deployment—critical for applications like robotics simulation, scientific visualization, or educational content where incorrect reasoning propagates downstream. Adopting MME-CoF-Pro means adding a reasoning evaluation step to your training pipeline: you'd benchmark candidate models against the 303 test cases and set minimum Reasoning Score thresholds for production eligibility, catching models that generate visually smooth but logically invalid sequences. The three-setting structure directly informs architecture choices—if models perform well with text hints but fail without them, you might architect systems to include reasoning text tokens during generation, increasing latency but improving reliability. Trade-offs are significant: comprehensive reasoning evaluation adds evaluation cost and latency to deployment pipelines, and the benchmark's 303 samples, while diverse across categories, may not cover domain-specific reasoning patterns your application requires, necessitating custom annotation effort.
Limitations and When Not to Use This
The benchmark's 303 samples across 16 categories, while diverse, remains relatively small for drawing statistical conclusions about model generalization—production systems processing millions of videos need larger test sets to validate reasoning coherence at scale. The paper assumes that intermediate reasoning steps can be clearly labeled and verified by human annotators, which becomes problematic for ambiguous reasoning scenarios or domains where multiple valid reasoning paths exist. The three evaluation settings (no hint / text hint / visual hint) are artificial compared to real-world deployment where models must reason with implicit domain knowledge and context not explicitly provided. The benchmark likely focuses on deterministic reasoning tasks where ground truth is unambiguous; it doesn't address stochastic or probabilistic reasoning, where multiple causally-valid futures exist, limiting applicability to real-world scenarios with inherent uncertainty.
Research Context
This work builds on the growing recognition that video generation models exhibit emergent reasoning capabilities, extending prior work on video consistency and temporal coherence into the reasoning domain. It responds to limitations in existing benchmarks like those focused purely on visual quality (Fréchet Video Distance) or temporal smoothness, which miss logical correctness—a requirement identified by practitioners deploying models in domains requiring causal reliability. The paper positions itself within the broader safety and evaluation movement in AI systems, alongside work on hallucination detection in language models and robustness testing in vision systems. By introducing a structured benchmark with multi-category coverage and process-level metrics, it opens research directions toward developing video models with explicit reasoning supervision and causal constraint mechanisms.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::