Structured Causal Video Reasoning via Multi-Objective Alignment
| Authors | Zinuo Li et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2604.04415 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Human understanding of video dynamics is typically grounded in a structured mental representation of entities, actions, and temporal relations, rather than relying solely on immediate deductive reasoning. In contrast, existing Video-LLMs largely depend on unstructured video reasoning, where critical visual evidence is embedded in verbose textual descriptions and temporal causality is often weakly modeled. This leads to inefficient processes and fragile causal inference. To bridge this cognitive gap, we propose constructing a compact representation of salient events and their causal relationships, which we name Structured Event Facts, prior to the reasoning stage. This structured prior serves as an explicit constraint to promote concise and causally grounded reasoning, while also making intermediate evidence easier to verify. To effectively train models on such structured facts, we introduce CausalFact-60K and a four-stage training pipeline comprising facts alignment, format warm-start, thinking warm-start, and reinforcement learning-based post-training. During RL stage, we find that this framework introduces competing objectives, as structural completeness and causal fidelity must be balanced against reasoning length, making it difficult to optimize. We address this challenge by formulating the optimization as a Multi-Objective Reinforcement Learning (MORL) problem and explicitly optimizing toward the Pareto-Frontier to balance these trade-offs. As a result, we introduce Factum-4B, which yields more reliable reasoning and delivers stronger performance on challenging video understanding tasks requiring fine-grained temporal inference.
Engineering Breakdown
Plain English
This paper addresses a critical weakness in current Video-LLMs: they reason about videos using unstructured text descriptions where temporal causality and critical visual evidence get buried in verbose outputs. The authors propose creating Structured Event Facts—a compact, explicit representation of salient events and their causal relationships—before the reasoning stage. This structured prior acts as a constraint that forces the model to ground its reasoning in concrete causal relationships rather than loose correlations. The approach bridges the gap between how humans mentally model video dynamics (via structured entities and relations) and how current Video-LLMs operate (via unstructured reasoning), leading to more efficient and robust causal inference.
Core Technical Contribution
The core innovation is a two-stage reasoning pipeline where event structuring precedes causal reasoning, rather than mixing them together. Instead of feeding raw video frames or unstructured captions directly to an LLM, the authors first extract and organize salient events into an explicit causal graph or knowledge structure. This structured prior then serves as a hard constraint during reasoning—the model must reference and respect the pre-computed causal relationships rather than inferring causality from text alone. This is fundamentally different from prior Video-LLMs that treat video understanding as an end-to-end language modeling problem; here, structure is built in before language reasoning begins, making causality interpretable and verifiable.
How It Works
The pipeline operates in two distinct phases. First, a video encoder identifies salient events and their temporal ordering, constructing Structured Event Facts—likely a graph or set of explicit propositions encoding entities (actors, objects), actions, and temporal-causal relations (before, after, causes, enables). This structure is compact and human-readable, avoiding the verbose text descriptions that plague current systems. Second, this structured representation is fed to a Video-LLM as a constraint or prefix that guides reasoning. During the reasoning phase, the model must answer causal questions (e.g., 'why did X happen?') by explicitly referencing edges in the event graph rather than generating free-form text. A multi-objective alignment loss likely trains the system to balance fidelity to the structured facts, accuracy of causal inference, and conciseness of outputs—preventing the model from ignoring the structure or generating verbose text that circumvents it.
Production Impact
For production systems, this approach solves a major interpretability and reliability problem: current Video-LLMs produce causal explanations that are hard to verify because the reasoning is implicit in text. Adopting this structured approach would require two key changes: (1) an event extraction module that must run on every video before querying the LLM, adding preprocessing latency and compute cost, and (2) integration of the structured facts into the prompt or model architecture, which may require retraining or fine-tuning existing models. The trade-off is worth it for high-stakes applications (autonomous systems, forensic analysis, safety-critical video understanding) where causal correctness is non-negotiable, but adds overhead for simple classification or retrieval tasks. The structured output also enables easier debugging, auditing, and integration with downstream systems that need causal guarantees—a major operational advantage over black-box Video-LLMs.
Limitations and When Not to Use This
The paper's core assumption—that salient events and their causal structure can be reliably extracted before reasoning—is fragile in real-world videos with ambiguous, overlapping, or hierarchical causality. The approach likely struggles with counterfactual reasoning (answering 'what if' questions about events not present in the video), weakly-motivated actions, or causal chains with long-range dependencies that are hard to represent explicitly. Compute and data requirements are understated: building accurate event extractors requires substantial labeled video data with causal annotations, and the two-stage pipeline may have higher total latency than end-to-end models for simple queries. The paper also doesn't address how to handle videos with missing information, conflicting causal interpretations, or continuous processes that don't decompose cleanly into discrete events—common in real-world video understanding.
Research Context
This work sits at the intersection of video understanding, structured reasoning, and neuro-symbolic AI. It builds on the Video-LLM literature (which scales vision-language models to videos) and explicitly critiques the unstructured reasoning pattern dominant since models like GPT-4V. The approach borrows from knowledge graph and causal inference research, where explicit structure has long been known to improve robustness and interpretability. This paper likely positions itself as a step toward more interpretable, causally-grounded video reasoning—opening research directions in multi-modal knowledge graphs, weakly-supervised event extraction from video, and the integration of symbolic causal models with large language models. It may evaluate on benchmarks like Something-Something v2 (for temporal reasoning) or custom causal video QA datasets, comparing against baseline Video-LLMs on metrics like causal reasoning accuracy, inference efficiency, and explanation quality.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::