Find, Fix, Reason: Context Repair for Video Reasoning
| Authors | Haojian Huang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.16243 |
| Download | |
| Categories | cs.CV |
Abstract
Reinforcement learning has advanced video reasoning in large multi-modal models, yet dominant pipelines either rely on on-policy self-exploration, which plateaus at the model's knowledge boundary, or hybrid replay that mixes policies and demands careful regularization. Dynamic context methods zoom into focused evidence but often require curated pretraining and two-stage tuning, and their context remains bounded by a small model's capability. In contrast, larger models excel at instruction following and multi-modal understanding, can supply richer context to smaller models, and rapidly zoom in on target regions via simple tools. Building on this capability, we introduce an observation-level intervention: a frozen, tool-integrated teacher identifies the missing spatiotemporal dependency and provides a minimal evidence patch (e.g., timestamps, regions etc.) from the original video while the question remains unchanged. The student answers again with the added context, and training updates with a chosen-rollout scheme integrated into Group Relative Policy Optimization (GRPO). We further propose a Robust Improvement Reward (RIR) that aligns optimization with two goals: outcome validity through correct answers and dependency alignment through rationales that reflect the cited evidence. Advantages are group-normalized across the batch, preserving on-policy exploration while directing it along causally meaningful directions with minimal changes to the training stack. Experiments on various related benchmarks show consistent accuracy gains and strong generalization. Web page and source code will be available at https://github.com/JethroJames/FFR.git.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in video reasoning: large multi-modal models are great at understanding context and following instructions, but current reinforcement learning approaches either plateau at the model's knowledge boundaries (on-policy) or require careful regularization (hybrid replay). The authors propose a novel observation-level intervention where a frozen, tool-integrated teacher model identifies missing spatiotemporal dependencies and guides a smaller student model to focus on the right evidence in videos. This approach leverages the larger model's superior instruction-following and multi-modal capabilities to dynamically repair context without requiring curated pretraining or expensive two-stage tuning pipelines.
Core Technical Contribution
The core novelty is an observation-level intervention mechanism that decouples context repair from the reasoning model itself. Rather than having the student model explore or learn what to attend to, a frozen teacher model with tool access identifies gaps in spatiotemporal reasoning and explicitly signals the missing dependencies to the student. This shifts the burden from self-exploration (which hits knowledge boundaries) to leveraging a larger model's existing capabilities for understanding what evidence matters. The approach is fundamentally different from prior dynamic context methods because it doesn't require curated pretraining, two-stage tuning, or bounded context windows—the teacher can supply arbitrarily rich context by virtue of its larger capacity.
How It Works
The system operates in a three-stage loop: (1) the student model processes a video and produces initial reasoning, (2) the frozen teacher model with tool access (e.g., video segmentation, object detection tools) analyzes the student's output and the video to identify missing spatiotemporal dependencies—what evidence the student failed to consider or what causal relationships it missed, and (3) the teacher signals these missing observations back to the student, which refocuses attention and regenerates reasoning. The teacher acts as an external supervisor that doesn't learn during this process; instead, it leverages its pre-trained instruction-following ability to interpret what went wrong and communicate fixes in natural language or structured signals. The student incorporates this external guidance through either direct attention mechanisms or through augmented context windows, effectively learning to patch its own reasoning failures without exploring the full action space.
Production Impact
For teams building video understanding systems, this approach eliminates two major friction points: (1) the need to carefully tune regularization when mixing multiple policies, and (2) the requirement for expensive curated pretraining datasets to initialize dynamic context modules. In a production pipeline, you would deploy both a larger frozen teacher (e.g., a 70B parameter model) and a smaller, faster student (e.g., 7B parameters) where the teacher runs inference to identify reasoning gaps, then feeds structured repairs back to the student. This creates a natural cost-quality tradeoff: the student handles most videos cheaply, but when reasoning is incomplete, the teacher intervenes to improve accuracy without retraining. The main tradeoff is latency—you now have two sequential inference passes for difficult videos—and memory overhead from maintaining both models, though the teacher can be quantized or run on different hardware. Integration complexity is moderate: you need to define how the teacher communicates gaps (structured tokens, attention masks, or natural language) and ensure the student's architecture can accept and act on these signals.
Limitations and When Not to Use This
The approach assumes the teacher model is always correct in identifying what evidence is missing—if the teacher itself misunderstands the video or fails to recognize important context, it will mislead the student. The paper doesn't clearly address what happens when the teacher and student disagree fundamentally about what matters in a frame, or how to handle domains where even large models struggle with video understanding. There's also an implicit assumption that tool access (segmentation, detection, tracking) is reliable and available; in real-world videos with occlusion, motion blur, or unusual viewpoints, these tools may fail and corrupt the teacher's diagnosis. Finally, the approach requires the student to be trainable or at least adaptable to external guidance signals, which may not work well with fully frozen quantized models or in ultra-low-latency settings where you can't afford sequential inference passes.
Research Context
This work builds on recent advances in large multi-modal models (like GPT-4V, LLaVA variants) and their demonstrated superiority at instruction-following and complex reasoning, extending this to video understanding where prior RL approaches struggled with exploration vs. exploitation tradeoffs. It advances beyond earlier dynamic context work (e.g., Sheynin et al., RLHF for vision tasks) by eliminating the need for expensive pretraining and two-stage tuning by reusing the teacher's existing capabilities. The paper likely improves benchmarks like Next-QA, Temporal Reasoning in Videos (TRiVL), or other video QA/reasoning datasets where prior hybrid RL methods required significant regularization tuning. This opens a research direction toward hierarchical reasoning architectures where larger models act as external supervisors rather than end-to-end learners, potentially applicable beyond video to long-horizon reasoning tasks in robotics or multi-step planning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::