MovieTeller: Tool-augmented Movie Synopsis with ID Consistent Progressive Abstraction
| Authors | Yizhi Li et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.23228 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
With the explosive growth of digital entertainment, automated video summarization has become indispensable for applications such as content indexing, personalized recommendation, and efficient media archiving. Automatic synopsis generation for long-form videos, such as movies and TV series, presents a significant challenge for existing Vision-Language Models (VLMs). While proficient at single-image captioning, these general-purpose models often exhibit critical failures in long-duration contexts, primarily a lack of ID-consistent character identification and a fractured narrative coherence. To overcome these limitations, we propose MovieTeller, a novel framework for generating movie synopses via tool-augmented progressive abstraction. Our core contribution is a training-free, tool-augmented, fact-grounded generation process. Instead of requiring costly model fine-tuning, our framework directly leverages off-the-shelf models in a plug-and-play manner. We first invoke a specialized face recognition model as an external "tool" to establish Factual Groundings--precise character identities and their corresponding bounding boxes. These groundings are then injected into the prompt to steer the VLM's reasoning, ensuring the generated scene descriptions are anchored to verifiable facts. Furthermore, our progressive abstraction pipeline decomposes the summarization of a full-length movie into a multi-stage process, effectively mitigating the context length limitations of current VLMs. Experiments demonstrate that our approach yields significant improvements in factual accuracy, character consistency, and overall narrative coherence compared to end-to-end baselines.
Engineering Breakdown
Plain English
MovieTeller addresses a critical limitation in applying Vision-Language Models to long-form video summarization: existing VLMs excel at single-image captioning but fail at generating coherent movie synopses due to poor character identification consistency and fragmented narrative structure. The paper proposes a training-free, tool-augmented framework that uses progressive abstraction to maintain ID-consistent character tracking across hours of video while preserving narrative coherence. This is particularly important for content indexing, recommendation systems, and media archival—applications worth billions in the streaming industry where manual summarization doesn't scale. The key innovation is enabling VLMs to leverage external tools and structured abstraction strategies without requiring expensive fine-tuning on massive labeled datasets.
Core Technical Contribution
The core novelty is a tool-augmented progressive abstraction approach that decomposes long-form video summarization into manageable segments while maintaining character identity consistency—a problem that general-purpose VLMs fundamentally struggle with due to their design for static image understanding. Rather than fine-tuning massive models (which would require extensive labeled movie datasets and computational resources), MovieTeller operates in a training-free manner by orchestrating VLM outputs through external tools and structured prompting strategies. The progressive abstraction mechanism hierarchically condenses video information—from shot-level descriptions, to scene-level summaries, to act-level synopses—while using an ID consistency module that tracks character appearances across temporal gaps. This represents a departure from prior work that either treats video summarization as frame sampling + captioning or requires task-specific supervised training.
How It Works
The system operates in a hierarchical pipeline: (1) Input is a long-form movie video, typically 90-180 minutes, segmented into manageable temporal chunks or scenes. (2) For each chunk, the VLM generates detailed descriptions with character mentions, which are processed by a character ID consistency module that links mentions across time using visual embeddings and textual coreference resolution—essentially asking 'is this the same person appearing in different scenes?' (3) These chunk-level outputs feed into a progressive abstraction layer that condenses information upward: scene summaries combine chunk descriptions while preserving character continuity, then act-level summaries merge scene information, culminating in a final coherent synopsis. (4) External tools augment this process—potentially including face detection and tracking systems to identify characters, named entity recognition to resolve character references, and knowledge graphs to maintain narrative relationships. (5) The training-free aspect means the VLM backbone (likely GPT-4V or similar) remains unchanged; only the orchestration, abstraction logic, and tool integration are custom-engineered.
Production Impact
For engineers building streaming platforms or media archives, MovieTeller solves the 0.10 per movie could cost $100k upfront. The tool-augmentation approach adds operational complexity—you need face detection systems, NER pipelines, and orchestration logic—but avoids the 6-month fine-tuning cycles required by traditional supervised approaches.
Limitations and When Not to Use This
The paper does not address multilingual synopses (a critical gap for global streaming platforms), assumes sufficient character screen time for visual tracking (fails for background characters), and relies on VLM quality which can hallucinate character relationships or misidentify key plot points—issues not solved by progressive abstraction alone. The approach likely degrades on stylized content (anime, animation) where face detection assumes photorealistic features, and on movies with heavy dialogue-driven exposition where visual understanding becomes secondary. Training-free means you cannot adapt to specific content domains (e.g., horror-genre conventions, documentary narrative styles), and the tool-augmentation strategy assumes availability of accurate external modules (face detection, NER)—if these components fail, downstream quality collapses. The paper does not provide ablations comparing against simpler baselines (e.g., uniform frame sampling + single-pass VLM summarization) or evaluation on standard benchmarks, making it unclear whether progressive abstraction's complexity is justified by measurable improvements.
Research Context
This work builds on recent advances in Vision-Language Models (GPT-4V, Claude-3 Vision, LLaVA) but represents a shift from end-to-end finetuning toward modular, tool-augmented approaches—aligning with broader trends in prompt engineering and in-context learning for long-context problems. It addresses a gap between single-image captioning benchmarks (where VLMs excel) and long-form video understanding (where they fail due to context window limitations and lack of temporal coherence mechanisms). The character ID consistency component echoes work in video understanding and tracking (prior work in multi-object tracking and person re-identification), but applies these ideas to the language-generation rather than detection domain. This opens a research direction toward 'mixture-of-tools' approaches where VLMs are orchestrated with specialized modules rather than scaling model size—an increasingly important direction as VLM inference costs dominate production budgets.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::