OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video
| Authors | Junfu Pu et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2604.11102 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Current multimodal large language models (MLLMs) have demonstrated remarkable capabilities in short-form video understanding, yet translating long-form cinematic videos into detailed, temporally grounded scripts remains a significant challenge. This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts encompassing character actions, dialogues, expressions, and audio cues. To facilitate this, we construct a first-of-its-kind human-annotated benchmark and propose a temporally-aware hierarchical evaluation framework. Furthermore, we present OmniScript, an 8B-parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension. OmniScript is trained via a progressive pipeline that leverages chain-of-thought supervised fine-tuning for plot and character reasoning, followed by reinforcement learning using temporally segmented rewards. Extensive experiments demonstrate that despite its parameter efficiency, OmniScript significantly outperforms larger open-source models and achieves performance comparable to state-of-the-art proprietary models, including Gemini 3-Pro, in both temporal localization and multi-field semantic accuracy.
Engineering Breakdown
Plain English
This paper introduces the video-to-script (V2S) task, which converts long-form cinematic videos into detailed, temporally grounded narrative scripts with character actions, dialogues, expressions, and audio cues. The authors create a human-annotated benchmark dataset and propose OmniScript, an 8-billion-parameter multimodal language model designed specifically for understanding long-form video narratives by processing both audio and visual information together. The key innovation is moving beyond short-form video understanding to tackle the significantly harder problem of generating hierarchical, scene-by-scene scripts from feature films or similar long-form content. They introduce a temporally-aware hierarchical evaluation framework to measure script generation quality, and train OmniScript using a progressive pipeline leveraging chain-of-thought reasoning to improve output accuracy.
Core Technical Contribution
The core contribution is the formulation and benchmarking of the video-to-script task itself—a new structured output problem that requires temporal grounding and hierarchical scene understanding rather than simple video captioning. OmniScript represents a specialized architecture combining audio-visual fusion at the multimodal level, designed to handle the extended context lengths needed for full-length film analysis where standard video models struggle. The temporally-aware hierarchical evaluation framework is novel because it measures not just caption quality but also whether the model correctly grounds script elements to specific timestamps and maintains narrative consistency across scenes. The progressive training pipeline incorporating chain-of-thought reasoning represents a methodological contribution showing how to teach language models to reason through complex, multi-step narrative structures implicit in video.
How It Works
The system takes long-form video as input (audio track + visual frames) and produces a hierarchical script organized by scenes, where each scene contains character actions, dialogues, facial expressions, and audio descriptions with precise temporal anchors. The OmniScript architecture operates as an 8B-parameter omni-modal language model—meaning it processes audio and visual tokens directly rather than relying on separate encoders and fusion layers, creating a unified semantic space for reasoning about multimodal content. During inference, the model uses chain-of-thought prompting to first identify scene boundaries and character introductions, then generates dialogue and action descriptions for each scene with timestamp annotations, ensuring temporal coherence. The training pipeline is progressive, meaning it starts with easier sub-tasks (e.g., generating action descriptions with timing) before advancing to full script generation with dialogue and expressions, allowing the model to build understanding incrementally rather than learning the full task end-to-end from scratch.
Production Impact
For studios and content platforms, this enables automated generation of video scripts, subtitles, and accessibility descriptions at scale—reducing manual annotation costs for archival, searchability, and accessibility compliance from weeks to hours. Production video editing pipelines could integrate OmniScript to automatically generate scene breakdowns, shot lists, and script notes during post-production, accelerating the review cycle and reducing human transcription labor. The temporal grounding aspect is critical for production workflows: unlike generic video captioning, this generates scripts where every action and dialogue line has a timestamp, making it directly usable for subtitle timing, animation synchronization, and film editing. However, the 8B parameter size and need to process full-length videos means high GPU memory requirements (likely 20-40GB VRAM for inference depending on video length) and significant inference latency—processing a 2-hour film might take 30-60 minutes on a single A100, creating constraints on interactive use cases. Integration challenges include needing to handle variable video codecs and frame rates, building temporal alignment pipelines to sync generated scripts with actual audio/visual content, and establishing quality thresholds since hallucination in script generation could introduce false dialogue.
Limitations and When Not to Use This
The paper's abstract doesn't provide quantitative results or error analysis, so it's unclear whether OmniScript actually achieves human-level script quality or how often it generates plausible but incorrect character dialogue—a critical failure mode for production use. Long-form video understanding introduces severe context window challenges; even with 8B parameters, the model likely struggles with films over 90 minutes unless using aggressive summarization, limiting applicability to feature-length content without breaking videos into chapters. The approach assumes well-structured cinematic content with clear audio dialogue and visible character actions; it would likely fail on experimental films, heavy-effects sequences with no dialogue, or low-quality/obscured video where temporal grounding becomes unreliable. The human-annotated benchmark dataset likely contains Western cinema conventions (lighting, framing, narrative structure), so generalization to other cinematic traditions, documentaries, or non-narrative video formats remains unexplored.
Research Context
This work builds on the trajectory of multimodal large language models (like GPT-4V, Gemini) that have shown strong performance on image and short video understanding, but extends them into a new domain where temporal reasoning and extended narrative coherence become primary challenges. The video-to-script task represents a natural evolution from image captioning → video captioning → long-form video understanding, filling a gap between generic video understanding and domain-specific applications like automatic subtitling or film analysis. The hierarchical evaluation framework contributes methodology for measuring structured output quality in multimodal tasks, which has broader applicability to other hierarchical generation problems (e.g., document summarization, code generation with function decomposition). The work opens research directions around temporal grounding in extended contexts, multimodal chain-of-thought reasoning, and how to efficiently handle variable-length video at inference time without massive computational overhead.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::