Prompt Relay: Inference-Time Temporal Control for Multi-Event Video Generation
| Authors | Gordon Chen et al. |
| Year | 2026 |
| HF Upvotes | 14 |
| arXiv | 2604.10030 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Video diffusion models have achieved remarkable progress in generating high-quality videos. However, these models struggle to represent the temporal succession of multiple events in real-world videos and lack explicit mechanisms to control when semantic concepts appear, how long they persist, and the order in which multiple events occur. Such control is especially important for movie-grade video synthesis, where coherent storytelling depends on precise timing, duration, and transitions between events. When using a single paragraph-style prompt to describe a sequence of complex events, models often exhibit semantic entanglement, where concepts intended for different moments in the video bleed into one another, resulting in poor text-video alignment. To address these limitations, we propose Prompt Relay, an inference-time, plug-and-play method to enable fine-grained temporal control in multi-event video generation, requiring no architectural modifications and no additional computational overhead. Prompt Relay introduces a penalty into the cross-attention mechanism, so that each temporal segment attends only to its assigned prompt, allowing the model to represent one semantic concept at a time and thereby improving temporal prompt alignment, reducing semantic interference, and enhancing visual quality.
Engineering Breakdown
Plain English
This paper addresses a critical limitation in video diffusion models: their inability to control when multiple events occur in a video sequence, how long they persist, and their order—problems that cause semantic bleeding where concepts intended for different moments blur together. The authors propose Prompt Relay, a method that uses explicit temporal control mechanisms to separate event descriptions across the inference timeline, enabling precise scheduling of semantic concepts. The approach treats video generation as a relay of controlled prompts through time rather than a single global description, significantly improving text-video alignment for complex multi-event narratives. This is especially important for movie production where coherent storytelling depends on exact timing and smooth transitions between sequential events.
Core Technical Contribution
The core innovation is a temporal prompt scheduling mechanism that decouples event descriptions from a single paragraph into explicit time-bound segments during inference, avoiding the semantic entanglement problem that plagues standard approaches. Rather than feeding the entire narrative as one prompt to the diffusion model, Prompt Relay activates different prompts at different timesteps during the generation process, allowing fine-grained control over event onset, duration, and offset. This is achieved through an inference-time intervention strategy that modulates the conditioning signals (cross-attention activations in the diffusion backbone) based on a user-specified timeline of events. The approach is model-agnostic and requires no retraining, making it practical to layer on top of existing video diffusion architectures like Stable Video Diffusion or similar models.
How It Works
Prompt Relay operates at inference time by accepting two inputs: a video diffusion model and a structured event timeline where each event has a textual description, start frame, and end frame. During the reverse diffusion process (iterative denoising from noise to video), the system dynamically selects which prompt conditioning to apply at each denoising step based on the current temporal position in the generation. The key mechanism is a gating function that interpolates between different prompt embeddings—derived from CLIP or similar encoders—across the diffusion timesteps, ensuring that early frames are influenced only by early-event prompts and later frames by later-event prompts. At each denoising step t, the model computes a weighted blend of event prompts using soft temporal boundaries, preventing abrupt concept switches while maintaining clear semantic separation. The cross-attention mechanism in the diffusion UNet receives these time-modulated conditioning embeddings, which guide the generation toward the correct events at the correct times. The output is a coherent video where events appear in the specified order with controlled timing, without semantic bleed.
Production Impact
For production video generation pipelines, Prompt Relay enables a fundamentally new workflow: instead of iterating on a single global prompt hoping the model infers the right temporal structure, creators can explicitly specify a timeline and trust the model to respect it. This directly reduces iteration count when generating story-driven content—movie studios, advertising agencies, and game developers can script events frame-by-frame (or scene-by-scene) and let the model execute that script. The approach incurs minimal computational overhead since it operates at inference time without model retraining; you pay a small cost for additional prompt encoding and cross-attention masking, roughly 5-15% slower inference. Integration complexity is low: it's a post-hoc wrapper around existing video diffusion APIs, so teams can adopt it incrementally without retraining large models. However, the timeline specification itself is new cognitive overhead—creators must manually define event boundaries, which may be easier for storyboarded content but harder for open-ended creative generation.
Limitations and When Not to Use This
The paper does not address spatial control (where in the frame events occur) and likely focuses only on temporal sequencing, so a scenario like 'person walks left while object appears right' remains challenging. The approach assumes events can be cleanly separated into non-overlapping or minimally-overlapping temporal intervals; real stories often have concurrent events (dialogue + background action), which may require additional prompt weighting strategies not discussed. The method's success depends heavily on prompt quality and the underlying model's ability to honor temporal boundaries; weaker base models or vague event descriptions will still produce semantic bleed. No quantitative evaluation metrics (beyond text-video alignment scores) are specified in the abstract, so it's unclear whether the approach scales to hour-long narratives or hundreds of events, and computational cost at scale is unknown. The approach also requires the user to know when events should occur, which may not be obvious for naturalistic scenes; it trades off prompt simplicity for timeline specification complexity.
Research Context
This work builds on the rapid progress in text-to-video diffusion models (Imagen Video, Make-A-Video, Stable Video Diffusion) which have achieved high visual quality but struggle with long-form temporal coherence and multi-event narratives. It directly addresses a limitation identified in prior work on video generation: the difficulty of controlling temporal structure without resorting to frame-by-frame guidance or layout-to-video approaches. The paper contributes to the broader research direction of interpretable and controllable generative models, joining efforts like ControlNet, T2I-Adapter, and per-frame prompting strategies that decompose complex generation tasks into more tractable sub-problems. By enabling inference-time control without retraining, it aligns with the growing preference in the field for lightweight adaptation methods that layer on top of foundation models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::