Seeing Fast and Slow: Learning the Flow of Time in Videos
| Authors | Yen-Siang Wu et al. |
| Year | 2026 |
| HF Upvotes | 17 |
| arXiv | 2604.21931 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
How can we tell whether a video has been sped up or slowed down? How can we generate videos at different speeds? Although videos have been central to modern computer vision research, little attention has been paid to perceiving and controlling the passage of time. In this paper, we study time as a learnable visual concept and develop models for reasoning about and manipulating the flow of time in videos. We first exploit the multimodal cues and temporal structure naturally present in videos to learn, in a self-supervised manner, to detect speed changes and estimate playback speed. We then show that these learned temporal reasoning models enable us to curate the largest slow-motion video dataset to date from noisy in-the-wild sources. Such slow-motion footage, typically filmed by high-speed cameras, contains substantially richer temporal detail than standard videos. Using this data, we further develop models capable of temporal control, including speed-conditioned video generation, which produces motion at specified playback speed, and temporal super-resolution, which tranforms low-FPS, blurry videos into high-FPS sequences with fine-grained temporal details. Our findings highlight time as a manipulable, perceptual dimension in video learning, opening doors to temporally controllable video generation, temporal forensics detection, and potentially richer world-models that understand how events unfold over time.
Engineering Breakdown
Plain English
This paper tackles temporal reasoning in videos by learning to detect and estimate playback speed in a self-supervised manner, without requiring labeled speed annotations. The authors develop models that understand temporal structure and multimodal cues naturally present in video to reason about time manipulation. Their key achievement is using these learned speed detection models to automatically curate the largest slow-motion video dataset from noisy internet sources, solving a major data bottleneck for video generation and analysis tasks. The work opens up practical applications in video retiming, understanding temporal dynamics, and generating high-quality slow-motion content.
Core Technical Contribution
The core novelty is framing temporal speed perception as a self-supervised learning problem rather than requiring expensive manual labeling of video playback speeds. The authors exploit the inherent multimodal structure in videos (motion patterns, audio timing, object dynamics) to create training signals for speed estimation without ground truth annotations. This enables them to build accurate speed detectors that can then be used as quality filters to identify and extract high-quality slow-motion videos from large-scale noisy video collections. The technical insight is that learning to detect speed changes implicitly teaches models rich temporal reasoning that transfers to downstream tasks like video generation and manipulation.
How It Works
The system operates in two stages. First, a self-supervised training phase learns speed estimation by taking a video input, artificially altering its playback speed by known factors (2x, 0.5x, etc.), and training a convolutional neural network to predict what speed modification was applied. The model operates on temporal windows of video frames and leverages multimodal cues: optical flow patterns change predictably with speed, audio pitch shifts correlate with temporal changes, and object motion velocities scale linearly with playback speed. These multiple signals are fused through the network architecture to create a robust speed detection signal. In the second stage, this trained speed detector is deployed as a quality filter on large-scale internet video datasets, scoring each video for slow-motion plausibility and selecting high-confidence examples to build a curated slow-motion dataset. The filtering process effectively identifies videos that exhibit the visual and acoustic characteristics of actual high-frame-rate slow-motion footage rather than artificially sped-down content.
Production Impact
For teams building video generation, enhancement, or analysis systems, this approach eliminates the expensive manual annotation bottleneck for temporal datasets. Instead of hiring annotators to label video speeds or curate slow-motion footage, engineers can apply the self-supervised speed detection model as an automated quality filter over any large video corpus. This dramatically reduces data preparation costs and enables scaling to internet-scale datasets. The learned temporal reasoning becomes a reusable component for downstream tasks: video interpolation systems can use speed estimation as an auxiliary task to improve frame prediction, video editing tools can enable intelligent speed ramping, and content recommendation systems can reason about temporal dynamics. However, engineers should expect that speed detection accuracy degrades on videos with complex camera motion, heavy occlusions, or ambiguous temporal semantics (like static scenes), requiring careful threshold tuning per use case. Compute-wise, inference is relatively cheap (per-video scoring is O(n) where n is video length), but initial dataset curation may require processing millions of videos to extract a high-quality subset.
Limitations and When Not to Use This
The self-supervised approach assumes that temporal structure is sufficiently predictable from multimodal cues, which breaks down for videos with static content, uniform scenes, or ambiguous motion patterns where multiple speeds appear equally plausible. The paper's dataset curation relies on the quality of the speed detector itself, creating a circular dependency: if the detector makes systematic errors, the curated dataset inherits those biases. Real-world deployment faces challenges with videos containing rapid cuts, complex camera movements, or out-of-distribution content where optical flow and motion assumptions fail. The work also doesn't address the inverse problem at scale: generating high-quality variable-speed video requires additional architecture beyond speed detection, and the paper's focus on detection leaves open how to best leverage these learned representations for generation tasks. Cross-domain generalization is not thoroughly evaluated—a detector trained on internet videos may not transfer well to specialized domains like medical imaging, scientific microscopy, or artistic video where temporal semantics differ significantly.
Research Context
This work builds on decades of temporal analysis in computer vision (optical flow, temporal CNNs, action recognition) and extends recent advances in self-supervised learning by showing that temporal structure can serve as a powerful supervisory signal without labels. It relates to video prediction and frame interpolation literature by tackling the temporal axis as a learnable concept rather than treating time as a fixed scaffold. The paper advances the broader emerging field of temporal reasoning in videos—most prior work focused on spatial understanding or action classification, while this explicitly treats time manipulation as a first-class problem. The large-scale slow-motion dataset contribution addresses a real bottleneck in video generation research, where high-frame-rate footage is rare and expensive to produce, enabling future work in perceptual slow-motion generation and video quality enhancement.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::