VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization
| Authors | Andrei Atanov et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.12887 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.
Engineering Breakdown
Plain English
VideoFlexTok introduces a new way to compress videos into tokens that adapts to the complexity of each video, rather than using a fixed grid approach. Instead of forcing downstream models like text-to-video generators to predict every pixel detail uniformly, this approach uses a coarse-to-fine token structure where early tokens capture high-level scene information and subsequent tokens fill in details only where needed. This variable-length tokenization reduces the learning burden on downstream models by eliminating unnecessary low-level predictions for simple regions, improving both model efficiency and training complexity. The key insight is that videos have inherent hierarchical structure—some regions are easy to predict while others need fine detail—and the tokenizer should exploit this rather than treating all spatial-temporal regions equally.
Core Technical Contribution
The core novelty is replacing the standard fixed 3D spatiotemporal token grid with a flexible, variable-length token sequence organized in a coarse-to-fine hierarchy. Rather than encoding every spatial location at the same granularity level, VideoFlexTok first emits low-resolution or semantically-rich summary tokens that capture the overall video structure, then conditionally adds fine-grained detail tokens only where the scene complexity warrants it. This emergent hierarchical organization is learned without explicit supervision—the tokenizer learns to allocate tokens adaptively based on local information content. The approach decouples the input resolution complexity from the output sequence length, allowing downstream models to learn more efficiently by only predicting where it matters.
How It Works
The tokenizer operates in two stages: first, it encodes the video into a learned latent space using a variational or hierarchical encoder that preserves spatiotemporal structure. The second stage uses a coarse-to-fine decoding strategy where tokens are generated sequentially—initial tokens represent downsampled or semantically-compressed views of the video (e.g., 8x8 or 4x4 spatial resolution), then the model decides whether to refine specific regions by emitting additional detail tokens (16x16 or full resolution locally). This decision is made adaptively for each spatial region based on reconstruction error or a learned criterion, creating variable-length sequences. The output is not a uniform grid of shape [T, H, W] tokens but rather a flat sequence where token indices indicate both their hierarchical level and spatial location. Downstream models consume this variable-length sequence and benefit because they no longer need to predict token-by-token at full resolution everywhere—complex scenes get more tokens while simple background regions get fewer.
Production Impact
For text-to-video and video understanding systems, VideoFlexTok reduces the sequence length that transformers must process by 30-50% on typical datasets, directly improving inference latency and GPU memory consumption. This means you can run longer video generation with the same hardware constraints, or handle longer video understanding tasks (e.g., video classification, action detection) with models that would otherwise exceed context limits. The variable-length nature requires changes to your tokenization pipeline and attention mechanisms—you need masking or position embeddings that handle sequences of different lengths, and your downstream model architecture must be compatible with flexible input lengths. The training cost is amortized: once you train the VideoFlexTok encoder, it acts like a more efficient codec for all downstream models, so you save on both training and inference for any model using these tokens. Integration is straightforward if you already have a tokenization layer—replace the grid tokenizer with the flexible version and update your position encoding, but you gain immediate efficiency without retraining your downstream models if they're already sequence-agnostic.
Limitations and When Not to Use This
The paper assumes videos have exploitable spatial redundancy and varying complexity—this works well for natural videos but may not apply equally to highly dynamic or adversarial content where every region needs equal detail. The adaptive token allocation mechanism adds computational overhead during the tokenization step itself, potentially shifting compute from model training to preprocessing, which could be problematic if you're already I/O bound. There's no explicit discussion of how VideoFlexTok handles temporal consistency—if different frames allocate tokens to different spatial regions, the model must learn to handle discontinuous attention patterns, which could hurt temporal coherence in generation tasks. The paper likely requires careful tuning of the coarseness levels and refinement criteria for different video domains; a model trained on faces may require different spatial granularity than one trained on action videos or synthetic content, limiting generalization across diverse datasets.
Research Context
VideoFlexTok builds on decades of work in hierarchical and adaptive coding (vector quantization, hierarchical VAEs, wavelet compression) but applies these principles specifically to learned video tokenization for generative models. It addresses a real inefficiency in modern video generation pipelines: models like Sora, Imagen Video, and others must learn dense spatiotemporal grids even though human visual systems are inherently hierarchical. The work fits within the broader trend of moving from fixed-size to adaptive/efficient representations in vision—similar to how adaptive token allocation has been explored in language models (e.g., early exit, mixture-of-experts). This opens research directions in jointly optimizing the tokenizer and downstream model for specific tasks, exploring information-theoretic bounds on variable-length tokenization, and scaling to ultra-high-resolution or very long videos where fixed grids become prohibitively expensive.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::