When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models
| Authors | Zhengyang Sun et al. |
| Year | 2026 |
| HF Upvotes | 109 |
| arXiv | 2604.08546 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Text-to-video diffusion models have enabled open-ended video synthesis, but often struggle with generating the correct number of objects specified in a prompt. We introduce NUMINA , a training-free identify-then-guide framework for improved numerical alignment. NUMINA identifies prompt-layout inconsistencies by selecting discriminative self- and cross-attention heads to derive a countable latent layout. It then refines this layout conservatively and modulates cross-attention to guide regeneration. On the introduced CountBench, NUMINA improves counting accuracy by up to 7.4% on Wan2.1-1.3B, and by 4.9% and 5.5% on 5B and 14B models, respectively. Furthermore, CLIP alignment is improved while maintaining temporal consistency. These results demonstrate that structural guidance complements seed search and prompt enhancement, offering a practical path toward count-accurate text-to-video diffusion. The code is available at https://github.com/H-EmbodVis/NUMINA.
Engineering Breakdown
Plain English
This paper addresses a critical failure mode in text-to-video diffusion models: they frequently generate the wrong number of objects specified in prompts. The authors introduce NUMINA, a training-free framework that works by first identifying inconsistencies between the prompt description and the model's internal attention patterns, then using that information to guide the generation process toward correct object counts. On their new CountBench benchmark, NUMINA improves counting accuracy by 4.9-7.4% across different model sizes (1.3B to 14B parameters), while also maintaining video temporal consistency and improving CLIP alignment—all without requiring any model retraining.
Core Technical Contribution
The core innovation is a two-stage identify-then-guide approach that operates entirely at inference time without model fine-tuning. NUMINA selectively extracts discriminative self-attention and cross-attention heads to build a 'countable latent layout'—essentially a structured spatial representation that reflects how many objects the model currently intends to generate at each timestep. This layout is then refined conservatively (avoiding aggressive modifications that break temporal coherence) and used to modulate the cross-attention mechanism during the diffusion process, steering the model toward numerically consistent outputs. The key novelty is recognizing that attention patterns already contain exploitable signals about object counts, and that selective, conservative guidance can improve numerical alignment without the computational overhead of full retraining or the instability of aggressive prompt weighting.
How It Works
The system operates on video diffusion models (tested on Wan models) and takes as input a natural language prompt specifying object counts (e.g., 'three cats') and the model's internal attention maps during the diffusion timesteps. In the identify phase, NUMINA analyzes self-attention (which captures spatial layouts within the latent space) and cross-attention heads (which map text tokens to image regions) to detect which timesteps have misaligned object counts—essentially checking whether the model is attending to the right number of distinct spatial regions. It then constructs a latent layout representation by aggregating signals from the most informative attention heads. In the guide phase, this layout is refined using conservative update rules to avoid erasing temporal information from earlier diffusion steps, then used to dynamically adjust cross-attention weights during subsequent denoising iterations, nudging the generation toward the intended number of objects. The entire process happens at inference time, requiring no model weight updates or training data.
Production Impact
For teams deploying text-to-video systems, this approach directly addresses a user-facing problem: when users specify 'show me five dogs,' the model often generates three or seven. NUMINA can be integrated as a lightweight post-processing module in the generation pipeline with negligible latency overhead (attention head analysis is fast relative to the diffusion loop itself). The training-free nature is a major practical advantage—you don't need to collect counting datasets, retrain models, or maintain separate fine-tuned checkpoints; the method works immediately on any compatible diffusion model. The trade-off is that improvements are modest (4-7% absolute gains) and the approach is specifically tuned to counting/numeracy; it won't help with other semantic alignment problems like incorrect object types or spatial relationships. For production systems, you'd likely combine this with other alignment techniques rather than viewing it as a complete solution.
Limitations and When Not to Use This
The paper's scope is narrow: it specifically targets numerical alignment in object counting and may not generalize to other semantic consistency problems (e.g., ensuring 'red cars' produces red cars, not blue ones). The conservative refinement strategy, while good for temporal consistency, may leave some count errors uncorrected if the initial misalignment is severe. The approach assumes that attention heads contain reliable signals about object counts—this assumption likely holds for well-trained diffusion models but may fail in edge cases like heavily overlapping objects or abstract concepts where 'counting' is ambiguous. The evaluation is limited to a new benchmark (CountBench) the authors introduce; external validation on existing video generation benchmarks or user studies would strengthen claims. Additionally, the method requires identifying which attention heads are discriminative for counting, which may require per-model or per-domain tuning, and there's no guidance on how to adapt the approach to radically different model architectures.
Research Context
This work builds on the rapidly growing literature of inference-time guidance for diffusion models—prior approaches like classifier-free guidance and attention re-weighting have shown that you can steer generation without retraining, but most focus on image-level or semantic properties. NUMINA contributes a structured, attention-based approach specifically to a problem (numerical consistency in video) that hasn't been heavily studied despite being common in user feedback. The introduction of CountBench is a contribution to the evaluation landscape, addressing a gap where standard video benchmarks (often focused on temporal quality) don't measure numerical accuracy. This opens a research direction into fine-grained semantic consistency in video generation, which could spawn follow-up work on consistency checking for other attributes (colors, relationships, actions) and more sophisticated guidance mechanisms that operate on explicit symbolic representations rather than implicit attention patterns.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::