AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation
| Authors | Ziwei Zhou et al. |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2604.08540 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Text-to-Audio-Video (T2AV) generation is rapidly becoming a core interface for media creation, yet its evaluation remains fragmented. Existing benchmarks largely assess audio and video in isolation or rely on coarse embedding similarity, failing to capture the fine-grained joint correctness required by realistic prompts. We introduce AVGen-Bench, a task-driven benchmark for T2AV generation featuring high-quality prompts across 11 real-world categories. To support comprehensive assessment, we propose a multi-granular evaluation framework that combines lightweight specialist models with Multimodal Large Language Models (MLLMs), enabling evaluation from perceptual quality to fine-grained semantic controllability. Our evaluation reveals a pronounced gap between strong audio-visual aesthetics and weak semantic reliability, including persistent failures in text rendering, speech coherence, physical reasoning, and a universal breakdown in musical pitch control. Code and benchmark resources are available at http://aka.ms/avgenbench.
Engineering Breakdown
Plain English
This paper introduces AVGen-Bench, a new benchmark for evaluating text-to-audio-video generation systems across 11 real-world categories with high-quality prompts. The core problem is that existing evaluation approaches treat audio and video separately or use coarse embedding similarity metrics, missing the fine-grained joint correctness needed for real-world applications. The authors propose a multi-granular evaluation framework combining lightweight specialist models with Multimodal Large Language Models (MLLMs) to assess everything from perceptual quality to semantic controllability. Their evaluation reveals a significant gap between strong audio-visual aesthetics and actual fine-grained semantic alignment, suggesting current T2AV systems are missing important correctness properties.
Core Technical Contribution
The core novelty is a task-driven benchmark and evaluation framework specifically designed for joint audio-video generation that goes beyond isolated modality assessment. Rather than comparing embeddings or evaluating modalities separately, AVGen-Bench uses a multi-granular approach: lightweight specialist models handle domain-specific quality checks (e.g., audio clarity, visual coherence) while MLLMs assess semantic alignment between prompt, audio, and video outputs. The framework captures what prior work missed—that audio and video must be jointly correct relative to the user's intent, not just individually high-quality. This represents a shift from single-metric evaluation to task-aware assessment that matches how humans actually use generative media systems.
How It Works
AVGen-Bench operates as a three-tier evaluation pipeline: (1) Input is a text prompt from one of 11 real-world categories plus a candidate T2AV system output (separate audio and video streams). (2) Lightweight specialist models evaluate modality-specific properties—audio metrics check clarity, speech intelligibility, and sound design quality while vision models assess visual coherence, motion consistency, and aesthetic appeal. (3) MLLMs then perform joint semantic assessment, examining whether the audio and video together satisfy the original prompt's intent, checking for fine-grained controllability like specific instructions about tone, style, or content. The framework outputs multiple scores (per-modality quality, cross-modal alignment, semantic coverage) rather than a single number, enabling detailed diagnosis of where systems fail. This multi-granular design allows practitioners to understand not just that a system failed, but specifically why—whether from weak audio, weak video, or misalignment between them.
Production Impact
Adopting this evaluation framework transforms how T2AV system developers debug and improve their models. In current production pipelines, teams typically use proxy metrics (audio SNR, video LPIPS, CLIP similarity) that don't correlate well with user satisfaction; this benchmark provides task-aligned evaluation that surfaces real issues. The multi-granular approach enables targeted optimization—if the gap is in audio-video alignment rather than individual modality quality, your training strategy changes completely. However, there are trade-offs: running MLLM-based evaluation is computationally expensive (likely 5-10x slower than simple embedding metrics) and requires maintaining multiple specialist models, increasing operational complexity. For teams shipping T2AV systems, this means allocating significant compute to evaluation infrastructure, but the payoff is catching semantic failures that users will immediately notice—misaligned audio and video, or outputs that technically look good but miss the prompt intent. Integration into CI/CD would require careful batching and caching strategies to keep iteration cycles reasonable.
Limitations and When Not to Use This
The benchmark is limited to 11 predefined real-world categories, which may not capture all use cases or emerging prompt patterns—generalization to novel or niche domains is unclear. The reliance on MLLMs for semantic assessment inherits their biases and limitations; if the MLLM misunderstands a prompt or has cultural blindspots, those errors propagate into the evaluation itself, and there's no guarantee the MLLM's assessment matches human preference. The paper doesn't address scalability for continuous generation—maintaining ground truth labels for millions of generations is infeasible, so this benchmark works best as a development/validation tool rather than production monitoring. Finally, the evaluation framework assumes you have access to both audio and video outputs; for systems generating them jointly or sequentially, the specialist model decomposition may not apply cleanly. The paper also doesn't deeply explore how evaluation quality degrades with modality degradation (e.g., what happens when audio is generated from video, or vice versa, rather than both from text).
Research Context
This work sits at the intersection of multimodal evaluation and generative modeling benchmarking. Prior benchmarks like AudioCaps or ActivityNet evaluate audio or video independently, while recent work on audio-visual alignment (e.g., using contrastive learning) treats alignment as a proxy for quality without task-level validation. AVGen-Bench builds on the insight that realistic media generation is inherently task-driven—users want audio and video that jointly satisfy their intent, not isolated modalities that score well by embedding distance. The paper extends evaluation methodology from single-modality generation (like text-to-image benchmarks such as COCO or PartiPrompts) to the harder multimodal case, opening research directions in joint generative modeling, prompt engineering for multimodal systems, and understanding why systems fail at audio-video coherence. This positions AVGen-Bench as infrastructure that will likely be adopted by other T2AV research groups, similar to how ImageNet and COCO became standard references.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::