Audio-Visual Intelligence in Large Foundation Models
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 26 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | You Qin et al. |
| Year | 2026 |
| HF Upvotes | 26 |
| arXiv | 2605.04045 |
| Download | |
| Code | https://github.com/JavisVerse/Awesome-AVI |
Abstract
Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence, bridging auditory and visual modalities to enable machines that can perceive, generate, and interact in the multimodal real world. In the era of large foundation models, joint modeling of audio and vision has become increasingly crucial, i.e., not only for understanding but also for controllable generation and reasoning across dynamic, temporally grounded signals. Recent advances, such as Meta MovieGen and Google Veo-3, highlight the growing industrial and academic focus on unified audio-vision architectures that learn from massive multimodal data. However, despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent taxonomies, and heterogeneous evaluation practices that impede systematic comparison and knowledge integration. This survey provides the first comprehensive review of AVI through the lens of large foundation models. We establish a unified taxonomy covering the broad landscape of AVI tasks, ranging from understanding (e.g., speech recognition, sound localization) to generation (e.g., audio-driven video synthesis, video-to-audio) and interaction (e.g., dialogue, embodied, or agentic interfaces). We synthesize methodological foundations, including modality tokenization, cross-modal fusion, autoregressive and diffusion-based generation, large-scale pretraining, instruction alignment, and preference optimization. Furthermore, we curate representative datasets, benchmarks, and evaluation metrics, offering a structured comparison across task families and identifying open challenges in synchronization, spatial reasoning, controllability, and safety. By consolidating this rapidly expanding field into a coherent framework, this survey aims to serve as a foundational reference for future research on large-scale AVI.
Engineering Breakdown
Plain English
This paper surveys the emerging field of Audio-Visual Intelligence (AVI) in large foundation models—systems that jointly process sound and video to understand and generate multimodal content. The authors document recent industrial breakthroughs like Meta's MovieGen and Google's Veo-3, which use unified architectures trained on massive multimodal datasets to handle tasks spanning understanding, controllable generation, and temporal reasoning across audio-visual signals.
Key Engineering Insight
The critical insight is that audio and vision must be modeled jointly in foundation models rather than as separate streams—this unified approach unlocks emergent capabilities in temporal grounding, generation control, and reasoning that neither modality provides alone.
Why It Matters for Engineers
For engineers building production systems, this matters because multimodal AI is moving from research to product (video generation, content understanding, real-time interaction). Understanding how to architect unified audio-vision models, handle temporal alignment, and evaluate across inconsistent benchmarks directly impacts feasibility and performance of systems like video search, content moderation at scale, and interactive AI assistants.
Research Context
Audio-visual AI previously existed as fragmented research (separate speech recognition, video understanding, cross-modal retrieval tasks with different evaluation metrics). This survey consolidates that landscape as foundation models emerge capable of end-to-end multimodal reasoning, enabling the next generation: coherent audio-visual generation and reasoning that maintains temporal consistency—something prior siloed approaches couldn't achieve.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::