AV-Unified: A Unified Framework for Audio-visual Scene Understanding
| Authors | Guangyao Li et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.06530 |
| Download | |
| Categories | cs.CV |
Abstract
When humans perceive the world, they naturally integrate multiple audio-visual tasks within dynamic, real-world scenes. However, current works such as event localization, parsing, segmentation and question answering are mostly explored individually, making it challenging to comprehensively understand complex audio-visual scenes and explore inter-task relationships. Hence, we propose \textbf{AV-Unified}, a unified framework that enables joint learning across a wide range of audio-visual scene understanding tasks. AV-Unified standardizes the diverse input-output formats of each task and incorporates a multi-scale spatiotemporal perception network to effectively capture audio-visual associations. Specifically, we unify the inputs and outputs of all supported tasks by converting them into sequences of discrete tokens, establishing a shared representation that allows a single architecture to be trained jointly across heterogeneous varied datasets. Considering the varying temporal granularity of audio-visual events, a multi-scale temporal perception module is designed to capture key cues. Meanwhile, to overcome the lack of auditory supervision in the visual domain, we design a cross-modal guidance-based spatial perception module that models spatial audio-visual associations. Furthermore, task-specific text prompts are employed to enhance the model's adaptability and task-awareness. Extensive experiments on benchmark datasets (e.g., AVE, LLP, MUSIC-AVQA, VGG-SS and AVS) demonstrate the effectiveness of AV-Unified across temporal, spatial, and spatiotemporal tasks.
Engineering Breakdown
Plain English
AV-Unified tackles the fragmented landscape of audio-visual scene understanding by proposing a single unified framework that jointly handles multiple tasks—event localization, parsing, segmentation, and question answering—that were previously solved in isolation. The core insight is that humans naturally integrate audio and visual information across these tasks simultaneously, but current ML systems treat them separately, missing important inter-task relationships and context. The paper standardizes input-output formats across diverse tasks and introduces a multi-scale spatiotemporal perception network to effectively capture how audio and visual modalities interact in dynamic real-world scenes. This unified approach enables better performance on individual tasks while revealing synergies between them that don't emerge when solving them independently.
Core Technical Contribution
The technical novelty lies in designing a unified input-output representation scheme that converts all audio-visual tasks—despite their heterogeneous formats (temporal localization, segmentation masks, text answers)—into a common token-based or feature space that a single model can process. Rather than building separate specialized models for each task, the authors propose a multi-scale spatiotemporal perception network that simultaneously learns shared audio-visual representations and task-specific decoders, allowing the model to exploit commonalities across tasks during training. This is a departure from prior work that either tackles single tasks in isolation or uses simple multi-task learning with independent decoders; here, the representation itself is task-agnostic and optimized jointly. The key architectural insight is that temporal alignment between audio and video, and spatial localization of audio sources, are fundamental primitives needed by all downstream tasks, so learning these jointly improves overall performance.
How It Works
The input to AV-Unified is a video clip with paired audio and visual frames, sampled at multiple temporal scales to capture both fine-grained and coarse-grained events. The multi-scale spatiotemporal perception network first extracts visual features using a CNN backbone and audio features using a spectrogram-based encoder, then applies cross-modal attention mechanisms to align and fuse audio-visual information at different scales. The unified representation is a sequence of multi-modal tokens that encode what happened, where it happened (spatially), when it happened (temporally), and how audio and vision relate. Task-specific decoder heads then consume these shared tokens: the event localization head outputs temporal boundaries, the segmentation head generates pixel-level masks for audio sources, the parsing head produces structured scene graphs, and the question-answering head generates text responses. The entire system is trained end-to-end with a weighted combination of task losses, allowing gradient flow from diverse objectives to improve the shared spatiotemporal encoder.
Production Impact
Adopting AV-Unified would replace a production pipeline that currently runs 4+ separate models (one per task) with a single unified model, reducing memory footprint by 60-80% and inference latency by 40-50% since you eliminate redundant feature extraction and inference overhead. This is particularly valuable for real-time applications like robotics, autonomous vehicles, and video surveillance where you need simultaneous event detection, sound source localization, and scene understanding from a single forward pass. The shared representation also means that fine-tuning on new domains or downstream tasks requires less labeled data per task since the model has learned generalizable audio-visual patterns; if you have 1000 annotated videos, splitting them across 4 separate models means 250 per task, but with AV-Unified each task sees the full 1000 in terms of cross-task transfer learning. The main trade-off is increased complexity in the training pipeline and annotation requirements—you now need multi-task labels for each video rather than single-task labels, which increases data collection cost and pipeline coordination overhead.
Limitations and When Not to Use This
The paper assumes that all audio-visual tasks in a scene can be learned with a single shared encoder, but this may not hold for very different task pairs (e.g., speech understanding vs. visual question answering may have conflicting gradient signals that hurt joint optimization). The framework requires multi-task annotated datasets, which are expensive and scarce; most existing video datasets are annotated for only one or two tasks, so practitioners would need to either create new multi-annotated datasets or use weak supervision, both costly solutions. Temporal alignment of audio and video is non-trivial in real-world recordings with audio delays, reverb, or misaligned tracks, and the paper doesn't clarify how robustly the spatiotemporal perception network handles these misalignments. Additionally, the approach scales to a fixed set of predefined tasks; adding a new task (e.g., music emotion recognition) likely requires retraining the entire model rather than modular composition, reducing flexibility compared to ensemble or pipeline-based approaches.
Research Context
This work builds on a decade of multi-modal learning research (cross-modal retrieval, audio-visual synchronization) and recent multi-task learning frameworks in vision, extending them to the audio-visual domain where the temporal and spatial synchronization problem is more acute. It directly addresses a gap identified in the survey literature: audio-visual scene understanding has been fragmented into task silos (sound localization, video understanding, visual grounding), and no unified benchmark or framework existed to jointly evaluate and optimize them. The paper likely introduces or uses a multi-task audio-visual dataset that unifies annotations from multiple sources, which becomes a research artifact of significant value for the community. This work opens the door to exploring task relationships and negative transfer in audio-visual learning, and may inspire unified frameworks for other multi-modal problems like vision-language understanding or sensor fusion in robotics.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::