X2SAM: Any Segmentation in Images and Videos
| Authors | Hao Wang et al. |
| Year | 2026 |
| HF Upvotes | 21 |
| arXiv | 2605.00891 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated strong image-level visual understanding and reasoning, yet their pixel-level perception across both images and videos remains limited. Foundation segmentation models such as the SAM series produce high-quality masks, but they rely on low-level visual prompts and cannot natively interpret complex conversational instructions. Existing segmentation MLLMs narrow this gap, but are usually specialized for either images or videos and rarely support both textual and visual prompts in one interface. We introduce X2SAM, a unified segmentation MLLM that extends any-segmentation capabilities from images to videos. Given conversational instructions and visual prompts, X2SAM couples an LLM with a Mask Memory module that stores guided vision features for temporally consistent video mask generation. The same formulation supports generic, open-vocabulary, referring, reasoning, grounded conversation generation, interactive, and visual grounded segmentation across image and video inputs. We further introduce the Video Visual Grounded (V-VGD) segmentation benchmark, which evaluates whether a model can segment object tracks in videos from interactive visual prompts. With a unified joint training strategy over heterogeneous image and video datasets, X2SAM delivers strong video segmentation performance, remains competitive on image segmentation benchmarks, and preserves general image and video chat ability.
Engineering Breakdown
Plain English
X2SAM addresses a critical gap in AI vision systems: while Large Language Models excel at understanding images through conversation, and foundation segmentation models like SAM produce precise masks, no single system combines both capabilities across images AND videos with natural language instructions. The authors built X2SAM, a unified segmentation model that takes conversational prompts and visual cues as input and generates pixel-level segmentation masks for both static images and video sequences. The key innovation is a Mask Memory module that the LLM controls, allowing it to store and reason over guided visual features across frames, bridging the gap between high-level reasoning and low-level pixel prediction that neither MLLMs nor SAM can do alone.
Core Technical Contribution
The core novelty is the Mask Memory architecture that enables an LLM to act as a reasoning engine for pixel-level tasks across both modalities. Rather than fine-tuning a foundation segmentation model or bolting an LLM onto SAM, X2SAM couples them through a learnable module that stores contextual vision features that the LLM can query and update across multiple video frames. This design allows the system to maintain temporal consistency in videos while interpreting complex, multi-turn conversational instructions that reference both textual concepts and visual prompts (bounding boxes, clicks, masks). The approach is modality-agnostic—the same model architecture works for single images and video sequences without retraining.
How It Works
The system takes two inputs: (1) conversational instructions in natural language, and (2) optional visual prompts like bounding boxes or clicked points. The LLM tokenizes and processes the conversation to understand what the user is asking for. In parallel, a vision encoder processes image frames and extracts guided visual features based on the visual prompts (e.g., features at locations the user clicked). These features are stored in the Mask Memory module, which acts as a dynamic retrieval system—the LLM generates tokens that index into this memory to retrieve and synthesize task-specific features. A lightweight mask decoder then generates segmentation masks from the synthesized features. For videos, the Mask Memory maintains features across frames, allowing the LLM to enforce temporal consistency by reasoning about which objects persist and how they move between frames.
Production Impact
This enables interactive video annotation and real-time segmentation tasks that previously required either manual annotation, chaining separate tools (conversation model + SAM), or expensive per-domain fine-tuning. A production system could support user stories like 'segment all instances of the red car in this 10-minute dashcam video based on this text description' or 'refine the mask you created in frame 5 because I need to exclude the shadow.' The main production challenge is inference latency—running an LLM on every frame or for every user interaction could be slow, so you'd need batching strategies or selective frame processing. Another consideration is the Mask Memory's computational footprint: storing rich visual features for long videos requires managing GPU memory carefully, particularly when users ask to process 30-minute videos. The integration surface is cleaner than chaining separate models, but you lose the fine-grained control that comes from using SAM directly.
Limitations and When Not to Use This
The paper doesn't detail how the system handles true zero-shot scenarios or whether it requires fine-tuning on downstream domains—if real production data differs significantly from training data, performance likely degrades. Temporal consistency in videos is not guaranteed mathematically; the LLM's reasoning about frame-to-frame relationships is learned heuristically and may fail for occlusions, extreme motion, or objects that change dramatically between frames. The approach requires a capable LLM at its core, so the model's limitations (hallucinations, misunderstanding rare concepts, context length limits) directly constrain X2SAM's capabilities. Finally, there's no discussion of computational budgets: it's unclear whether this is practical for long video streams on edge devices or if it demands GPU resources comparable to running an LLM inference loop, which would be prohibitive for many production settings.
Research Context
X2SAM builds directly on the SAM (Segment Anything Model) family and multimodal LLMs like LLaVA, representing the next wave of unifying vision and language for dense prediction tasks. It advances beyond prior segmentation MLLMs like SEEM or Otter that typically target either images or videos in isolation, not both. The work sits at the intersection of three research directions: (1) expanding LLM capabilities from image classification to pixel-level tasks, (2) making foundation models like SAM more accessible via natural language without learning prompt engineering, and (3) extending image understanding to video understanding in a principled way. This likely opens follow-up work on temporal grounding, more efficient Mask Memory designs, and how to handle longer videos or multi-object reasoning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::