VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images
| Authors | Guanyu Zhou et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.09531 |
| Download | |
| Categories | cs.CV, cs.AI, cs.CL |
Abstract
Vision-language models (VLMs) still struggle with visual perception tasks such as spatial understanding and viewpoint recognition. One plausible contributing factor is that natural image datasets provide limited supervision for low-level visual skills. This motivates a practical question: can targeted synthetic supervision, generated from only a task keyword such as Depth Order, address these weaknesses? To investigate this question, we introduce VisionFoundry, a task-aware synthetic data generation pipeline that takes only the task name as input and uses large language models (LLMs) to generate questions, answers, and text-to-image (T2I) prompts, then synthesizes images with T2I models and verifies consistency with a proprietary VLM, requiring no reference images or human annotation. Using VisionFoundry, we construct VisionFoundry-10K, a synthetic visual question answering (VQA) dataset containing 10k image-question-answer triples spanning 10 tasks. Models trained on VisionFoundry-10K achieve substantial improvements on visual perception benchmarks: +7% on MMVP and +10% on CV-Bench-3D, while preserving broader capabilities and showing favorable scaling behavior as data size increases. Our results suggest that limited task-targeted supervision is an important contributor to this bottleneck and that synthetic supervision is a promising path toward more systematic training for VLMs.
Engineering Breakdown
Plain English
VisionFoundry addresses a critical weakness in vision-language models: they struggle with low-level visual perception tasks like spatial understanding and viewpoint recognition, partly because natural image datasets don't provide enough supervision for these skills. The paper proposes an automated synthetic data generation pipeline that takes only a task name (like 'Depth Order') as input, uses an LLM to generate questions/answers and image prompts, synthesizes images with a text-to-image model, and validates consistency using a VLM—all without manual annotation or reference images. This approach aims to improve VLM performance on visual perception benchmarks by injecting targeted synthetic supervision where natural data is sparse.
Core Technical Contribution
The core novelty is a fully automated, task-aware synthetic data generation pipeline that requires minimal human input—just a task keyword—to generate high-quality training data for VLMs. Unlike prior synthetic data approaches that often require reference images, detailed annotations, or manual quality control, VisionFoundry uses LLMs to generate diverse questions and T2I prompts, then leverages a VLM itself to verify consistency and filter low-quality synthetic examples. This creates a closed-loop system where the model teaches itself on targeted weaknesses without human-in-the-loop annotation, making it scalable across multiple visual perception tasks.
How It Works
The pipeline operates in three stages: (1) Input a task name like 'Depth Order' into an LLM, which generates a diverse set of question-answer pairs and corresponding T2I prompts that specifically target that skill; (2) Feed the generated T2I prompts into a text-to-image model (likely Stable Diffusion or similar) to create synthetic images paired with QA data; (3) Filter the generated dataset by running it through a proprietary VLM that verifies consistency—rejecting examples where the generated answer doesn't match what the VLM actually sees in the image, ensuring only coherent examples are retained. The validated synthetic dataset then trains or fine-tunes the target VLM on the specific perception task. The key insight is that using the LLM to generate both captions and prompts ensures semantic alignment between questions and images, while VLM-based verification acts as an automatic quality gate.
Production Impact
For production systems, this approach solves the expensive problem of collecting and annotating training data for niche visual perception tasks—instead of hiring annotators to label 10k images for spatial reasoning or viewpoint estimation, engineers can generate diverse synthetic data in hours. This directly reduces time-to-train and eliminates annotation bottlenecks when deploying VLMs to new perception tasks. The trade-off is computational cost: you need to run an LLM inference per task (minimal), a T2I model to generate images (moderate cost), and potentially multiple VLM inference passes for validation (can be expensive at scale). Integration is straightforward—the synthetic data is standard image+text pairs that fit existing VLM training pipelines. The main production concern is whether synthetic data from T2I models introduces distribution shift that doesn't transfer to real-world images, which would limit performance gains.
Limitations and When Not to Use This
The paper doesn't address whether synthetic supervision actually transfers well to real-world test sets—synthetic images from T2I models have visual artifacts and statistical properties that differ from natural images, so improvements on synthetic-trained models may not generalize. The approach assumes you have access to a capable LLM and proprietary VLM for validation, limiting reproducibility and accessibility (the validation VLM is proprietary, not open-source). The paper doesn't discuss computational cost of the full pipeline or how it scales to generating massive datasets; synthetic image generation and VLM validation can be slow at scale. Finally, the abstract doesn't provide concrete results on benchmark improvements, making it unclear whether this actually outperforms simpler approaches like just collecting more natural data or using standard fine-tuning.
Research Context
This work builds on the growing recognition that VLMs have significant gaps in low-level visual understanding despite strong performance on high-level tasks. It extends prior work on synthetic data for vision models and leverages recent advances in text-to-image generation and LLMs as reasoning engines. The paper contributes to the emerging paradigm of using LLMs and generative models as tools for automating data engineering workflows, rather than just for final prediction. It opens a research direction around task-specific synthetic supervision and automated quality filtering, which could be applied to other modalities and multimodal learning settings.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::