The Evolution of Multimodal AI
How vision and language went from separate models to a single unified architecture.
The multimodal story runs on two parallel tracks: understanding (models that see and reason) and generation (models that create images from text). This roadmap covers both tracks in chronological order, showing how they converged into unified models that can both perceive and generate across modalities.
Read in order. Each paper fixes a failure mode exposed by the previous one. Click any card to collapse it.
CLIP — Contrastive Image-Text Pretraining
Radford, Kim, Hallacy et al. (OpenAI)
Learning Transferable Visual Models From Natural Language Supervision
Trains an image encoder and text encoder jointly using contrastive loss on 400M image-text pairs from the internet. At inference, classifies images by comparing image embeddings to text embeddings of candidate labels — zero-shot, no fine-tuning. Matches supervised ResNet-50 on ImageNet.
Key insight: Natural language is the most flexible label space. By aligning image and text embeddings, CLIP enables zero-shot transfer to any visual task you can describe in text. This is the foundational insight for all vision-language models: the embedding space is shared.
DALL-E — Text-to-Image Generation
Ramesh, Pavlov, Goh et al. (OpenAI)
Zero-Shot Text-to-Image Generation
Trains a transformer on text-image pairs to autoregressively generate image tokens conditioned on text tokens. Demonstrates that a large enough transformer can generate coherent, creative images from arbitrary text descriptions — zero-shot, at scale.
Key insight: The same transformer architecture that models text can model images — if you tokenise images (using dVAE) and train on enough paired data. This conceptual unification is more important than the specific model, which was quickly superseded by diffusion models.
Stable Diffusion — Latent Diffusion
Rombach, Blattmann, Lorenz et al. (LMU Munich / Runway)
High-Resolution Image Synthesis with Latent Diffusion Models
Moves the diffusion process into a compressed latent space (trained by a VQ-regularised autoencoder) rather than pixel space. Cross-attention layers condition the denoising on text embeddings. Reduces compute by orders of magnitude while maintaining quality. This is Stable Diffusion.
Key insight: You don't need to denoise in pixel space — you can denoise in a compressed representation and decode at the end. Latent diffusion makes high-resolution generation feasible on consumer hardware. The open-source release of Stable Diffusion democratised image generation.
Flamingo — Few-Shot Visual Understanding
Alayrac, Donahue, Luc et al. (DeepMind)
Flamingo: a Visual Language Model for Few-Shot Learning
Bridges a pretrained vision encoder and a pretrained language model using cross-attention layers (Perceiver Resampler). Freezes both pretrained components and trains only the bridge. Achieves few-shot performance competitive with fine-tuned models across VQA, captioning, and classification tasks.
Key insight: You don't need to retrain the vision encoder or the language model to connect them — you just need a learned bridge. Flamingo's architectural insight (freeze both, train the bridge) becomes the template for most open-source VLMs that follow.
InstructBLIP — Instruction-Tuned VLM
Dai, Li, Li et al. (Salesforce Research)
InstructBLIP: Towards General Visual-Language Models with Instruction Tuning
Applies instruction tuning to BLIP-2, training on a diverse collection of vision-language tasks reformatted as instructions. Introduces instruction-aware Query Transformer (Q-Former) that extracts image features relevant to the specific instruction. Strong zero-shot generalisation.
Key insight: A VLM that follows instructions is dramatically more useful than one that doesn't. Instruction tuning — the same technique that made GPT-3 into InstructGPT — transfers directly to multimodal models. The Q-Former's instruction-awareness is the key innovation.
LLaVA — Open-Source VLM
Liu, Li, Wu, Lee (University of Wisconsin-Madison / Microsoft)
Visual Instruction Tuning
Uses GPT-4 to generate multimodal instruction-following data from image captions and bounding box annotations. Connects a CLIP visual encoder to LLaMA using a simple linear projection layer. Achieves 85.1% relative score to GPT-4 on visual tasks with dramatically less compute. Fully open-source.
Key insight: You can use a strong LLM to generate training data for a weaker multimodal model. LLaVA proves that the architecture doesn't need to be complex — a linear projection between CLIP and an LLM is enough with good instruction data. The open release enabled the entire open VLM research ecosystem.
SAM — Universal Image Segmentation
Kirillov, Mintun, Ravi et al. (Meta AI Research)
Segment Anything
Trains a promptable segmentation model on 1 billion masks across 11 million images. Given any prompt — point, box, text, or mask — SAM segments the corresponding object. Zero-shot transfer to unseen tasks and domains. The first foundation model for segmentation.
Key insight: The foundation model paradigm that worked for language (pretrain big, prompt to adapt) works for visual segmentation too. SAM's promptability means you can integrate it into any pipeline without fine-tuning. This is now standard infrastructure for computer vision applications.
GPT-4V — Multimodal at Frontier Scale
OpenAI
GPT-4 Technical Report
Extends GPT-4 with vision capabilities. Accepts interleaved text and image inputs. Demonstrates near-human performance on academic visual benchmarks, complex document understanding, chart interpretation, and multi-step visual reasoning. Sets the capability ceiling that all other VLMs are measured against.
Key insight: At sufficient scale, the gap between language and vision closes. GPT-4V doesn't just describe images — it reasons about them, catches errors in diagrams, and solves visual puzzles. This resets expectations for what VLMs should be capable of.
Engineering note: Architecture details are not disclosed. The technical report is deliberately sparse. Evaluate capabilities empirically rather than relying on the paper.
Gemini — Natively Multimodal
Gemini Team (Google DeepMind)
Gemini: A Family of Highly Capable Multimodal Models
Designed from the ground up as a multimodal model — not a language model with vision bolted on. Processes text, images, audio, and video natively in a single model. Outperforms GPT-4V on several benchmarks including MMMU. Available in three sizes (Ultra, Pro, Nano).
Key insight: Native multimodality vs adapted multimodality is a fundamental architectural distinction. When vision is part of the pretraining objective rather than an adapter, the model develops richer cross-modal representations. Gemini Nano demonstrates that multimodal capability can fit on-device.
The Two Tracks Today
Understanding track (see → reason):
CLIP → Flamingo → InstructBLIP → LLaVA → GPT-4V / Gemini
Generation track (text → create):
DALL-E → Stable Diffusion → DALL-E 3 → Stable Diffusion XL → Flux
Both tracks are now converging into unified models that both perceive and generate. Watch this space.
Related