Embedding Models - The Landscape

Reading time: 25 min | Relevance: AI Engineer, ML Engineer, Research Engineer

Picking the Wrong Model for Production

Your RAG pipeline retrieves top-5 chunks, your LLM generates responses, and early testing looks great. Then you deploy. Users report the chatbot misses obvious answers - questions about your product's pricing return marketing copy about your company's history. Your development dataset had clean, topically-matched query-document pairs. Production has messier, more specific queries that don't share vocabulary with your documentation.

You check your embedding model: you used bert-base-uncased embeddings, the default in many tutorials. BERT embeddings are not optimized for retrieval - they're designed for fine-tuning on classification and NER tasks. The retrieval quality is fundamentally limited by this choice, and improving it doesn't require changing any other part of your pipeline.

The embedding model is the most consequential single decision in a semantic search or RAG system. The right model for your use case depends on: your domain, your query types (symmetric vs asymmetric retrieval), your language requirements, your latency budget, and whether you control the embedding pipeline or use an API. This lesson maps the full landscape so you can make an informed choice.

Why This Exists - The Gap Between Pre-Training and Retrieval

Pre-trained language models like BERT are excellent at understanding text. But their training objectives - predicting masked tokens, predicting next sentences - don't directly optimize the embedding space for retrieval.

Consider two documents:

"The treatment for Type 2 diabetes includes metformin and lifestyle changes."
"Controlling blood sugar with medication and diet for adult-onset diabetes."

These are semantically very similar. A human would immediately recognize them as nearly equivalent. But a BERT embedding comparison gives a cosine similarity of around 0.62 - lower than many unrelated documents that happen to share surface vocabulary.

The problem: BERT was not trained to make semantically similar documents have similar embeddings. It was trained to predict masked tokens, which requires understanding individual word contexts, not document-level meaning.

The solution: fine-tune the pre-trained model on a task that directly rewards good embedding space geometry for retrieval. This is what all modern embedding models do, using various forms of contrastive learning.

Historical Context

2019 - Reimers and Gurevych publish Sentence-BERT (SBERT). The key innovation: a Siamese network that passes two sentences through BERT simultaneously and optimizes a similarity loss. First practically useful sentence embeddings for retrieval.

2021 - Gao, Yao, and Chen publish SimCSE ("Simple Contrastive Learning of Sentence Embeddings"). Shows that using the same sentence twice with different dropout masks as positive pairs, with in-batch negatives, produces better embeddings than more complex augmentation strategies.

2022 - Wang et al. publish E5 ("Text Embeddings by Weakly Supervised Contrastive Pre-Training"). Scale contrastive pre-training to 1.3 billion weak supervision pairs from the web (using title-body pairs, query-passage pairs from CCPairs).

2023 - BGE (BAAI General Embedding) and GTE (General Text Embeddings by Alibaba) release competitive open-source models, approaching OpenAI API quality.

2024 - OpenAI releases text-embedding-3 series with Matryoshka training. Voyage AI and Cohere Embed v3 release commercial alternatives that outperform OpenAI on many benchmarks. The MTEB leaderboard becomes the standard comparison tool.

Contrastive Learning: The Core Training Paradigm

Modern embedding models are trained with contrastive learning - a training approach that explicitly shapes the geometry of the embedding space.

The core idea

Given a batch of (anchor, positive, negative) triplets:

Anchor: The query or reference text
Positive: A semantically similar text (the correct retrieval target)
Negative: A semantically dissimilar text (an incorrect retrieval target)

Contrastive loss pushes anchor-positive pairs together and anchor-negative pairs apart:

$\mathcal{L}_{\text{triplet}} = \max(0, d(a, p) - d(a, n) + \text{margin})$

where $d(\cdot, \cdot)$ is a distance function (often $1 - \cos(\cdot, \cdot)$ ).

Or using the InfoNCE (Noise Contrastive Estimation) loss, which scales better:

$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(a, p) / \tau)}{\sum_{j=1}^{N} \exp(\text{sim}(a, n_j) / \tau)}$

where $\tau$ is a temperature hyperparameter and the denominator sums over all negatives (including in-batch negatives).

In-batch negatives

The most efficient contrastive training strategy uses in-batch negatives: for each anchor in the batch, all other examples' positives serve as negatives. A batch of $N$ examples provides $N$ positive pairs and $N(N-1)$ negative pairs - dramatically more efficient than explicit triplet construction.

import torch
import torch.nn.functional as F

def info_nce_loss(
    query_embeddings: torch.Tensor,  # (batch, dim)
    doc_embeddings: torch.Tensor,    # (batch, dim)
    temperature: float = 0.05
) -> torch.Tensor:
    """
    InfoNCE loss for contrastive learning with in-batch negatives.
    query_embeddings[i] should be similar to doc_embeddings[i].
    query_embeddings[i] should be dissimilar to doc_embeddings[j] for j != i.
    """
    # Normalize embeddings to unit sphere
    q = F.normalize(query_embeddings, dim=-1)  # (batch, dim)
    d = F.normalize(doc_embeddings, dim=-1)    # (batch, dim)

    # Compute all pairwise similarities
    sim_matrix = q @ d.T / temperature  # (batch, batch)

    # Labels: diagonal entries are positives
    labels = torch.arange(q.shape[0], device=q.device)

    # Cross-entropy: maximize similarity with positive, minimize with all negatives
    loss = F.cross_entropy(sim_matrix, labels)

    # Also compute in reverse direction (doc → query)
    loss += F.cross_entropy(sim_matrix.T, labels)

    return loss / 2.0


def compute_retrieval_accuracy(
    query_embeddings: torch.Tensor,
    doc_embeddings: torch.Tensor,
) -> float:
    """
    Measure alignment: fraction of queries where the correct
    document is the top-1 nearest neighbor.
    """
    q = F.normalize(query_embeddings, dim=-1)
    d = F.normalize(doc_embeddings, dim=-1)
    sim = q @ d.T  # (batch, batch)
    predictions = sim.argmax(dim=-1)
    labels = torch.arange(q.shape[0], device=q.device)
    return (predictions == labels).float().mean().item()

SBERT: The Inflection Point

Sentence-BERT (Reimers & Gurevych, 2019) is the model that made practical semantic search possible. Two key innovations:

1. Siamese training: Two identical BERT models (shared weights) process two sentences in parallel. The pooled embeddings are compared with a similarity objective. This directly optimizes the embedding space for similarity comparison, unlike the original BERT NSP objective.

2. Efficient inference: SBERT embeddings for individual sentences can be pre-computed and stored. At query time, only the query is embedded, and similarity is computed with a simple dot product. This reduces semantic search from $O(n)$ BERT forward passes to $O(1)$ - transforming a research technique into a practical system.

The training used the SNLI (Stanford Natural Language Inference) and MultiNLI datasets, which contain hundreds of thousands of sentence pairs labeled as entailment, contradiction, or neutral. Entailment pairs → high similarity, contradiction pairs → low similarity.

SBERT performance gap: On STSbenchmark (a standard semantic similarity benchmark), raw BERT achieved 0.49 Pearson correlation. SBERT achieved 0.86 - a massive improvement that made dense retrieval competitive with BM25 for the first time.

SimCSE: Simple Contrastive Sentence Embeddings

Gao et al. (2021) published a surprising finding: the simplest possible augmentation strategy for contrastive learning - passing the same sentence through the model twice with different dropout masks - produces better sentence embeddings than more complex augmentation strategies.

Unsupervised SimCSE

For each sentence, create two "views" by enabling dropout:

Pass sentence through BERT once with dropout enabled → embedding $z_i$
Pass same sentence through BERT again with different dropout → embedding $z_i^+$
These form a positive pair

In-batch negatives: all other sentences in the batch are negatives.

def simcse_unsupervised_loss(
    model,
    input_ids: torch.Tensor,
    attention_mask: torch.Tensor,
    temperature: float = 0.05,
) -> torch.Tensor:
    """
    Unsupervised SimCSE: same sentence twice with different dropout.
    The model must be in training mode for dropout to vary between passes.
    """
    # model.training should be True for dropout to vary between passes
    # Pass same batch twice
    emb1 = model(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
    emb2 = model(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state

    # Mean pool
    mask = attention_mask.unsqueeze(-1).float()
    emb1 = (emb1 * mask).sum(1) / mask.sum(1)
    emb2 = (emb2 * mask).sum(1) / mask.sum(1)

    return info_nce_loss(emb1, emb2, temperature)

Why dropout augmentation works

Dropout randomly zeroes out neurons, creating different computational paths through the model. Two passes of the same sentence with different dropout masks produce slightly different embeddings - close in space (because the sentence is the same) but not identical (because different neurons were active). This provides soft positive pairs where the model must learn that the "true content" of a sentence is invariant to specific neuron activations.

The key insight: unlike other augmentation strategies (word deletion, synonym replacement, back-translation), dropout augmentation is completely content-preserving - the information in the embeddings only changes due to the stochasticity of dropout, not due to any semantic modification.

Supervised SimCSE

With supervised data (NLI dataset), SimCSE uses entailment pairs as positives and contradiction pairs as hard negatives:

$\mathcal{L} = -\log \frac{e^{\text{sim}(h_i, h_i^+) / \tau}}{e^{\text{sim}(h_i, h_i^+) / \tau} + e^{\text{sim}(h_i, h_i^-) / \tau} + \sum_{j \neq i} e^{\text{sim}(h_i, h_j^+) / \tau}}$

Supervised SimCSE achieves state-of-the-art on semantic similarity benchmarks at time of publication, showing that label signal + in-batch negatives is highly effective.

E5: Scaling Contrastive Pre-Training

Wang et al. (2022) at Microsoft published E5 (Text Embeddings by Weakly Supervised Contrastive Pre-Training), scaling contrastive training to 1.3 billion text pairs from the web.

Data sources

E5 uses weakly supervised data - pairs that are likely related without human annotation:

CC-Pairs: 270M paired title-body pairs from Common Crawl web text
Reddit: Thread pairs (post → comment)
MS MARCO: Bing search query-passage pairs
Wikipedia: Sections from the same article

This weak supervision is noisy (not all title-body pairs are semantically similar) but at 1.3 billion scale, the signal dominates the noise.

Asymmetric encoding

E5 uses special query and document prefixes:

Queries: prepend "query: " to the text
Documents: prepend "passage: " to the text

This tells the model to produce asymmetric embeddings: query embeddings are optimized to match document embeddings, not other queries. This matches the actual retrieval setting, where queries are short questions and documents are longer passages.

from sentence_transformers import SentenceTransformer

# E5 models require query/passage prefixes
model = SentenceTransformer("intfloat/e5-large-v2")

def embed_queries(queries: list[str]) -> list[list[float]]:
    """Embed queries with the required 'query: ' prefix."""
    prefixed = [f"query: {q}" for q in queries]
    return model.encode(prefixed, normalize_embeddings=True).tolist()

def embed_documents(docs: list[str]) -> list[list[float]]:
    """Embed documents with the required 'passage: ' prefix."""
    prefixed = [f"passage: {d}" for d in docs]
    return model.encode(prefixed, normalize_embeddings=True).tolist()

# Critical: if you forget the prefix, retrieval quality drops significantly
# E5 without prefix: ~60% of optimal retrieval quality
# E5 with prefix: full retrieval quality

BGE: BAAI General Embedding

BGE (released by the Beijing Academy of Artificial Intelligence, 2023) became one of the most competitive open-source embedding models, regularly appearing at the top of the MTEB leaderboard.

BGE's training approach adds several innovations over E5:

RetroMAE pre-training: A retrieval-oriented masked autoencoder pre-training objective that produces better initial representations for retrieval before contrastive fine-tuning.

Hard negative mining: BGE trains with hard negatives - documents that are semantically close to the query but not actually relevant. Hard negatives are much more informative than random negatives. Mining them requires an initial embedding model to find semantically similar but non-relevant passages.

FlagEmbedding instruction format: BGE-M3 (the multilingual version) accepts instructions like "Retrieve a legal document that resolves the following query:" - allowing the model to specialize its embedding for the downstream task.

GTE: General Text Embeddings

GTE (Alibaba, 2023) is another competitive open-source family. GTE-large and GTE-base are strong alternatives to BGE, particularly for multilingual settings.

Key GTE contributions:

Multi-stage training: pre-training on 800M pairs, then fine-tuning on task-specific data
Efficient architecture: GTE achieves competitive performance at smaller model sizes
Strong multilingual performance: GTE-multilingual handles 100+ languages

Commercial Embedding APIs

For teams that don't want to manage embedding infrastructure, commercial APIs provide high-quality embeddings with minimal setup:

OpenAI text-embedding-3-small:

1536 dimensions
Matryoshka-trained: can truncate to 256–1536 dimensions
Strong MTEB performance, especially on English retrieval
$0.02/million tokens

OpenAI text-embedding-3-large:

3072 dimensions
Best quality in the OpenAI lineup
Matryoshka-trained: can truncate to 256–3072 dimensions
$0.13/million tokens

Voyage AI voyage-3:

1024 dimensions
Consistently outperforms OpenAI on MTEB retrieval tasks
Designed for long document retrieval (up to 32k context)
~$0.06/million tokens

Cohere Embed v3:

1024 dimensions
Multimodal: embeds images and text in the same space
Strong on multilingual tasks
$0.10/million tokens

Jina Embeddings v3:

1024 dimensions, up to 8192 context
Strong on long document retrieval
Available via API and self-hosted

The MTEB Leaderboard

MTEB (Massive Text Embedding Benchmark, Muennighoff et al. 2022) is the standard comparison tool for embedding models. It evaluates models across 56 datasets and 8 task types.

How MTEB works

For each task type, MTEB evaluates the model on standardized datasets:

Retrieval (most important for RAG): Uses datasets like MSMARCO, HotpotQA, NFCorpus, ArguAna. Metric: nDCG@10 (Normalized Discounted Cumulative Gain at rank 10). This is the primary metric for search applications.

Clustering: Uses datasets covering topics from ArXiv papers to Reddit posts. Metric: V-measure (harmonic mean of homogeneity and completeness).

Classification: Uses datasets for sentiment analysis, topic classification, etc. Metric: accuracy or F1.

Semantic Textual Similarity (STS): Uses STS-B, SICK-R, and similar datasets. Metric: Spearman correlation between model similarity scores and human ratings.

# Running your own model on MTEB

from mteb import MTEB
from sentence_transformers import SentenceTransformer

model_name = "intfloat/e5-large-v2"
model = SentenceTransformer(model_name)

# Run on retrieval tasks only
evaluation = MTEB(tasks=["NFCorpus", "HotpotQA", "MSMARCO"])
results = evaluation.run(model, output_folder=f"results/{model_name}")

# Results saved as JSON files
# Each result contains nDCG@10, recall@k, MAP, etc.

Reading the MTEB leaderboard

The MTEB leaderboard is at huggingface.co/spaces/mteb/leaderboard. Key columns:

Average (all tasks): Overall score, but this includes many task types that may not be relevant to your use case
Retrieval Average: The most important column for RAG applications
Model Size: Smaller models at competitive performance are often preferable for production

Warning: Large models often overfit to MTEB by being tuned specifically on MTEB training sets. For domain-specific applications, MTEB performance may not predict your application performance. Always evaluate on your own data.

Choosing the Right Model

For production RAG with general English text: Start with intfloat/e5-large-v2 or BAAI/bge-large-en-v1.5. Both are strong open-source options that run on a single GPU and approach API quality.

For multilingual applications: BAAI/bge-m3 handles 100+ languages with a single model. intfloat/multilingual-e5-large is an alternative.

For maximum quality via API: Voyage AI voyage-3 consistently tops MTEB retrieval benchmarks. Start here if you're willing to pay per API call.

For specialized domains: Fine-tune a general model on domain-specific data. General MTEB performance doesn't predict domain-specific performance. Lesson 04 covers fine-tuning in depth.

Common Mistakes

:::danger Choosing a model based on general MTEB score alone MTEB measures general embedding quality across 56 diverse datasets. If your application is specialized (medical Q&A, legal document retrieval, code search), general MTEB scores may not reflect performance on your task. Always evaluate the top 3-5 MTEB candidates on a held-out sample of your actual query-document pairs before committing to a model. :::

:::danger Forgetting prefix requirements for E5 models E5 models require "query: " and "passage: " prefixes. Omitting them degrades performance significantly (estimated 20-30% quality drop on retrieval tasks). Check your chosen model's documentation for required prefixes or instruction formats. BGE models also use instruction prefixes in some configurations. :::

:::warning Using a large model when a small one is sufficient Large embedding models (E5-large, BGE-large) are 3–5× slower than their small counterparts and require significantly more GPU memory. For many applications, E5-small or BGE-small achieves 95% of the large model's quality at 20% of the cost. Benchmark small vs large on your task before defaulting to the large variant. :::

:::tip Always run your own evaluation before choosing a model The fastest path to a bad embedding model choice is trusting the MTEB leaderboard for domain-specific applications. Before any deployment, create a small evaluation set (200-500 query-document pairs with relevance labels) from your domain and evaluate your top candidates on it. This takes 2 hours and saves weeks of debugging poor retrieval quality in production. :::

Interview Q&A

Q1: What is contrastive learning and why is it used for training embedding models?

Contrastive learning is a training approach that shapes the geometry of an embedding space by explicitly specifying which pairs of embeddings should be close and which should be far apart. Positive pairs (semantically similar items) are pushed together; negative pairs (semantically dissimilar items) are pushed apart. The InfoNCE loss is the standard: it maximizes the similarity of positive pairs relative to all negative pairs in the batch.

It's used for embedding models because standard language model pre-training (MLM, next token prediction) doesn't directly optimize the embedding space for retrieval. A model trained with contrastive learning on (query, relevant document) pairs directly learns to place query and document embeddings close together - which is exactly what retrieval requires.

Q2: What is MTEB and how should you interpret its results?

MTEB (Massive Text Embedding Benchmark) evaluates embedding models across 56 datasets and 8 task types including retrieval, clustering, classification, and semantic similarity. It's the standard comparison tool because it covers a broad enough range of tasks to give a reliable picture of general embedding quality.

For production use, interpret MTEB results with three caveats. First, the Retrieval Average column is most relevant for RAG and search applications - overall MTEB average includes many task types that may not reflect your use case. Second, top-ranked models often have been specifically tuned on MTEB datasets; domain-specific performance may differ significantly. Third, model size matters for production: a model at 95% of top MTEB performance but 20% of the size may be preferable.

Q3: What's the difference between symmetric and asymmetric retrieval, and which models support each?

Symmetric retrieval: the query and document are similar in style and length (e.g., finding similar news articles, deduplication). Both query and document embeddings can use the same model with the same input format.

Asymmetric retrieval: the query is short and question-like, the document is long and passage-like (e.g., Q&A, RAG). The query and document have different statistical properties. E5 and BGE handle this with prefixes ("query: " and "passage: ") that signal the model to produce asymmetric embeddings optimized for the retrieval setting. Without these prefixes, retrieval quality on asymmetric tasks drops significantly. For symmetric tasks, the prefixes are either omitted or both use "passage: ".

Q4: When should you use an API embedding service vs a self-hosted model?

Use API when: (1) You're prototyping or early-stage - API removes infrastructure complexity. (2) Your embedding volume is moderate - API is cost-effective up to ~100M tokens/month. (3) You need state-of-the-art quality without fine-tuning infrastructure. (4) Latency is not critical - API round-trip adds 50-200ms.

Use self-hosted when: (1) Your data is sensitive and can't be sent to external APIs. (2) Your volume is large - self-hosted becomes cost-effective above ~100M-1B tokens/month. (3) You need fine-tuning - most API providers don't support custom fine-tuning of embedding models. (4) You need low-latency embedding - local inference can be 5-50× faster than API for batched requests. (5) You need complete control over the model for compliance or security.

Summary

The embedding model landscape has evolved rapidly from Word2Vec (2013) through SBERT (2019) to the current competitive ecosystem:

Training paradigm: All modern embedding models use contrastive learning with in-batch negatives. The specific training data and hard negative strategy differentiate models.

Key models:

SBERT: First practical sentence embedding for retrieval
SimCSE: Showed dropout-based augmentation outperforms complex augmentation
E5: Scaled contrastive pre-training to 1.3B web pairs; requires query/passage prefixes
BGE: Strong open-source model with hard negative mining; top MTEB performer
GTE: Competitive multilingual alternative from Alibaba

Commercial APIs: OpenAI text-embedding-3 (Matryoshka-trained), Voyage AI voyage-3 (often top MTEB retrieval), Cohere Embed v3 (multimodal)

Evaluation: MTEB is the standard benchmark - use Retrieval Average for RAG applications, and always validate on your own domain data before deployment.

:::tip 🎮 Interactive Playground

Visualize this concept: Try the Embedding Space Explorer demo on the EngineersOfAI Playground - no code required.

:::

Picking the Wrong Model for Production​

Why This Exists - The Gap Between Pre-Training and Retrieval​

Historical Context​

Contrastive Learning: The Core Training Paradigm​

The core idea​

In-batch negatives​

SBERT: The Inflection Point​

SimCSE: Simple Contrastive Sentence Embeddings​

Unsupervised SimCSE​

Why dropout augmentation works​

Supervised SimCSE​

E5: Scaling Contrastive Pre-Training​

Data sources​

Asymmetric encoding​

BGE: BAAI General Embedding​

GTE: General Text Embeddings​

Commercial Embedding APIs​

The MTEB Leaderboard​

How MTEB works​

Reading the MTEB leaderboard​

Choosing the Right Model​

Common Mistakes​

Interview Q&A​

Summary​