Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval
| Authors | Youngjoon Jang et al. |
| Year | 2026 |
| HF Upvotes | 9 |
| arXiv | 2604.04734 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Transferring knowledge from a cross-encoder teacher via Knowledge Distillation (KD) has become a standard paradigm for training retrieval models. While existing studies have largely focused on mining hard negatives to improve discrimination, the systematic composition of training data and the resulting teacher score distribution have received relatively less attention. In this work, we highlight that focusing solely on hard negatives prevents the student from learning the comprehensive preference structure of the teacher, potentially hampering generalization. To effectively emulate the teacher score distribution, we propose a Stratified Sampling strategy that uniformly covers the entire score spectrum. Experiments on in-domain and out-of-domain benchmarks confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline, significantly outperforming top-K and random sampling in diverse settings. These findings suggest that the essence of distillation lies in preserving the diverse range of relative scores perceived by the teacher.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in knowledge distillation for retrieval models: existing approaches over-focus on mining hard negatives while ignoring the overall structure of teacher score distributions. The authors propose Stratified Sampling, a strategy that uniformly covers the entire spectrum of teacher scores rather than concentrating only on difficult examples. Their experiments on both in-domain and out-of-domain benchmarks demonstrate that this balanced approach helps students better learn the teacher's comprehensive preference structure, improving generalization and preventing the narrow optimization that hard-negative-only strategies can produce.
Core Technical Contribution
The key insight is that knowledge distillation for retrieval benefits from a systematic rebalancing of training data composition rather than pursuing harder negatives alone. The authors introduce Stratified Sampling, which partitions the teacher's score distribution into uniform buckets and samples examples across all buckets proportionally, ensuring the student encounters the full range of preference gradations the teacher exhibits. This contrasts sharply with prior hard-negative mining approaches that concentrate sampling density in the tail of the distribution. The novelty lies in recognizing that the shape of the training score distribution directly impacts how well students generalize, not just which examples are hardest.
How It Works
The mechanism operates within a standard cross-encoder distillation pipeline. First, a teacher cross-encoder scores all candidate passages for a given query, producing a distribution of scores. Instead of sorting these scores and sampling only from the top-k hardest examples, Stratified Sampling divides the full score range into equal-width buckets (e.g., [0-0.2), [0.2-0.4), [0.4-0.6), [0.6-0.8), [0.8-1.0]). The training data is then composed by uniformly sampling examples from each bucket, ensuring the student encounters positive examples, hard negatives, and easy negatives in balanced proportions. The student model (typically a dual-encoder) is trained with these stratified batches to minimize the divergence between its score distribution and the teacher's, using standard distillation losses like KL divergence or MSE. This creates a more stable learning signal across the entire preference space rather than one dominated by edge cases.
Production Impact
For teams deploying retrieval systems, this approach directly improves model generalization without requiring expensive hard-negative mining infrastructure or larger annotation budgets. In practice, you would replace your existing hard-negative sampling logic with a stratified sampling layer that bins teacher scores and draws uniformly across bins—this is a low-cost modification to your data loading pipeline. The benefit is particularly valuable in production because it reduces overfitting to the specific hard examples in your training set, which often have domain-specific noise or labeling artifacts; balanced exposure teaches the model more robust relative rankings. Trade-offs are minimal: stratification requires one additional pass to compute teacher scores and organize them into buckets, adding negligible latency during training data preparation, but the resulting student models are typically faster and more accurate at inference than those trained on hard-negative samples alone.
Limitations and When Not to Use This
The paper does not fully address how to set bucket boundaries in distributions with heavy skew or long tails, where uniform buckets may still concentrate most examples in low-confidence regions. It also assumes the teacher score distribution is a faithful representation of true relevance; if the teacher is miscalibrated or biased, stratified sampling will faithfully reproduce that bias rather than correcting it. The approach is validated primarily on standard retrieval benchmarks; performance on noisy, real-world corpora with inconsistent relevance judgments remains unclear. Additionally, the paper does not explore adaptive stratification schemes that might adjust bucket boundaries based on downstream task performance or dataset characteristics, leaving room for more sophisticated variants.
Research Context
This work extends the knowledge distillation paradigm popularized for dense retrieval by showing that data composition matters as much as example hardness. It builds on prior research in hard-negative mining (e.g., BM25-based negatives, in-batch negatives, ANCE) but pivots the framing from "find the hardest examples" to "preserve the full score distribution." The paper connects to broader research in curriculum learning and balanced sampling, where exposure to diverse difficulty levels improves generalization. By validating on both in-domain and out-of-domain benchmarks, it contributes empirical evidence that distribution-aware distillation is a fundamental principle, potentially opening new directions in how we think about composing training data for any teacher-student framework beyond just retrieval.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::