A Dataset is Worth 1 MB
| Authors | Elad Kimchi Shoshani et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2602.23358 |
| Download | |
| Categories | cs.LG, cs.CV |
Abstract
A dataset server must often distribute the same large payload to many clients, incurring massive communication costs. Since clients frequently operate on diverse hardware and software frameworks, transmitting a pre-trained model is often infeasible; instead, agents require raw data to train their own task-specific models locally. While dataset distillation attempts to compress training signals, current methods struggle to scale to high-resolution data and rarely achieve sufficiently small files. In this paper, we propose Pseudo-Labels as Data (PLADA), a method that completely eliminates pixel transmission. We assume agents are preloaded with a large, generic, unlabeled reference dataset (e.g., ImageNet-1K, ImageNet-21K) and communicate a new task by transmitting only the class labels for specific images. To address the distribution mismatch between the reference and target datasets, we introduce a pruning mechanism that filters the reference dataset to retain only the labels of the most semantically relevant images for the target task. This selection process simultaneously maximizes training efficiency and minimizes transmission payload. Experiments on 10 diverse datasets demonstrate that our approach can transfer task knowledge with a payload of less than 1 MB while retaining high classification accuracy, offering a promising solution for efficient dataset serving.
Engineering Breakdown
Plain English
This paper addresses the problem of efficiently distributing training data to multiple clients with heterogeneous hardware and software setups, where transmitting raw datasets or pre-trained models is prohibitively expensive. The authors propose PLADA (Pseudo-Labels as Data), a method that compresses dataset distribution to approximately 1 MB by eliminating pixel transmission entirely. Instead of sending images, the system assumes clients have access to a large unlabeled reference dataset (like ImageNet-1K or ImageNet-21K) and transmits only the class labels for a subset of reference examples, allowing clients to train their own task-specific models locally. This approach achieves significant communication cost reduction while maintaining the ability for clients to perform local model training without downloading the full original dataset.
Core Technical Contribution
The core innovation is the insight that instead of transmitting raw data or distilled representations, you can leverage a shared, unlabeled reference dataset that clients already possess to communicate new tasks through pseudo-labeling. PLADA identifies which reference examples are most informative for the target task and transmits only their labels, effectively using the reference dataset as a compression basis. This is fundamentally different from dataset distillation methods that attempt to synthesize or compress pixels directly; instead, it exploits the statistical structure already present in commonly available large datasets. The method effectively converts a data distribution problem into a label assignment problem, reducing communication from gigabytes (raw data) or megabytes (distilled data) down to kilobytes or low megabytes (label indices).
How It Works
The system operates under the assumption that all clients have pre-loaded access to a large, generic, unlabeled reference dataset (e.g., ImageNet-1K with 1.2M images). To communicate a new task, the server: (1) identifies a subset of reference images that best represent the target task distribution, (2) generates pseudo-labels for these reference examples using the target task's label space, (3) transmits only the indices and labels of these selected reference images. Clients receive this compact label set, use their local copy of the reference dataset to retrieve the actual images, and train their own task-specific models on the pseudo-labeled reference subset. The server-side identification of informative reference samples likely uses some form of task-aware selection (possibly based on gradient matching, information density, or semantic similarity to the target task), and the pseudo-label generation bridges the semantic gap between the reference dataset's unlabeled examples and the target task's label space.
Production Impact
undefined
Limitations and When Not to Use This
The approach fundamentally depends on having a suitable pre-loaded reference dataset available at every client, which limits applicability to niche domains where such datasets don't exist or are prohibitively large. The quality of pseudo-labels is bounded by the reference dataset's semantic overlap with the target task—if the target task is out-of-distribution from ImageNet, the pseudo-labeling accuracy will degrade, directly hurting downstream model quality. The paper doesn't clearly address how to handle tasks with novel label spaces or fine-grained categories that may be underrepresented in generic reference datasets. Additionally, the method assumes clients have sufficient local compute to train models; it doesn't reduce compute cost, only communication cost, so for compute-constrained edge devices this may be insufficient. There's also a potential privacy trade-off not discussed: knowing which reference examples are pseudo-labeled for a task leaks information about the task structure itself.
Research Context
This work directly builds on and improves upon dataset distillation literature (which seeks to compress datasets into smaller synthetic versions) by taking an orthogonal approach that leverages existing unlabeled reference datasets rather than synthesizing new data. It addresses a gap in federated learning and edge ML where communication is the primary bottleneck—prior work focused on model compression (quantization, pruning, LoRA) but took raw data transmission as given. The paper opens a new research direction around reference-dataset-based communication and implicit data representation, with potential connections to few-shot learning and transfer learning paradigms. The 1 MB target is a provocative benchmark that reframes dataset distribution as a discrete optimization problem (selecting informative reference examples) rather than a continuous compression problem (pixel space distillation).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::