Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation
| Authors | Muquan Li et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.24144 |
| Download | |
| Categories | cs.CV |
Abstract
Decoupled dataset distillation (DD) compresses large corpora into a few synthetic images by matching a frozen teacher's statistics. However, current residual-matching pipelines rely on static real patches, creating a fit-complexity gap and a pull-to-anchor effect that reduce intra-class diversity and hurt generalization. To address these issues, we introduce RETA -- a Retrieval and Topology Alignment framework for decoupled DD. First, Dynamic Retrieval Connection (DRC) selects a real patch from a prebuilt pool by minimizing a fit-complexity score in teacher feature space; the chosen patch is injected via a residual connection to tighten feature fit while controlling injected complexity. Second, Persistent Topology Alignment (PTA) regularizes synthesis with persistent homology: we build a mutual k-NN feature graph, compute persistence images of components and loops, and penalize topology discrepancies between real and synthetic sets, mitigating pull-to-anchor effect. Across CIFAR-100, Tiny-ImageNet, ImageNet-1K, and multiple ImageNet subsets, RETA consistently outperforms various baselines under comparable time and memory, especially reaching 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at 50 images per class, +3.1% over the best prior.
Engineering Breakdown
Plain English
This paper tackles dataset distillation—the problem of compressing large image datasets into a handful of synthetic images that preserve the knowledge needed to train neural networks. Current methods use static "anchor" patches from real images and try to match frozen teacher network statistics, but this creates two problems: the synthetic images become less diverse within classes, and there's a gap between what can be fit and actual model generalization. The authors introduce RETA, a framework combining Dynamic Retrieval Connection (which picks the best real patch from a pool for each synthetic image based on a fit-complexity trade-off) and Persistent Topology Alignment (which uses topological structure from persistent homology to regularize synthetic image generation). The key insight is that fixed anchors constrain the solution space too much—dynamically selecting patches and preserving topological properties leads to better intra-class diversity and improved generalization on downstream tasks.
Core Technical Contribution
The core novelty is the combination of two technical components that address fundamental limitations in decoupled dataset distillation. Dynamic Retrieval Connection (DRC) replaces static patch anchoring with a learned selection mechanism that picks real patches from a prebuilt pool by minimizing a fit-complexity score in teacher feature space—this allows the optimization to find better residual connections without getting locked into suboptimal anchors. Persistent Topology Alignment (PTA) introduces topological regularization via persistent homology, building a multi-scale topological descriptor of the target class and regularizing synthetic image generation to preserve this structure, which directly combats the mode collapse and loss of intra-class diversity. Together, these innovations move beyond the assumption that frozen, static anchors are sufficient, and demonstrate that dynamic selection plus topological constraints yield better synthetic datasets that generalize further.
How It Works
The system operates in two main phases. First, a real image patch pool is prebuilt from the training dataset and encoded using the frozen teacher network. For each synthetic image being optimized, DRC scores all available patches using a fit-complexity objective—this balances how well a patch's features align with the target synthetic image (fit term) against the complexity of the injected residual (to avoid overfitting). The chosen patch is then injected via a residual connection into the synthetic image, allowing fine-grained feature matching without anchor lock-in. In parallel, PTA computes persistent homology on the target class's real images, creating a multi-scale topological signature that captures structural relationships between instances. During synthesis optimization, this topological descriptor is converted into a regularization loss that penalizes synthetic images deviating from the expected topological structure, encouraging the optimizer to maintain intra-class diversity. The two components work synergistically: DRC handles local feature fidelity through dynamic patch selection, while PTA ensures global class structure is preserved across the synthetic set.
Production Impact
For teams deploying dataset distillation in production (e.g., federated learning, on-device training, or data-efficient fine-tuning), this approach significantly improves the quality of synthetic datasets with measurable downstream accuracy gains. Instead of handcrafting or freezing anchor patches, engineers get a principled method to dynamically select the most informative real patches for each synthetic image, reducing manual tuning and improving robustness across different class distributions. The topological regularization component provides a new lever for controlling synthetic dataset quality without requiring additional labeled data or human annotation—this is particularly valuable when distilling from large, imbalanced, or multi-domain datasets. The main trade-off is computational cost during the distillation phase: DRC requires scoring against a patch pool (potentially thousands of candidates) and PTA requires computing persistent homology descriptors, so the synthesis process will be slower than baseline methods, though the resulting datasets are smaller and faster to use downstream. Integration complexity is moderate—this requires implementing persistent homology computation (available in libraries like Ripser or Giotto-tda) and modifying the residual connection injection logic in existing distillation codebases.
Limitations and When Not to Use This
The approach assumes a prebuilt patch pool is available and that patches from the training set are representative—in domain shift scenarios or with severely imbalanced classes, patch selection may become ineffective. Persistent homology computation scales poorly to very high-dimensional feature spaces and large numbers of instances, so it may struggle with massive datasets or models with thousand-dimensional embeddings; the paper does not discuss scalability empirically beyond typical image classification. The method is framed for decoupled distillation (where teacher and student are frozen) and may not directly extend to coupled/end-to-end distillation pipelines where both networks are optimized jointly. Additionally, the paper's abstract is incomplete (cuts off mid-sentence at PTA), so critical details about hyperparameter sensitivity, computational costs, and ablation studies are unknown from the provided text; the full generalization performance gains and failure modes cannot be fully assessed. Finally, topological alignment assumes that class structure captured by persistent homology in real images should be preserved in synthetic images—this assumption may not hold when target downstream models have very different inductive biases than the teacher used for distillation.
Research Context
This work builds on recent advances in decoupled dataset distillation, which separates the teacher (whose knowledge is being extracted) from the student (which learns from synthetic data), enabling knowledge transfer without co-optimizing both networks. It improves upon static anchor-based methods by incorporating ideas from topological data analysis—specifically persistent homology—which is increasingly used in ML to capture multi-scale structure and has shown promise in improving robustness. The paper positions itself as addressing a key limitation of residual-matching pipelines: the 'pull-to-anchor' effect that causes mode collapse and reduced diversity, a problem recognized in recent work on synthetic data quality. By introducing dynamic retrieval alongside topological regularization, the work opens a new direction for dataset distillation that combines algorithmic selection (which patch to use) with geometric regularization (what structure to preserve), potentially inspiring similar hybrid approaches in other data compression and synthesis tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::