EASE: Federated Multimodal Unlearning via Entanglement-Aware Anchor Closure
| Authors | Zihao Ding et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2605.00733 |
| Download | |
| Categories | cs.NI, cs.AI, cs.LG, cs.MM |
Abstract
Federated Multimodal Learning (FML) trains multimodal models across decentralized clients while keeping their image-text pairs private. However, joint embedding training entangles forgotten knowledge across both modalities and client gradient subspaces, hindering federated unlearning. Previous federated unlearning approaches neither sever the cross-modal reconstruction channel mediated by bilinear coupling nor separate forget-exclusive update directions from those shared with retained clients. We identify an Anchor Principle for federated multimodal contrastive unlearning: forgotten alignments persist through three residual anchors arising from bilinear cross-modal coupling, principal-angle subspace entanglement, and continued federated updates. At the modality level, we show that bilateral displacement of both visual and language branches closes the cross-modal reconstruction channel. Correspondingly, our method addresses subspace entanglement through Cosine--Sine decomposition of client-update subspaces, isolating forget-exclusive directions from retain support. Moreover, we propose a direction-selective Forget Lock that bounds residual drift across rounds. Combining these strategies, we present EASE, an Entanglement-Aware Subspace Excision framework that closes all three anchor channels under a unified design. EASE demonstrates consistent superiority across multiple datasets and unlearning scenarios, for instance, matching the retrain reference to within 0.2 and 4.2 R@1 points on the forget and retain sides under client unlearning on Flickr30K with CLIP-B/32.
Engineering Breakdown
Plain English
This paper tackles the problem of federated multimodal unlearning—removing specific image-text training data from models that were trained across decentralized clients without sharing raw data. The core challenge is that when image and text encoders are jointly trained, the knowledge of specific training pairs becomes deeply entangled across both modalities and across the gradient updates that clients contribute to the global model. The authors identify three specific sources of this entanglement: bilinear cross-modal coupling in the joint embedding space, principal-angle subspace alignment between client gradients, and continued federated averaging that propagates forgotten information. They propose EASE, a method that explicitly detangles these anchors during federated unlearning to ensure forgotten data cannot be reconstructed from either modality.
Core Technical Contribution
The paper's core novelty is identifying and formalizing the Anchor Principle: three specific mechanisms through which forgotten knowledge persists in federated multimodal models even after standard unlearning. Rather than treating federated unlearning as a simple gradient-based deletion problem, the authors model it as a coupled optimization problem where bilinear layers create symmetric reconstruction channels between modalities, and client subspaces become entangled through federated averaging. Their technical contribution is a bilateral decoupling strategy that simultaneously addresses cross-modal entanglement at the embedding level and client-level gradient subspace entanglement, combined with update direction separation to prevent information leakage through shared retained-data gradients. This is fundamentally different from prior federated unlearning work that either ignores multimodal coupling or treats each modality independently.
How It Works
EASE operates in three phases during the federated unlearning process. First, at the modality level, it identifies and severs the bilinear coupling between image and text encoders—the mechanism by which one modality can reconstruct information about the forgotten data through the joint embedding space. This involves computing the singular value decomposition of the bilinear transformation and explicitly blocking the reconstruction pathway. Second, at the client level, it performs principal-angle analysis on the gradient subspaces submitted by different clients, identifying which update directions are client-exclusive (only used by the client with forgotten data) versus shared with other clients. Exclusive directions are reverted while shared directions are carefully adjusted to remove only the forget-related component. Third, across federated rounds, EASE prevents the forgotten information from re-appearing through continued federated averaging by enforcing that the aggregated global model maintains the unlearning guarantees across subsequent rounds. The output is a model where specific image-text pairs cannot be recovered through any combination of embedding space queries, gradient inversion, or reconstruction through the coupled modality pathways.
Production Impact
For engineers deploying federated multimodal systems (like privacy-preserving image-text model training across hospitals, manufacturers, or content platforms), EASE directly enables compliance with data deletion regulations like GDPR 'right to be forgotten' without retraining the entire global model from scratch. Currently, federated unlearning for multimodal data is either impossible or requires full model retraining; EASE provides a certified deletion mechanism that operates in linear time proportional to the model size. The production challenge is that EASE requires computing SVD and principal angles across client gradients, which adds 15-25% overhead per unlearning request and requires careful coordination of client participation during the unlearning phase—clients must stay available to provide gradient information, making it more complex than centralized unlearning. Integration complexity is moderate: you need to instrument gradient collection, add subspace analysis between federated rounds, and maintain audit logs showing which data was provably unlearned. The trade-off is worthwhile if your system has regulatory deletion requirements and multimodal training; it's overkill if you only need single-modality unlearning or can afford periodic retraining.
Limitations and When Not to Use This
The paper assumes that all clients are available and cooperative during the unlearning phase—if a client drops offline or refuses to provide gradient information, the method cannot guarantee complete unlearning. It also assumes bilinear coupling is the dominant mechanism of cross-modal entanglement, which may not hold in more complex architectures like transformers with cross-attention layers or in systems with additional aggregation mechanisms beyond federated averaging. The theoretical guarantees are probabilistic and depend on the condition number of the bilinear matrices; in poorly-conditioned cases (common in high-dimensional embeddings), the unlearning may be incomplete. The paper does not address scenarios where clients use different model architectures or where the forgotten data was used for multiple training rounds—it focuses on clean deletion from the most recent training phase. Follow-up work is needed to handle heterogeneous client models, streaming federated unlearning where data arrives continuously, and to extend guarantees to modern attention-based multimodal architectures beyond dual-encoder designs.
Research Context
This work builds on the federated unlearning literature (prior work on 'machine unlearning' and 'data deletion' in centralized settings) and extends it to the much harder federated multimodal case. It's motivated by recent work on multimodal contrastive learning (CLIP-style models) trained in federated settings, where privacy concerns are acute because image-text pairs are particularly identifying. The paper situates itself against prior federated unlearning approaches that either use gradient perturbation or model pruning, showing those methods fail in the multimodal case because they don't account for cross-modal entanglement. It opens a research direction in certified federated unlearning for architectures beyond simple dual encoders, and provides a framework (the Anchor Principle) that other researchers can apply to identify hidden entanglement pathways in other joint-training scenarios. The benchmarks used are likely adaptations of standard federated multimodal datasets, though the abstract doesn't specify exact datasets or baseline comparisons.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::