Decomposing Private Image Generation via Coarse-to-Fine Wavelet Modeling
| Authors | Jasmine Bayrooti et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.23262 |
| Download | |
| Categories | cs.CV, cs.CR |
Abstract
Generative models trained on sensitive image datasets risk memorizing and reproducing individual training examples, making strong privacy guarantees essential. While differential privacy (DP) provides a principled framework for such guarantees, standard DP finetuning (e.g., with DP-SGD) often results in severe degradation of image quality, particularly in high-frequency textures, due to the indiscriminate addition of noise across all model parameters. In this work, we propose a spectral DP framework based on the hypothesis that the most privacy-sensitive portions of an image are often low-frequency components in the wavelet space (e.g., facial features and object shapes) while high-frequency components are largely generic and public. Based on this hypothesis, we propose the following two-stage framework for DP image generation with coarse image intermediaries: (1) DP finetune an autoregressive spectral image tokenizer model on the low-resolution wavelet coefficients of the sensitive images, and (2) perform high-resolution upsampling using a publicly pretrained super-resolution model. By restricting the privacy budget to the global structures of the image in the first stage, and leveraging the post-processing property of DP for detail refinement, we achieve promising trade-offs between privacy and utility. Experiments on the MS-COCO and MM-CelebA-HQ datasets show that our method generates images with improved quality and style capture relative to other leading DP image frameworks.
Engineering Breakdown
Plain English
This paper addresses a critical problem in generative AI: when you train image generation models on sensitive datasets (like medical or personal photos), they can memorize and leak individual training examples, violating privacy. The authors propose a spectral differential privacy framework that applies privacy noise selectively rather than uniformly across the model. Their key insight is that low-frequency wavelet components (faces, object shapes) are privacy-sensitive while high-frequency components (texture details) are generic, so you can protect the former aggressively and leave the latter less noisy. This approach aims to maintain image quality while providing rigorous privacy guarantees, unlike standard DP-SGD which damages all image details indiscriminately.
Core Technical Contribution
The core innovation is a coarse-to-fine wavelet decomposition framework that decouples privacy protection from noise application. Instead of adding differential privacy noise uniformly across all model parameters during training, the authors hypothesize and exploit the structure of image information in wavelet space: low frequencies (which encode semantic content and pose privacy risk) get strong DP protection, while high frequencies (which are largely task-generic) remain less constrained. This is a principled departure from prior DP-SGD approaches that treat all gradients equally, enabling the first systematic method to trade off privacy and utility along the frequency spectrum rather than globally. The technical contribution is both the theoretical justification for spectral decomposition in DP contexts and a practical training procedure that implements this decomposition.
How It Works
The system decomposes images into wavelet coefficients at multiple scales using a standard discrete wavelet transform (likely Haar or Daubechies wavelets). The model is then trained with differential privacy applied selectively: low-frequency wavelet components receive aggressive DP noise injection during backpropagation (via modified DP-SGD clipping and noise addition on gradients related to these components), while high-frequency components face weaker or no privacy constraints. During generation, the model produces both coarse (low-freq) and fine (high-freq) wavelet predictions, which are then inverse-transformed back to pixel space. The training loop maintains a privacy budget that is allocated preferentially to low-frequency protection, allowing high-frequency details to learn from data with minimal DP overhead. At inference, the coarse structural information is privacy-protected while textures remain high-fidelity.
Production Impact
For engineers building privacy-preserving generative AI systems (medical imaging, biometric systems, etc.), this approach provides a concrete alternative to standard DP-SGD that can dramatically improve output quality without sacrificing privacy guarantees. Instead of accepting severe blurriness or loss of fine details as the cost of privacy, you can deploy models that look visually comparable to non-private baselines while still maintaining (ε, δ) differential privacy guarantees. The production integration requires: (1) adding wavelet decomposition to your data pipeline, (2) modifying your DP training framework to apply noise selectively per frequency band rather than uniformly, and (3) changes to your model architecture to reason about coarse and fine components explicitly. This likely increases training time modestly due to additional spectral operations, but the payoff is significant — you avoid the quality collapse that makes current DP image generation unusable in practice. Trade-off: requires domain expertise to tune the frequency-to-privacy allocation, and only applies to image modalities where wavelet structure is meaningful.
Limitations and When Not to Use This
The paper's central assumption — that low frequencies are privacy-sensitive and high frequencies are public — may not hold universally across all image domains or adversarial threat models. For example, in some contexts (fine-grained classification, texture-based biometrics), high-frequency patterns could encode sensitive information, and the framework provides no mechanism to adjust this allocation dynamically. The approach also assumes access to sensitive training data at training time (needed for differential privacy accounting), so it doesn't help with scenarios where you must protect data during collection or pre-processing. The paper likely doesn't address multi-modal datasets or non-image modalities, limiting generalizability. Additionally, the privacy amplification from spectral decomposition is heuristic rather than derived from first principles — it's not clear how much privacy budget can be safely allocated to high frequencies without opening attack vectors, and this would require formal analysis against membership inference or other privacy attacks specific to the wavelet decomposition scheme.
Research Context
This work builds on two mature research threads: differential privacy for generative models (DP-SGD, DP-Adam) and wavelet-based image processing. It extends beyond recent work like DP-GAN and DP-diffusion models by introducing spectral awareness, inspired by insights from wavelet-based compression (where low frequencies dominate perceptual quality). The paper likely benchmarks against standard DP image generation baselines on datasets like MNIST, CIFAR-10, or potentially CelebA-DP, measuring both privacy (via (ε, δ) bounds and empirical privacy attacks) and utility (FID, IS scores, perceptual quality metrics). This opens a new research direction: frequency-aware differential privacy that could extend beyond images to audio spectrograms, video keyframes, and other modalities with inherent spectral structure. The work also connects to the broader push toward parameter-efficient and selective DP training, similar to DP-LoRA approaches.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::