DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning
<|reference_start|>DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning: Text-to-image diffusion models have been shown to suffer from sample-level memorization, possibly reproducing near-perfect replica of images that they are trained on, which may be undesirable. To remedy this issue, we develop the first differentially private (DP) retrieval-augmented generation algorithm that is capable of generating high-quality image samples while providing provable privacy guarantees. Specifically, we assume access to a text-to-image diffusion model trained on a small amount of public data, and design a DP retrieval mechanism to augment the text prompt with samples retrieved from a private retrieval dataset. Our \emph{differentially private retrieval-augmented diffusion model} (DP-RDM) requires no fine-tuning on the retrieval dataset to adapt to another domain, and can use state-of-the-art generative models to generate high-quality image samples while satisfying rigorous DP guarantees. For instance, when evaluated on MS-COCO, our DP-RDM can generate samples with a privacy budget of $\epsilon=10$, while providing a $3.5$ point improvement in FID compared to public-only retrieval for up to $10,000$ queries.<|reference_end|>
@article{lebensold2024dp-rdm:,
    title={DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning},
    author={Jonathan Lebensold, Maziar Sanjabi, Pietro Astolfi, Adriana
  Romero-Soriano, Kamalika Chaudhuri, Mike Rabbat, Chuan Guo},
    journal={arXiv preprint arXiv:2403.14421},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.14421},
    primaryClass={cs.LG cs.CR cs.CV}
}