Efficient Differentially Private Fine-Tuning of Diffusion Models
<|reference_start|>Efficient Differentially Private Fine-Tuning of Diffusion Models: The recent developments of Diffusion Models (DMs) enable generation of astonishingly high-quality synthetic samples. Recent work showed that the synthetic samples generated by the diffusion model, which is pre-trained on public data and fully fine-tuned with differential privacy on private data, can train a downstream classifier, while achieving a good privacy-utility tradeoff. However, fully fine-tuning such large diffusion models with DP-SGD can be very resource-demanding in terms of memory usage and computation. In this work, we investigate Parameter-Efficient Fine-Tuning (PEFT) of diffusion models using Low-Dimensional Adaptation (LoDA) with Differential Privacy. We evaluate the proposed method with the MNIST and CIFAR-10 datasets and demonstrate that such efficient fine-tuning can also generate useful synthetic samples for training downstream classifiers, with guaranteed privacy protection of fine-tuning data. Our source code will be made available on GitHub.<|reference_end|>
@article{liu2024efficient,
    title={Efficient Differentially Private Fine-Tuning of Diffusion Models},
    author={Jing Liu, Andrew Lowy, Toshiaki Koike-Akino, Kieran Parsons, Ye Wang},
    journal={arXiv preprint arXiv:2406.05257},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.05257},
    primaryClass={cs.LG cs.CR}
}