Gradient Inversion of Federated Diffusion Models
<|reference_start|>Gradient Inversion of Federated Diffusion Models: Diffusion models are becoming defector generative models, which generate exceptionally high-resolution image data. Training effective diffusion models require massive real data, which is privately owned by distributed parties. Each data party can collaboratively train diffusion models in a federated learning manner by sharing gradients instead of the raw data. In this paper, we study the privacy leakage risk of gradient inversion attacks. First, we design a two-phase fusion optimization, GIDM, to leverage the well-trained generative model itself as prior knowledge to constrain the inversion search (latent) space, followed by pixel-wise fine-tuning. GIDM is shown to be able to reconstruct images almost identical to the original ones. Considering a more privacy-preserving training scenario, we then argue that locally initialized private training noise $\epsilon$ and sampling step t may raise additional challenges for the inversion attack. To solve this, we propose a triple-optimization GIDM+ that coordinates the optimization of the unknown data, $\epsilon$ and $t$. Our extensive evaluation results demonstrate the vulnerability of sharing gradient for data protection of diffusion models, even high-resolution images can be reconstructed with high quality.<|reference_end|>
@article{huang2024gradient,
    title={Gradient Inversion of Federated Diffusion Models},
    author={Jiyue Huang, Chi Hong, Lydia Y. Chen, Stefanie Roos},
    journal={arXiv preprint arXiv:2405.20380},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.20380},
    primaryClass={cs.AI cs.CR cs.CV}
}