Differentially Private Block-wise Gradient Shuffle for Deep Learning
<|reference_start|>Differentially Private Block-wise Gradient Shuffle for Deep Learning: Traditional Differentially Private Stochastic Gradient Descent (DP-SGD) introduces statistical noise on top of gradients drawn from a Gaussian distribution to ensure privacy. This paper introduces the novel Differentially Private Block-wise Gradient Shuffle (DP-BloGS) algorithm for deep learning. BloGS builds off of existing private deep learning literature, but makes a definitive shift by taking a probabilistic approach to gradient noise introduction through shuffling modeled after information theoretic privacy analyses. The theoretical results presented in this paper show that the combination of shuffling, parameter-specific block size selection, batch layer clipping, and gradient accumulation allows DP-BloGS to achieve training times close to that of non-private training while maintaining similar privacy and utility guarantees to DP-SGD. DP-BloGS is found to be significantly more resistant to data extraction attempts than DP-SGD. The theoretical results are validated by the experimental findings.<|reference_end|>
@article{zagardo2024differentially,
    title={Differentially Private Block-wise Gradient Shuffle for Deep Learning},
    author={David Zagardo},
    journal={arXiv preprint arXiv:2407.21347},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.21347},
    primaryClass={cs.LG cs.AI cs.CR}
}