Scaling up the Banded Matrix Factorization Mechanism for Differentially Private ML
<|reference_start|>Scaling up the Banded Matrix Factorization Mechanism for Differentially Private ML: Correlated noise mechanisms such as DP Matrix Factorization (DP-MF) have proven to be effective alternatives to DP-SGD in large-epsilon few-epoch training regimes. Significant work has been done to find the best correlated noise strategies, and the current state-of-the-art approach is DP-BandMF, which optimally balances the benefits of privacy amplification and noise correlation. Despite it's utility advantages, severe scalability limitations prevent this mechanism from handling large-scale training scenarios where the number of training iterations may exceed $10^4$ and the number of model parameters may exceed $10^7$. In this work, we present techniques to scale up DP-BandMF along these two dimensions, significantly extending it's reach and enabling it to effectively handle settings with over $10^6$ training iterations and $10^9$ model parameters, with negligible utility degradation.<|reference_end|>
@article{mckenna2024scaling,
    title={Scaling up the Banded Matrix Factorization Mechanism for Differentially
  Private ML},
    author={Ryan McKenna},
    journal={arXiv preprint arXiv:2405.15913},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.15913},
    primaryClass={cs.LG cs.CR cs.DS}
}