Differentially private Bayesian learning on distributed data
Many applications of machine learning, for example in health care, would benefit from methods that can guarantee privacy of data subjects. Differential privacy (DP) has become established as a standard for protecting learning results. The standard DP algorithms require a single trusted party to have access to the entire data, which is a clear weakness, or add prohibitive amounts of noise. We consider DP Bayesian learning in a distributed setting, where each party only holds a single sample or a few samples of the data. We propose a learning strategy based on a secure multi-party sum function for aggregating summaries from data holders and the Gaussian mechanism for DP. Our method builds on an asymptotically optimal and practically efficient DP Bayesian inference with rapidly diminishing extra cost.
Title: Differentially private Bayesian learning on distributed data
Comments:
- This paper develops a method for differential privacy (DP) Bayesian learning in a distributed setting, where data is split up over multiple clients. This differs from the traditional DP Bayesian learning setting, in which a single party has access to the full dataset. The main issue here is that performing DP methods separately on each client would yield too much noise; the goal is then to find a way to add an appropriate amount of noise, without compromising privacy, in this setting. To solve this, the authors introduce a method that combines existing DP Bayesian learning methods with a secure multi-party communication method called the DCA algorithm. Theoretically, this paper shows that the method satisfies differential privacy. It also shows specifically how to combine the method with existing DP methods that run on each client machine. Experiments are shown on Bayesian linear regression, on both synthetic and real data.
- Overall, I feel that this method provides a nice contribution to work on Bayesian DP, and extends where one can use these methods to data-distributed settings, which seems like a practical improvement.
- I feel that this paper is written quite nicely. It is therefore easy to understand previous work, the new method, and how the new method fits in context with previous work.
- One small note: I might’ve missed it, but please make sure the acronym TA (for trusted aggregator, I believe) is defined the paper. I believe “TA” might be used in a few places without definition.