Differentially Private Bayesian Inference for Exponential Families
The study of private inference has been sparked by growing concern regarding the analysis of data when it stems from sensitive sources. We present the first method for private Bayesian inference in exponential families that properly accounts for noise introduced by the privacy mechanism. It is efficient because it works only with sufficient statistics and not individual data. Unlike other methods, it gives properly calibrated posterior beliefs in the non-asymptotic data regime.
After considering author responses & other reviews, I keep my score unchanged. Some additional comments to the author responses  On truncation bounds (rows 23-26): In my opinion, this definitely deserves to be mentioned in the paper. I agree that there are plenty of problems where a reasonable, say, 95%-bound is known accurately enough a priori. However, this is still an exception and  more generally, especially in the strict privacy regime you are currently using, I would guess the bounds estimated under DP could actually be quite a bit off from the true values. Given that the truncation bound is a key hyperparamater that could completely break the method, I encourage you to add some (even simple) testing on this to the paper.  Ops vs Private MCMC (44-47): I don't doubt that the calibration would be more or less off, the question in my mind is how much. But granted, I agree it's not generally feasible, or necessary in this case, to start comparing against (eps,delta)-methods as well.  Other comments (14-17,41-43): Thank you, I'm happy with these.   Differentially Private Bayesian Inference for Exponential Families  The paper considers Bayesian posterior estimation for exponential family models under differential privacy (DP). More concretely, the main problem is how to infer a well-calibrated posterior distribution from noisy sufficient statistics, where the noise guarantees privacy by standard arguments. As a solution, the authors propose 2 versions of a Gibbs sampler: one for sampling from a general exponential family model with bounded sufficient statistics, and a variant that works with unbounded sufficient statistics but assumes the data is very low-dimensional.  The main novelty in the approach is to include the DP noise properly into the modelling instead of simply "forgetting" it and treating the actually noisy posterior as a noiseless one, as is often done in existing literature. Intuitively, this should lead to better calibrated posteriors, which also is the case in the empirical tests shown in the paper. The idea is not totally unheard of (see e.g.  references in the paper), but the formulation is nice, and the solution is novel. The problem as such is certainly an important one that needs to be solved properly, and the paper is a good step in the right direction.  The paper is clearly written and the main ideas are easy to understand. The mathematical proofs provided in the paper are mostly already known or small modifications to existing proofs. There doesn't seem to be any major problems with the theory, and it seems well-suited to the problem.  Pros:     * Including the DP noise into the modelling is a very relevant idea, and there does not seem to be      much papers on the topic.     * The proposed method is novel, and seems to give well-calibrated posteriors with limited sample sizes     under strict privacy.  Cons:     * The proposed solution has an exponential dependency on the dimensionality of the (unbounded)      sufficient statistics (as noted by the authors).     * No theory or testing on how the assumed bounds [a,b] for s_c affect the results.      * Fairly simplistic empirical tests.  Specific questions/requests for the authors:  1) Should the bounds [a,b] for the central interval s_c be given as (fixed) input to Algorithm 2? It seems like they are fixed in the algorithm, but are not provided as input or elsewhere.  2) How much the assumed bounds for s_c affect the results? Since the bounds affect the sensitivity, and thus the DP noise variance in the Laplace mechanism, the results can be made arbitrarily bad by choosing very poor bounds. Even if the bounds can be estimated under privacy (as suggested) and hence are unlikely to be totally off, the estimate is going to be noisy. It seems important to know how much this affects the reported results. Currently, I can't seem to find the bounds used in the actual tests reported in the paper anywhere, nor can I find any hint of how they were chosen. Additionally, it would be nice to have at least some test which shows the performance degradation with sub-optimal bounds.  3) On lines 247-48 you say "OPS suffices as a simpler straw man comparison". Do you think Private MCMC sampling would do better than OPS, or why is OPS a straw man? And if so, do you think you should then also compare to the private MCMC in addition to OPS?  Some minor comments/questions:  i) On line 36 you define calibration as being able to efficiently estimate p(theta|y). I would think calibration refers to something closer to the idea presented in the evaluation section (starting from line 252). This could be clarified.  ii) On line 61 you say your "contribution is a simple release mechanism A" etc. Since the release mechanism is simply adding Laplace noise to the sufficient statistics, i.e., doing the standard stuff (see lines 111-12), I don't think this should count as a contribution, or did I misinterpret something?  iii) On line 226 you mention that s_c needs to be drawn afresh for updating sigma^2. What happens if you use the previously drawn value used in s=s_l+s_c+s_u?  iv) I hope you release the code for the paper (at least after the paper is published).