Differentially Private Learning of Structured Discrete Distributions
We investigate the problem of learning an unknown probability distribution over a discrete population from random samples. Our goal is to design efficient algorithms that simultaneously achieve low error in total variation norm while guaranteeing Differential Privacy to the individuals of the population. We describe a general approach that yields near sample-optimal and computationally efficient differentially private estimators for a wide range of well-studied and natural distribution families. Our theoretical results show that for a wide variety of structured distributions there exist private estimation algorithms that are nearly as efficient-both in terms of sample size and running time-as their non-private counterparts. We complement our theoretical guarantees with an experimental evaluation. Our experiments illustrate the speed and accuracy of our private estimators on both synthetic mixture models and a large public data set.
This paper studies the problem of learning discrete probability distributions over large finite sets under privacy constraints. The contributions are split into three parts: a theoretical study with formal accuracy, privacy, and computational guarantees; a fast empirical algorithm with privacy guarantees; and, a set of experiments on synthetic data and a real dataset.Broadly speaking, the main idea of the paper is to leverage previous work on algorithms for learning adaptive one-dimensional histograms over large sets and combine them with noise-generating procedures to obtain private algorithms. This is an important theoretical problem, and the authors manage to show in Section 3 that privacy can be achieved at a very reasonable price in terms of statistical accuracy. The techniques involved in Theorems 1 and 2 are not especially novel: they rely on the Laplace mechanism and a simple reduction. Theorem 3 is more interesting, but two important details were missing from the original submission. These were provided in the rebuttal and, though they are relatively simple, my suggestion is the authors incorporate them in the supplementary material in further versions of the paper.The practical impact of the contributions presented in the paper might be limited. The authors mention several families of univariate discrete distributions which can be agnostically learned using their techniques. These are quite broad, and seem obviously related to the cases where a data analyst would benefit from a histogram representation of the target distribution. However, the one-dimensional assumptions seems a rather strong assumption for many practical applications. If one had access to samples from a high-dimensional discrete distribution, it does not seem possible that the algorithms presented in the paper can be useful in extracting relevant information about the geometry of such distribution, since different projections to a one-dimensional setting would probably yield quite different histograms. It would be nice if the authors commented on these limitations and whether there are obvious extensions of their techniques for such cases, or on the other hand one would need completely different methods.*** Detalied Comments and Typos ***- In general, I think the material in Section 3 would benefit from some restructuring where algorithms and analyses are decoupled; after all, statements of the form "there exists an algorithm for doing that and this" are less satisfying that statements about a clearly specified algorithm.- According to Definition 1, an (alpha,beta)-agnostic learning algorithm produces a "hypothesis distribution h". However, the output of the algorithm analyzed in Theorem 1 might not be a distribution due to the random perturbation. Therefore, this algorithm is not strictly an (alpha,beta)-agnostic learning algorithm in the sense of Definition 1. This mismatch should be discussed and/or fixed.- Citations for the results mentioned in Lines 60-62 are missing.- When Theorems 4 and 5 are cited in Line 183, please say they can be found in the supplementary material. Though most theory conferences accept long papers describing their essential contributions in the first pages, NIPS still has a clear division between main paper and supplementary material, and the former is supposed to be self-contained.- Lemma 1 should say that the algorithm in the thesis is also private. Furthermore, the relevance of requiring in the hypothesis of the lemma that the first algorithm has a particular sample complexity is not entirely clear.- Non-increasing -> Non-decreasing (Line 240)- \hat{P}_n -> \hat{p}_n (Line 240)- distance Combining (Line 215)- curve For example (Line 251)