The Large Margin Mechanism for Differentially Private Maximization
A basic problem in the design of privacy-preserving algorithms is the private maximization problem: the goal is to pick an item from a universe that (approximately) maximizes a data-dependent function, all under the constraint of differential privacy. This problem has been used as a sub-routine in many privacy-preserving algorithms for statistics and machine learning. Previous algorithms for this problem are either range-dependent-i.e., their utility diminishes with the size of the universe-or only apply to very restricted function classes. This work provides the first general purpose, range-independent algorithm for private maximization that guarantees approximate differential privacy. Its applicability is demonstrated on two fundamental tasks in data mining and machine learning.
This paper looks at differentially private algorithms for a generic maximization problem ("private argmax" might be a good name). Given a collection of K of items, and a data set D of n individuals, and a "score" function f that assigns each item i a data-based score f(i;D), the goal is to find an item i with approximately maximal score, while preserving differential privacy. This "private argmax" has proven to be a fundamental problem in the theory of private data analysis. It was first formulated by McSherry and Talwar (2007), who proposed the "exponential mechanism" to solve it. Because of its generality, that mechanism has been a central tool for designing many differentially private algorithms. The performance of that algorithm is optimal in the worst case, even among "approximately DP" algorithms, but it has been known for some time that there are various simple conditions under which it can be improved substantially.The authors give a private argmax algorithm which is differentially private (actually, a commonly-studied relaxation called "approximate DP") and which does better than the exponential mechanism for a large class of inputs. The authors call these instances "large margin" instances -- essentially, they assume that there are not too many items with near-optimal margin. Previously, it was known how to solve this problem when there is a large gap between the best and second-best items; this paper generalizes that technique considerably. The authors illustrate their results by showing theoretical improvements for a two problems. The more interesting of these is PAC learning -- the authors show that their algorithms allows one to exploit "shell bounds" (essentially, conditions which imply that not too many concept classes are good for the data set with high probability). This is a nice result that has the potential to influence future work considerably, as it provides a useful and basic algorithmic tool. It is not completely unexpected, given recent work of a similar flavor, but it takes those ideas much further. I recommend accepting the paper. The paper also presents lower bounds showing that the parameters of the algorithm cannot be asymptotically improved. The second of these (for (alpha,delta)-DP) is a nice contribution independently of the rest of the paper. Comments: I think the presentation of the algorithm could be somewhat simplified. - For any given ell, there is a simple mechanism which does well and is private *under the (ell,gamma) assumption* for gamma >= log(\ell/delta)/alpha. Namely, just run the exponential mechanism on the top \ell items. (In a nutshell, it is private since the probability that one outputs an item with frequency f_\ell is at most delta). This is somewhat simpler than the truncated exponential mechanism the authors employ. - One can select a good ell using the sparse vector technique. This is essentially what the authors do.Such a two-stage explanation might make the algorithm a little easier to parse.This paper would probably be more appealing to the NIPS community with some sort of empirical evaluation, but I think it is more than strong enough as it is for acceptance.