Differentially Private k-Means with Constant Multiplicative Error
We design new differentially private algorithms for the Euclidean k-means problem, both in the centralized model and in the local model of differential privacy. In both models, our algorithms achieve significantly improved error guarantees than the previous state-of-the-art. In addition, in the local model, our algorithm significantly reduces the number of interaction rounds. Although the problem has been widely studied in the context of differential privacy, all of the existing constructions achieve only super constant approximation factors. We present-for the first time-efficient private algorithms for the problem with constant multiplicative error. Furthermore, we show how to modify our algorithms so they compute private coresets for k-means clustering in both models.
: Private algorithms for k-means. Here n is the number of input points, k is the number of centers, and d is the dimension. For simplicity, we assume that input points come from the unit ball, and omit the dependency in ε, as well as logarithmic factors in k, n, d, β, δ, from the additive error.
Before describing our new results, we define our setting more precisely. Consider an input database S = (x 1 , . . . , x n ) ∈ (R d ) n containing n points in R d , where every point x i ∈ S is the (sensitive) information of one individual. The goal is to identify a set of k centers C = {c 1 , . . . , c k } in R d approximately minimizing the following quantity, referred to as the cost of the centers
The privacy requirement is that the output of our algorithm (the set of centers) does not reveal information that is specific to any single individual. Formally, Definition 1.1 ( [12]). A randomized algorithm A : X n → Y is (ε, δ) differentially private if for every two databases S, S ∈ X n that differ in one point, and every set T ⊆ Y , we have Pr[A(S) ∈ T ] ≤ e ε · Pr[A(S ) ∈ T ] + δ.
Combining the utility and privacy requirements, we are seeking for a computationally efficient differentially private algorithm that identifies a set of k centers C such that w.h.p. cost S (C) ≤ γ · OPT S +η, where OPT S is the optimal cost. We want γ and η to be as small as possible, as a function of the number of input points n, the dimension d, the number of centers k, the failure probability β, and the privacy parameters ε, δ.
We remark that a direct consequence of the definition of differential privacy is that, unlike in the non-private literature, every private algorithm for this problem must have additive error η > 0. In fact, if all points reside with the d-dimensional ball, B(0, Λ), of radius Λ around the origin (as we assume in this paper) then η must be at least Λ 2 . To see this, consider k + 1 locations p 1 , . . . , p k+1 at pairwise distances Λ, and consider the following two neighboring datasets. The first dataset S 1 contains n − k + 1 copies of p 1 , and (one copy of) p 2 , . . . , p k . The second dataset S 2 is obtained from S 1 by replacing p k with p k+1 . Since in both cases there are only k distinct input points, the optimal cost for each of these datasets is zero. On the other hand, by the constraint of differential privacy, the set of centers we compute essentially cannot be affected by this change. Therefore we must have expected error of Ω(Λ 2 ) on at least one of these inputs. To simplify the presentation we assume that Λ = 1 in rest of the introduction.
Traditionally, in the non-private literature, the goal is to minimize the multiplicative error γ, with the current state-of-the-art (non-private) algorithm achieving multiplicative error of γ = 6.357 (with no additive error) [2]. In contrast, in spite of the long line of works on private k-means [8,28,14,18,27,33,31,32,17,6,29,21], all of the existing polynomial time private algorithms for the problem obtained only a super constant multiplicative error. We present the first polynomial time differentially private algorithm for the Euclidean k-means problem with constant multiplicative error, while essentially keeping the additive error the same as in previous state-of-the-art results. See Table 1 for a comparison.
The paper considers k-means under differential privacy. The main idea of differential privacy (applied to k-means) is that changing one point in the input should not significantly change the centers that are computed. This immediately means that any approximation algorithm for the differentially private k-means problem needs to have additive error: The introduction includes a simple example of points in k locations (=> optimum has zero cost). Now when moving one of these points to a different location, the centers may not change significantly; ergo one of the solutions has to have a large additive error; basically an error of D^2 where D is the largest pairwise distance. Thus, a common approach for this problem is to compute an approximation that is both multiplicative and additive. The best known result prior to this paper achieves a quality of O(k)*OPT + O~(d^0.51+k^1.51).  The paper dramatically reduces this guarantee to O(1)*OPT + O~(k^1.01 d^0.51 + k^1.5). The new algorithm works both in the centralized variant where a trusted server provides anonymization, and in the local variant, where the users themselves randomly change their point before sending it to an untrusted server. The results look very good to me; better multiplicative approximation, no increase in the additive term, and in the local model, the number of rounds is also decreased. My only concern is whether the paper really fits NIPS, because a) not very much of the proof makes it into the actual paper, and b) there are no experiments. It's not clear to me how practical the algorithm is; the main routine does not look to complicated, but the subroutine might be. Aside from the missing experiments, this is clearly the best paper in my (not necessarily representative) batch. I do not know whether a paper without experiments can appear at NIPS; I'll vote for acceptance and wait for the other reviewer's opinion on that.  The introduction is very well written. The problem definition becomes clear immediately, and the need for additive approximation is explained nicely. The best non-private approximation for k-means is cited. I'm no expert on differentially private k-means algorithms, but the related work section wrt that topic looks solid to me as well. I appreciated the paragraph about classical algorithm not being private, and the pointer to the k-means++ adaptation. The technical contribution is also made very clear; previous work is summarized clearly. The paper builds upon a previously obtained result which essentially gives a differentially private algorithm for k-means if and only if one can supplement it with a set of good center candidates of poly(n) size. At first glance, this seems easy, since for non-private k-means, one can simply take the input points as the candidate set, this is of size n and has an error of 2 compared to centers from R^d. However, for private k-means clustering, this is not possible -- the centers can not be a subset of the input points, because otherwise, removing a point may significantly change the computed centers. The paper has two main contributions. First, it shows how to compute a good center set; this part is based on a different related paper, but seems refined. Second, it embeds both the candidate set computation and the subroutine that computes good centers given such a candidate set into a larger main algorithm. Roughly, this algorithm computes a candidate set, uses it to compute centers, and then eliminates points which it now deems "covered" by the new centers. This process is then repeated log log n times, and then the probability that all points are covered is sufficiently high.  A question for the author feedback: Just for my clarification: How does the fact that we can not pick S as the candidate center set show in Theorem 2.1? What if I just hand the algorithm in [16,6] Y=S, how does reading the theorem tell me that that does not lead to a differentially private algorithm?  Smaller remarks: - In Table 1, the ~ notation hides log n - factors. As far as I thought, the O~(X) is usually used to hide factors that are polylogarithmic in X -- but n is not a part of the expression here. It is stated in the caption of Table 1 that logarithmic factors in n are omitted; yet I think the dependence on log n should be stated somewhere in the introduction in the text (if I missed it, sorry). - The last sentence on page 3 (line 112+113) confused me quite a bit. It might be a good point in the paper to describe that one can not simply take the input points as a candidate set. This is obvious once observed, but when usually working on non-private k-means, the statement that an O(1)-approximation would work for a discrete set and is simply not applicable in R^d just sounds strange. - The citations for non-private coresets in line 178 seem a bit arbitrary; they should probably contain at least Dan Feldman, Michael Langberg: A unified framework for approximating and clustering data. STOC 2011: 569-578 which is to my knowledge the currently best for k-means. - Theorem 2.1: "a finite set centers" -> "a finite set of centers" - The sentence above Lemma 3.5 that the proof is omitted seems to be not true, since the proof is there.  Edit after author feedback: I like this submission, and this has not changed.