Differentially Private Subspace Clustering
Subspace clustering is an unsupervised learning problem that aims at grouping data points into multiple "clusters" so that data points in a single cluster lie approximately on a low-dimensional linear subspace. It is originally motivated by 3D motion segmentation in computer vision, but has recently been generically applied to a wide range of statistical machine learning problems, which often involves sensitive datasets about human subjects. This raises a dire concern for data privacy. In this work, we build on the framework of differential privacy and present two provably private subspace clustering algorithms. We demonstrate via both theory and experiments that one of the presented methods enjoys formal privacy and utility guarantees; the other one asymptotically preserves differential privacy while having good performance in practice. Along the course of the proof, we also obtain two new provable guarantees for the agnostic subspace clustering and the graph connectivity problem which might be of independent interests.
General comments:The paper is well written. I would just remove "cryptographic" in the abstract. Not a lot of people are familiar with differential privacy and many of them would get confused with the notion of cryptography in which there exists an algorithm that reverts the "encrypted" data to its original form (I consider that once the data is noisified, its original form is essentially "lost"). Technical comments:Following the argument in the supplementary information of ref. [30], the authors argue that being able to find a perfect SSC clustering breaches the notion of differential privacy (DP). Agreed. * But * the point is that finding a perfect SSC clustering is just computationally impossible.Consider the following problem:Name: SSC approximation. Instance: a set of vectors in R ^ d, integer q\ geq 1, integer k\ geq 1, bound b\ geq 0. Question: does there exist a set of k subspaces of dimension q each such that the subspace clustering objective (as defined by (3) in the paper) is not more than b.This problem is APX-Complete from reduction from Vertex cover (vertices X, edges E): d is the number of vertices, and for each element of E, create one vector with the indicator variable for the two vertices of X. Fix k = the same as in X3C, b = 1, q = 1. Since SSC solution gives orthonormal bases, solution to SSC approximation gives exactly the indexes of the vertices in X which cover all edges (since b = 1).Hence, there exists a constant\ gamma\geq 0 such that if P\ neq NP, SSC approximation is hard to approximate within 1+ \ gamma.So any SSC clustering can only guarantee to find an approximation to the optimum. Finally, SSC clustering is no "tougher" for privacy than other models, like the (more) popular barycenter-based clustering algorithms of k-means/EM.However, of course, it is crucial to address SSC clustering in a private framework.Then, the authors justify the use of a local-sensitivity approach by the fact that global sensitivity would be too high and thus preclude de facto any efficient DP method for the clustering problem. Agreed. * But * then the weaker model somehow imposes to find very efficient algorithms, as otherwise there would be no reason to weaken privacy for (still) inefficient algorithms. From this standpoint, Nissim's approach (ref. [23]) makes clear the fact that the dependence on noise is crucial, and non-trivial to solve even in the SAF framework. In the longer version of their paper, Nissim et al. make this clear (Section 5.1 of www.cse.psu.edu / ~ads22 / pubs / NRS07 / NRS07-full-draft-v1.pdf). Unfortunately, the submission fails to bring to SSC clustering what Nissim et al. could not bring to k-means-type clustering. Actually, the results look comparable from a high-level standpoint. Nissim et al approximate the Wasserstein distance DW (Lemma 5.6 in the longer version) to O (sigma k sqrt(d / n)) while the guarantee in DW for the submission is O (sigma k q log n). Why is it that DW does not decrease at least reasonably when n increases in Theorem 3.6 ? The data being i.i.d., I would expect this to happen. Furthermore, Lemma 3.5 (the key to Theorem 3.6) is very difficult to appreciate from the qualitative and quantitative standpoints. Drilling down into the conditions, we arrive at the following conclusions: for the probability to be 1-O(1), we *must* have n = O (exp (sqrt(d) / 2)) (first minus parameter in probability), which precludes too large samples. However, looking at the second minus parameter, Jensen's inequality brings n sum exp(-n_l / 400)\ geq k n exp (-n / (400k)). If k is, say, 10, then to have that last parameter say no more than 1 / 10, we must have exp(n / 4000) \geq 100 n, and thus n \geq 50 000. If d is not large enough, say d \leq 500, the bound is meaningless. And I am not even discussing the three assumptions of the lemma, that are just incredibly hard to capture to figure out a "competency map" of the Lemma that would give sufficient high-level constraints for the "1 - lots_of_things" to be not far from 1 (or even not negative...).Figuring out such "competency maps" is important and would prevent some "nasty" collateral consequences. For example, Lemma 3.3 is stated with $ \ phi ^ 2 \leq 1 / 802$. However, the constraint cost(hat(C)) \leq a cost (Copt) with a \leq (1-802phi ^ 2) / (800phi ^ 2) imposes, for the right hand side to be larger than 1 (otherwise the statement cannot hold), (1-802phi ^ 2) / (800phi ^ 2) \geq 1, and so... phi ^ 2 \leq 1 / 1602... The exponential mechanism with Gibbs sampling is original and interesting.The experiments give interesting results, * But * the following downsides apply: 1) the value of delta is a bit large, considering that delta = 1 / n is the trivial lower bound for which not doing any private mechanism still yields (epsilon, delta) differential privacy. 2) the value of n, d, k are clearly out of the applicable bounds for the theory, so we cannot use the experiments to validate in any way the ideas of the theory. 3) the values of epsilon in the plots is also too large, considering that in most applications, we would have epsilon \leq 1, thus truncating the plots to the [-1, 0] interval on the x-axis. In this case, the approach proposed by the authors clearly gives less compelling results.Finally, the lack of a conclusion, even short, summarising the pros and the cons, and opening future research directions is not a good thing either.Summary: the paper is original and its objectives are clear, but it needs some important technical revamping for the theory to be of broad relevance and be technically clear, in particular for the underlying assumptions to be unquestionable. The dependence in noise needs to be improved (or, at least, it must be shown why this large dependence in noise is absolutely necessary) to improve the paper's significance.