Improved Distributed Principal Component Analysis
We study the distributed computing setting in which there are multiple servers, each holding a set of points, who wish to compute functions on the union of their point sets. A key task in this setting is Principal Component Analysis (PCA), in which the servers would like to compute a low dimensional subspace capturing as much of the variance of the union of their point sets as possible. Given a procedure for approximate PCA, one can use it to approximately solve problems such as k-means clustering and low rank approximation. The essential properties of an approximate distributed PCA algorithm are its communication cost and computational efficiency for a given desired accuracy in downstream applications. We give new algorithms and analyses for distributed PCA which lead to improved communication and computational costs for k-means clustering and related problems. Our empirical study on real world data shows a speedup of orders of magnitude, preserving communication with only a negligible degradation in solution quality. Some of these techniques we develop, such as a general transformation from a constant success probability subspace embedding to a high success probability subspace embedding with a dimension and sparsity independent of the success probability, may be of independent interest.
This paper considers the problem of trading off the communication and computation cost of distributed computation and proposes a new distributed k L-2 error fitting algorithm. The proposed algorithm can be seen as a combination of many previous speed up techniques for distributed PCA and clustering methods. However, the authors also contribute optimizations over the base methods and further improves the communication and computation efficiency. The theoretical guarantee is sound and experiments are convincing.What is the subroutine of A_\alpha in Algorithm 1? Moreover, algorithm 2 involves algorithm 1 which uses A_\alpha. However, it seems in Theorem 6 that both the communication and computation cost is independent of \alpha? Could the authors provide some explanations?Line 275, how shall we understand that for the subspace embedding, there is \|HAy\|_2 = (1\pm \epsilon)\|Ay\|_2?In Theorem 6, the failure probability depends on s and t. I suppose t is equal to t1 and t2, which also depends on s. Could the authors optimize the probability and remove the dependence on t? As for the current statement, suppose t1=t2=O(\log s), then the probability is 1-O(1/s+1). Is this result good enough? Could the authors provide the explicit form for the constant c_0? Such that the readers can better understand the tightness of the bound.My biggest concern is about the performance measure. As mentioned in line 85, it is desired to find a center set \mathcal{L}’ such that the relative error is small, d^2(P, \mathcal{L}’ ) \leq (1+\epsilon) \min_{\mathcal{L}}d^2(P,\mathcal{L}). It is expected to see a similar relative error bound for algorithm 2 in Theorem 6. In the experiments, the authors should provide the value of parameter \epsilon, or the values of t1 and t2. Minor comments:Line 93, I suggest to use “reducing communication cost” as the paragraph title, which more clearly expressed the contributions compared with “improved communication”. Similarly, line 108, “improved computation” should also be revised.