Communication-Optimal Distributed Clustering *
Clustering large datasets is a fundamental problem with a number of applications in machine learning. Data is often collected on different sites and clustering needs to be performed in a distributed manner with low communication. We would like the quality of the clustering in the distributed setting to match that in the centralized setting for which all the data resides on a single site. In this work, we study both graph and geometric clustering problems in two distributed models: (1) a point-to-point model, and (2) a model with a broadcast channel. We give protocols in both models which we show are nearly optimal by proving almost matching communication lower bounds. Our work highlights the surprising power of a broadcast channel for clustering problems; roughly speaking, to spectrally cluster n points or n vertices in a graph distributed across s servers, for a worst-case partitioning the communication complexity in a point-to-point model is n · s, while in the broadcast model it is n + s. A similar phenomenon holds for the geometric setting as well. We implement our algorithms and demonstrate this phenomenon on real life datasets, showing that our algorithms are also very efficient in practice.
The paper proposes several distributed clustering algorithms and study the communication complexity in two models (point to point and blackboard). The authors prove lower and upper bounds for graph spectral clustering, and geometric clustering (i.e. k-means, k-median,...). The algorithms are implemented and evaluated on large graphs (the edge size is larger than 70 million).         
  In this paper, a large dataset is maintained in s sites that communicate in a point to point protocol (private message passing between sites and one coordinator) or in a blackboard protocol (broadcast messages to a centralized site).   The proposed algorithm for spectral graph clustering within the message passing model is based on spectral sparsification.   The contribution is to show the the sparsification can be computed  locally. The proof  that the clustering the union of  the sparse subgraphs is close to the clustering on the whole graph highly relies on [19, Peng et al, COLT 2015]. The lower bound is based on a reduction to the multiparty set disjointness problem.  The algorithm in the Blackboard model relies on the iterative construction of a sequence of matrices of length O(log n) whose last element is a spectral sparsifier. The presentation of this algorithm is less clear than the previous one. In particular it is hard to follow what can be done locally and when the communication with the coordinator is needed (I think that a communication is needed at each step to compute \tau_i).  In both cases, the proofs in appendix seem to be necessary to understand the algorithms. So the presentation could be improved.     The experiments confirm the theoretical results.    I think that the paper is interesting and deserves publication.    Typos and technical remarks.  - Lines 469/471: \gamma_u is \lambda_u. Line 475: I don't understand the append operator.  - I think that a concluding Theorem could be added to Section 3.2.  - It should be more clear if you announce sooner that the reduction will encode X_i^j=1 if item j *is not* in site i.