Communication-Efficient Distributed Dual Coordinate Ascent
Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning. In this paper, we propose a communication-efficient framework, COCOA, that uses local computation in a primal-dual setting to dramatically reduce the amount of necessary communication. We provide a strong convergence rate analysis for this class of algorithms, as well as experiments on real-world distributed datasets with implementations in Spark. In our experiments, we find that as compared to stateof-the-art mini-batch versions of SGD and SDCA algorithms, COCOA converges to the same .001-accurate solution quality on average 25× as quickly.
The paper describes a parallel variant of SDCA algorithm which each machine executes SDCA updates using local parameters and these local parameters are periodically averaged.  Theorem 3 provides the convergence guarantee of the algorithm.Quality: Unfortunately, Theorem 3 is not very interesting since the bound is a function of T/K; as T should be a linear function of K at best, there will be no speed-up by increasing K, the number of workers.  What this means is that if we use beta_K = 1, which is the only case the theorem covers, then the algorithm will become more and more conservative as we increase the number of machines, and compared to the case of single-thread execution the algorithm will not be any faster.  In practice we could make the algorithm more aggressive by using beta_K > 1, but the paper does not provide theoretical support.  The empirical comparison against other methods seems motivating, but unfortunately an experiment on scaling behavior of the algorithm as a function of K is missing, which I believe is critically important considering Theorem 3 does not guarantee speed-up.Clarity: Lemmas and theorems in the paper are well-organized and easy to follow, since authors have done a good job in providing intuitive explanation of the proof.Originality: The proposed algorithm is a generalization of a popular heuristic which local parameters are periodically averaged, and the proof does seem to provide new techniques or insights; the proof of Lemma 2 does not depart much from the original proof for SDCA, and then the main theorem (Theorem 3) is just an application of Jensen's on top of Lemma 2.  This is why Theorem 3 is weak and does not provide any speed-up guarantee;  other parallel algorithms such as 11 or 21 departs from the proof of single-thread case by taking account of correlation between data points and therefore are able to provide speed-up guarantees.  I believe it should be necessary as well for parallel SDCA algorithm to consider correlation between data points, if one wants to achieve a nontrivial convergence guarantee.Significance: The paper provides a nice insight that H, the period of synchronization, might have to be a function of C, the local convergence speed.  However, this insight is based on a loose convergence guarantee (Theorem 3) which does not provide any parallelization speedup.  Even worse, since C is not known a priori, one cannot choose H without actually executing the algorithm.