Asynchronous Parallel Greedy Coordinate Descent
In this paper, we propose and study an Asynchronous parallel Greedy Coordinate Descent (Asy-GCD) algorithm for minimizing a smooth function with bounded constraints. At each iteration, workers asynchronously conduct greedy coordinate descent updates on a block of variables. In the first part of the paper, we analyze the theoretical behavior of Asy-GCD and prove a linear convergence rate. In the second part, we develop an efficient kernel SVM solver based on Asy-GCD in the shared memory multi-core setting. Since our algorithm is fully asynchronous-each core does not need to idle and wait for the other cores-the resulting algorithm enjoys good speedup and outperforms existing multi-core kernel SVM solvers including asynchronous stochastic coordinate descent and multi-core LIBSVM.
This paper considers asynchronous parallel greedy coordinate descent (Asy-GCD) algorithm for minimizing a smooth function with decomposed constraints.  The algorithm works in a parameter server framework, with each worker performing greedy CD asynchronous.  This is different from [17] which picks the coordinate stochastically.  Efficient coordinate search algorithm is proposed for kernel SVM by maintaining the gradient, which is updated by atomic writes.  The results are mainly two folds.  First in theory, a linear convergence rate is established.  Second in practice, by using kernel caching and sensible variable partitioning, the proposed algorithm outperforms existing solvers in a shared-memory multi-core setting.
This paper is well written and the innovations and improvements can be seen clearly.  Although changing stochastic into greedy is not that big innovation and the O(l) algorithm for coordinate selection based on maintaining the gradient is also not new, I can kind of appreciate the practical improvement after putting together some existing tricks (Sec 4.3).  My major concern is, however, the rates of convergence.  There are five “constants” involved in Theorem 2, \kappa, L_res, L_max, L, and L_T.  It is critical to unfold their values for concrete problems, e.g. the kernel SVM problem considered in the paper.  Linear rates are useful only if its condition number depends on the dimensions of the problem mildly.  What is most unsettling is the L_T^2 term in b.  How does L_T depend on T?  If it is \Omega(\sqrt{T}), then this will translate to an iteration complexity that is linear in T.  Since “T is usually proportional to the total number of cores involved in the computation”, this will cancel the speedup proffered by parallelization and asynchronous updates.  It is therefore crucial that the paper clarify the value of these “constants” explicitly.  =================== Post-rebuttal:  I read the rebuttal, and now feel less worried about the dependence on L_T (or T). However, based on the numbers provided in the rebuttal, now I am rather worried about the constants \kappa and L.  Using the rebuttal's argument that b is almost a universal constant, the rate of convergence is roughly  L n \kappa^2 \log (1/\epsilon)  The values of L and \kappa provided in the rebuttal seem to grow linearly in n.  This is unsettling because such a high order dependency on n essentially leaves the linear rate in vain.  The submission allows parallelization up to n^{1/4} cores.  So roughly speaking, the cost is   O(n^{15/4} \log (1/\epsilon))   15/4 might be over-estimated a little, but it is far worse than the following rate for training nonlinear kernel SVM given by [a]:  O(n / \epsilon).  The disadvantage is even more evident considering that \epsilon rarely needs to be less than 1/n (consider n = 4*10^5 in the experiment where #example is denoted as \ell instead of n).  Having said that, I think think this paper actually has some high quality material.  The experiments are surely good.  The linear rates were first shown for this problem, although further refinement is needed on its condition number.  Overall, this paper is above the acceptance threshold of NIPS, and I've updated my ratings accordingly.  [a] Bernd Gärtner and Martin Jaggi, Coresets for Polytope Distance, Symposium on Computational Geometry, 2009.