Gradient Sparsification for Communication-Efficient Distributed Optimization
Modern large-scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communication cost, we propose a convex optimization formulation to minimize the coding length of stochastic gradients. The key idea is to randomly drop out coordinates of the stochastic gradient vectors and amplify the remaining coordinates appropriately to ensure the sparsified gradient to be unbiased. To solve the optimal sparsification efficiently, a simple and fast algorithm is proposed for an approximate solution, with a theoretical guarantee for sparseness. Experiments on 2 -regularized logistic regression, support vector machines and convolutional neural networks validate our sparsification approaches.
After rebuttal:  In my opinion, this paper falls slightly short of the mark for NIPS, for the reasons outlined in my original review, but I believe it's worth communicating, and will not oppose its acceptance. I slightly upgraded my score since I agree to some extent that the algorithmic idea is not the same as QSGD; the ultimate decision lies with the area chair.   ====  There's been recently a lot of work on the communication-variance trade-offs of distributed SGD. The present submission fits squarely into this space, by looking at the problem of sparsifying SGD gradients stochastically, to fit a certain variance budget. Thus, it aims to improve upon recent work like TernGrad and QSGD, which mainly try to reduce the communication budget by reducing bit-width (not necessarily sparsifying) gradients.   The results can be summarized as follows:  - The authors give an optimal scheme for fitting the sparsity in a gradient stochastically to a fixed variance budget. This is done by modeling the problem of setting choice probabilities for gradient entries in stochastic quantization as an optimization problem, for which the authors show that there is a nice closed-form solution. They then give an efficient way of implementing this. - The scheme is shown to provide both sparsity and variance bound guarantees. - The authors provide a relatively wide array of small benchmarks, for convex objectives (regression), non-convex (toy neural nets), and asynchronous parallel SGD. The experiments show that the scheme works well when compared with random and naive sampling approaches, and similarly (or a bit better) when compared with QSGD.   My evaluation is as follows:  On the positive side, the solution given here is reasonable, and appears correct to my reading. I think this is a nice set of results, which should be communicated in some form.   On the negative side, I have serious doubts that this crosses the high bar set by NIPS.   First, in terms of significance and improvement over previous work, the contribution here is, in my opinion, limited. The fact that stochastic quantization induces sparsity for low target bit width is not new. A basic version of this observation appears to be given in the QSGD paper: if we take their version with two quantization levels (0 and 1) it seems to me we get a version of this sparsification method with sqrt n density and sqrt n variance. The authors' contribution here is to set the question of choosing sampling probabilities efficiently given a specific variance budget as an optimization problem, which they solve efficiently. However, their optimal solution is quite similar to the QSGD scheme for a single non-zero quantization level. Thus, simplistically, it would seem to me like the contribution here is to determine the optimal scaling parameter for the sampling probabilities?  Moreover, there are other related non-stochastic delay-based sparsification schemes, e.g. [1]. To my knowledge, these schemes do not have convergence guarantees, but they seem to ensure extremely high sparsity in practice for neural nets (an order of magnitude larger than what is considered here in the experimental section). Yet, there is minimal comparison with these, even just at the experimental level. Is the authors' claim that such schemes do not converge for convex objectives? This point is left ambiguous.   Second, the experimental section is at the level of toy synthetic experiments, and therefore leaves much to be desired. I believe that validation on convex objectives is nice, and I haven't seen it in other papers in this area. But this is because communication-compression for regression problems is not really a critical performance problem these days, at least not at the scale shown here.  The non-convex validation is done on toy CNNs, which would have no problem scaling on any decent setup. By contrast, the standard in the area is ImageNet or large-scale speech datasets, for which scaling is indeed a problem. Such experiments are not given here.  The asynchronous experiment is more intriguing, but ultimately flawed. Basically, the idea here is that sparsity will help if threads compete on a bunch of memory locations (corresponding to the parameter). To boost contention, the authors *lock* each individual location (to my understanding). But why is this necessary(other than making the experiments look more impressive)? Modern computer architectures have compare&swap or fetch-and-add operations which could be used here, which wouldn't block access to those locations like locking does. These should be much faster in practice, and I believe standard analyses cover such asynchronous variants as well. Have the authors actually tried this?  Another thing which the authors might want to keep in mind here is that, even though they ensure non-trivial sparsity (e.g. 70%) at each node, collectively the summed updates (gradients) may well become dense for reasonable node count. Hence this technique will not be very useful in any of the cases considered unless #dimension / #nodes >> sparsity.    Summing up, while I have enjoyed reading this paper, I believe this is clearly below the threshold for NIPS, both on the theory and on the experimental side.   Some minor comments:  - sparsification technology -> "technique" is probably better here - you keep changing between [] and () for expectation - "The algorithm is much easier to implement, and computationally more efficient on parallel computing architecture." Why? - "after sparsified" -> please rephrase - "baseline of uniform sampling approach" -> please rephrase - Figure 2: overwriting the experimental results with the legend seems like a bad idea to me. - "converges much faster" -> what is "much" here? please be more precise - "employs a locked read" please be clearer with your experimental setup here. Are these reader-writer locks? phtread locks? test&set locks? The whole section should be re-written for clarity.