ATOMO: Communication-efficient Learning via Atomic Sparsification
Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any possible atomic decomposition. Notable examples include element-wise, singular value, and Fourier decompositions. We present ATOMO, a general framework for atomic sparsification of stochastic gradients. Given a gradient, an atomic decomposition, and a sparsity budget, ATOMO gives a random unbiased sparsification of the atoms minimizing variance. We show that recent methods such as QSGD and TernGrad are special cases of ATOMO and that sparsifiying the singular value decomposition of neural networks gradients, rather than their coordinates, can lead to significantly faster distributed training.
After rebutal; I do not wish to change my evaluation. Regarding convergence, I think that this should be clarified in the paper, to at least ensure that this is not producting divergent sequences under resaonable assumptions. As for the variance, the author control the variance of a certain variable \hat{g} given g but they should control the variance of \hat{g} without conditioning to invoke general convergence results. This is very minor but should be mentioned.      The authors consider the problem of empirical risk minimization using a distributed stochastic gradient descent algorithm. In this setting, communication cost constitutes a significant bottleneck. The authors generalize recently proposed scheme which aims at obtaining random sparse unbiased estimates of a vector while ensuring minimal variance. The authors describe an active set algorithm which allows to perform this operation given decomposition in an orthonormal basis. Extensive experiments based on low rank approximation of the gradient matrix suggest that the proposed approach allows to speedup convergence by reducing the communication cost.   Main comments I have very limited knowledge about distributed optimization, so I might miss some important points or shortcomings in the paper.  Overall the paper is well written and seems to reference correctly the corresponding litterature. The proposed algorithm is quite natural. I did not have a careful look at all proof details but I could not see anything which looked abnormal. I am not really able to criticize numerical experiments beyond the fact that they are convincing to me.  The overall mathematical exposition is quite elementary, some elements could be cut out, for example all the Lemmas related to algorithm 1 may not be necessary in the main text, this is actually a simple active set method. Another example is the notion of equivalence of norms and Theorem 10, this can be found in a first year analysis course and does not bring much to the paper. Lemma 8 is also well known.   The question of the convergence of this process is not touched. How does the sparsification affect convergence?  Why not comparing to ternGrad?    Minor comments  I am not quite sure about the relevance of the concept of "atomic decomposition". From definition 1, this is just decomposition in an orthonormal basis, which is a less fancy name but more standard as a mathematical concept.  In Theorem 5, what does $j = argmax$ mean in the second statement? Similarly what is "i" in the first statement. Logical quantifiers should be used here.  Figure 1, I would rather say top 2 values standing out  The constraint "can be represented with k bits" is not really meaningful. Any character string can be compressed to a single bit using the appropriate dictionary. May be the authors have in mind a specific representation or model.  "In what follows, you may think of g as the stochastic gradient" is not at the usual level of formality in scientific papers.  Line 203: we s and suppose