The Convergence of Sparsified Gradient Methods
Stochastic Gradient Descent (SGD) has become the standard tool for distributed training of massive machine learning models, in particular deep neural networks. Several families of communication-reduction methods, such as quantization, largebatch methods, and gradient sparsification, have been proposed to reduce the overheads of distribution. To date, gradient sparsification methods-where each node sorts gradients by magnitude, and only communicates a subset of the components, accumulating the rest locally-are known to yield some of the largest practical gains. Such methods can reduce the amount of communication per step by up to three orders of magnitude, while preserving model accuracy. Yet, this family of methods currently has no theoretical justification. This is the question we address in this paper. We prove that, under analytic assumptions, sparsifying gradients by magnitude with local error correction provides convergence guarantees, for both convex and non-convex smooth objectives, for data-parallel SGD. The main insight is that sparsification methods implicitly maintain bounds on the maximum impact of stale updates, thanks to selection by magnitude. Our analysis also reveals that these methods do require analytical conditions to converge well, justifying and complementing existing heuristics.
The paper proposes a theoretical study of a heuristic method for distributed SGD, using the sparsification gradients techniques.  The technique is not new, but this theoretical study is, in particular authors claim that their analysis does not require a bound on the maximum delay, which departs from already known analysis for asynchronous variants of SGD. I am however not confortable with the main assumption of the paper (Assumption 1), which is not discussed enough in my opinion, or numerically checked in experiments. It is I believe a strong assumption, that provides an ad-hoc control of the TopK operator. A comparison with the bounded delay assumption for asynchronous SGD is not fair, since this assumption is usually satisfied / numerically checked in experiments, while it is hardly the case of Assumption 1 for the TopK SGD. I think that the numerical experiments are weak : the comparison with asynchronous SGD is on a convex problem only (while the paper markets the use of TopK SGD for deep nets) and only illustrate marginal improvements. As a conclusion, I find the paper interesting, and won't oppose its acceptance, but I think that it is only marginally above the acceptance threshold for the NIPS venue.