Combining Global Sparse Gradients with Local Gradients
Data-parallel neural network training is network-intensive, so gradient dropping was designed to exchange only large gradients. However, gradient dropping has been shown to slow convergence. We propose to improve convergence by having each node combine its locally computed gradient with the sparse global gradient exchanged over the network. We empirically confirm with machine translation tasks that gradient dropping with local gradients approaches convergence 48% faster than non-compressed multi-node training and 28% faster compared to vanilla gradient dropping. We also show that gradient dropping with a local gradient update does not reduce the model*s final quality.
 This paper proposes a 3 modes for combining local and global gradients to better use more computing nodes. The paper is well-written and well-motivated. I question why it is a good idea to use sparse gradient in a neural network where all coordinates (i.e. hidden units) are exchangeable. This paper claims that "gradient values are skewed, as most are close to zero", but there is no references nor statistics. The only source of sparsity for a typical MT model are those for word vectors, but in that case, perhaps it would be better to develop algorithms that specifically make use of that type of sparsity. The quality degradation in methods like gradient dropping seems to suggest that some arbitrary cutoff for sparsity is not a good idea. For the results, the proposed method running on 4 nodes is compared to synchronous SGD, and only got 25% and 45% speedups on the 2 tests. Notably, they do not compare to asynchronous SGD with stale gradients. Such a test would help offer empirical support/refutation on the assumption that the sparse gradient is a good idea in the first place. I think these results are not good enough to justify the extra complications in the proposed method. The paper is well-written, however AllReduce seems undefined, and I assumed it aggregates sparse gradients from all nodes. I also a bit confused about the characteristics of ApplyOptimizer. In gradient dropping, ApplyOptimizer takes a sparse gradient, but in the proposed methods, it takes a dense gradient. I am a bit confused on how exactly the synchronous update is done, and I*d appreciate it if you can make your model of parallel updates more explicit.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Combining Global Sparse Gradients with Local Gradients
Data-parallel neural network training is network-intensive, so gradient dropping was designed to exchange only large gradients. However, gradient dropping has been shown to slow convergence. We propose to improve convergence by having each node combine its locally computed gradient with the sparse global gradient exchanged over the network. We empirically confirm with machine translation tasks that gradient dropping with local gradients approaches convergence 48% faster than non-compressed multi-node training and 28% faster compared to vanilla gradient dropping. We also show that gradient dropping with a local gradient update does not reduce the model*s final quality.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes a 3 modes for combining local and global gradients to better use more computing nodes. The paper is well-written and well-motivated. I question why it is a good idea to use sparse gradient in a neural network where all coordinates (i.e. hidden units) are exchangeable. This paper claims that "gradient values are skewed, as most are close to zero", but there is no references nor statistics. The only source of sparsity for a typical MT model are those for word vectors, but in that case, perhaps it would be better to develop algorithms that specifically make use of that type of sparsity. The quality degradation in methods like gradient dropping seems to suggest that some arbitrary cutoff for sparsity is not a good idea. For the results, the proposed method running on 4 nodes is compared to synchronous SGD, and only got 25% and 45% speedups on the 2 tests. Notably, they do not compare to asynchronous SGD with stale gradients. Such a test would help offer empirical support/refutation on the assumption that the sparse gradient is a good idea in the first place. I think these results are not good enough to justify the extra complications in the proposed method. The paper is well-written, however AllReduce seems undefined, and I assumed it aggregates sparse gradients from all nodes. I also a bit confused about the characteristics of ApplyOptimizer. In gradient dropping, ApplyOptimizer takes a sparse gradient, but in the proposed methods, it takes a dense gradient. I am a bit confused on how exactly the synchronous update is done, and I*d appreciate it if you can make your model of parallel updates more explicit.
assistant
 This paper proposes a 3 modes for combining local and global gradients to better use more computing nodes. The paper is well-written and well-motivated. I question why it is a good idea to use sparse gradient in a neural network where all coordinates (i.e. hidden units) are exchangeable. This paper claims that "gradient values are skewed, as most are close to zero", but there is no references nor statistics. The only source of sparsity for a typical MT model are those for word vectors, but in that case, perhaps it would be better to develop algorithms that specifically make use of that type of sparsity. The quality degradation in methods like gradient dropping seems to suggest that some arbitrary cutoff for sparsity is not a good idea. For the results, the proposed method running on 4 nodes is compared to synchronous SGD, and only got 25% and 45% speedups on the 2 tests. Notably, they do not compare to asynchronous SGD with stale gradients. Such a test would help offer empirical support/refutation on the assumption that the sparse gradient is a good idea in the first place. I think these results are not good enough to justify the extra complications in the proposed method. The paper is well-written, however AllReduce seems undefined, and I assumed it aggregates sparse gradients from all nodes. I also a bit confused about the characteristics of ApplyOptimizer. In gradient dropping, ApplyOptimizer takes a sparse gradient, but in the proposed methods, it takes a dense gradient. I am a bit confused on how exactly the synchronous update is done, and I*d appreciate it if you can make your model of parallel updates more explicit.