Currently, progressively larger deep neural networks are trained on ever growing data corpora. In result, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes or prohibitive communication cost in general. To mitigate this problem we propose Sparse Binary Compression (SBC), a compression framework that allows for a drastic reduction of communication cost for distributed training. SBC combines existing techniques of communication delay and gradient sparsification with a novel binarization method and optimal weight update encoding to push compression gains to new limits. By doing so, our method also allows us to smoothly trade-off gradient sparsity and temporal sparsity to adapt to the requirements of the learning task. Our experiments show, that SBC can reduce the upstream communication on a variety of convolutional and recurrent neural network architectures by more than four orders of magnitude without significantly harming the convergence speed in terms of forward-backward passes. For instance, we can train ResNet50 on ImageNet in the same number of iterations to the baseline accuracy, using ×3531 less bits or train it to a 1% lower accuracy using ×37208 less bits. In the latter case, the total upstream communication required is cut from 125 terabytes to 3.35 gigabytes for every participating client. Our method also achieves state-of-the-art compression rates in a Federated Learning setting with 400 clients.
The paper once again looks at the problem of reducing the communication requirement for implementing the distributed optimization techniques, in particular, SGD. This problem has been looked at from multiple angles by many authors. And although there are many unanswered questions in this area, I do not see the authors providing any compelling contribution to answering those questions. A big chunk of the paper is devoted to expressing some shallow theorems, which in some cases I do not even see their importance or connection to the main point of the paper; see my comments below. In terms of the techniques for reducing the communication burden, the authors seem to just put all the other approaches together with minimal novelty.
More detailed comments:
- I do not really understand what the authors mean by noise damping. I would appreciate it if they could clarify that point. This seems to be a very important point as the model they propose for the noise in the process is basically based on this notion. It is a great failure on the authors' part that such a crucial notion in their paper is not clearly described. 
- The model that is proposed for noise is too strong and too simplistic. Do you guys have any evidence to back this up?
- Theorem 2.1 is not a theorem. The result is super shallow and relatively trivial. 
- In corollary 2.1 it seems that no matter what the randomness in the system is, the algorithm is going to converge to the same solution. This is not true even for the non-strongly convex objectives, let alone the non-convex problems where there are so many stationary solutions and whatnot.
- With regards to Fig 3 (and other related figures in the appendix) and the discussion on the multiplicative nature of compression: The figure does not seem to suggest multiplicative nature in all the regimes. It seems to hold in high compression/ low-frequency communication regime. But on the other side of the spectrum, it does not seem to hold very strongly.
- The residual accumulation only applies when all the nodes update in all the iterations. I do not believe this would generalize to the federated learning, where nodes do not participate in all the updates. I do not know if the authors have noted this point in their federated learning experiments.
- Theorem 3.1 is very poorly stated. Other that than it is shallow and in my opinion irrelevant. What is the argument in favor of the authors' thought that could be built based on the result of Theorem 3.1?
- One major point that is missing in the experiments (and probably in the experiments in other papers on the same topic) is to see how much do all these compressions affect the speed of learning in different scenarios in realistic scenarios? Note that in realistic scenarios many things other than communication could affect the convergence time.