As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed on clusters to perform parallel model training.  describe two variants of data-parallel SGD that quantize and encode gradients to lessen communication costs. For the first variant, QSGD, they provide strong theoretical guarantees. For the second variant, which we call QSGDinf, they demonstrate impressive empirical gains for distributed training of large neural networks. Building on their work, we propose an alternative scheme for quantizing gradients and show that it yields stronger theoretical guarantees than exist for QSGD while matching the empirical performance of QSGDinf.
The authors propose a new scheme for quantizing gradients which are followed by the previous work QSGD [1]. They show that it yields stronger theoretical guarantees than QSGD while showing a great empirical performance. 
The main difference between their scheme NUQSGD and QSGD is that they use nonuniform quantization (0, 1/2^{s},  …., 2^{s-1}/2^{s}, 1) instead of uniform quantization (0, 1/s, …, (s-1)/s,1).  Intuitively, by the way, it could reduce quantization error and variance by better matching the properties of normalized vectors.
The results are in 2 parts. First comparing with QSGD, they establish stronger convergence guarantees for NUQSGD, under standard assumptions. They also establish theoretical results for the variance upper bound and expected communication cost of their scheme. Second, they show strong empirical performance on deep models and a large dataset, with an efficient implementation in PyTorch.
However, there are several issues and questions that if fixed or illustrated could be a great paper.
	1) The author claim NUQSGD achieves stronger convergence guarantees comparing with QSGD but hasn't illustrated the point in detail. On page 6, the paragraph named 'NUQSGD vs QSGD' mainly claims that variance upper bound controls the guarantee on the convergence speed by empirically showing the results of variance upper bound. It would be great to include more theoretical analysis which demonstrates the importance of variance upper bound for convergence speed guarantee.
	2) In the experimental part, they control the hyperparameters including batch-size, base learning rate, momentum, and weight decay to be identical with each method. This may cause tuning biases (the setting may favor one method but hurt others' performance).
	3) Although the paper mainly focuses on comparing with QSGD, there are several relative communication efficient training algorithms which I think are worth to compare empirically (at least one of them). For example:
		a. Deep Gradient compression [2]
		b. signSGD [3]
		c. TernGrad [4]
	4) In figure 4, the encoding cost is significantly increased from 4-bit to 8-bit NUQSGD. Any reason why it happens? Is it due to inefficient encoding implementation? 
I agree with the authors' point that it's worth to explore the interaction between NUQSGD with more complex reduction patterns like ring-based. Since the ring-based algorithm like all-reduce is more popular in practice nowadays, interacting with it would have a better practical meaning. 
[1] D. Alistarh, D. Grubic, J. Z. Li, R. Tomioka, and M. Vojnovic. QSGD: Communication-efﬁcient SGD via gradient quantization and encoding. In Proc. Advances in Neural Information Processing Systems (NIPS), 2017.
[2] Lin Y, Han S, Mao H, Wang Y, Dally WJ. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887. 2017 Dec 5.
[3] Bernstein J, Zhao J, Azizzadenesheli K, Anandkumar A. signSGD with majority vote is communication efficient and fault-tolerant. arXiv. 2018 Oct 11.
[4] W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. TernGrad: Ternary gradients to reduce communication in distributed deep learning. In Proc. Advances in Neural Information Processing Systems (NIPS), 2017.