Low-precision training is a promising way of decreasing the time and energy cost of training machine learning models. Previous work has analyzed low-precision training algorithms, such as low-precision stochastic gradient descent, and derived theoretical bounds on their convergence rates. These bounds tend to depend on the dimension of the model d in that the number of bits needed to achieve a particular error bound increases as d increases. This is undesirable because a motivating application for low-precision training is large-scale models, such as deep learning, where d can be huge. In this paper, we prove dimension-independent bounds for low-precision training algorithms that use fixed-point arithmetic, which lets us better understand what affects the convergence of these algorithms as parameters scale. Our methods also generalize naturally to let us prove new convergence bounds on low-precision training with other quantization schemes, such as lowprecision floating-point computation and logarithmic quantization.
The paper considers the problem of low precision stochastic gradient descent. Specifically, they study updates of the form x_{t + 1} = Q (x_t - alpha * g_t), where g_t is a stochastic gradient, and Q is a quantization function. The goal is to produce quantization functions that simultaneously increase the convergence rate as little as possible, while also requiring few bits to represent. This is motivated by the desire to perform SGD on low precision machines.
The paper shows that under a set of somewhat nonstandard assumptions, previously studied quantization functions as well as other low precision training algorithms are able to match the performance of non-quantized SGD, specifically, losing no additional dimension factors. Previous papers, to the best of my knowledge, did not prove such bounds, except under strong sparsity conditions on the gradients. I did not check their proofs line-by-line however they seem correct at a high level.
I think the main discussion about the paper should be about the assumptions made in the analysis.  As the authors point out, besides the standard smoothness and variance conditions on the functions, some additional assumptions about the function must be made for such dimension independent bounds to hold. Therefore I believe the main contribution of this paper is to identify a set of conditions under which these sorts of bounds can be proven. 
Specifically, I wish to highlight Assumption 2, namely, that the ell_1 smoothness of the gradients can be controlled by the ell_2 difference between the points, and Assumption 4, which states that each individual function (not just the overall average), has gradients with bounded ell_2 and ell_1 norm at the optimal point. I believe that Assumption 2 is a natural condition to consider, although it does already pose some limitations on the applicability of the analysis. I am less sold on Assumption 4; it is unclear how natural this bound is, or how necessary it is to the analysis. 
The main pros of these assumptions are that they are quite natural conditions from a theoretical perspective (at least, Assumption 2 is). For instance, as the authors point out, this gives very good results for sparse updates. Given these assumptions, I don’t think it’s surprising that such bounds can be proven, although it appears somewhat nontrivial.  The main downside is that these assumptions are somewhat limiting, and don’t seem to be able to explain why quantization works well for neural network training. If I understand Figure 4b correctly, the bound is quite loose for even logistic regression on MNIST. However, despite this, I think formalizing these assumptions is a solid contribution.
The paper is generally well-written (at least the first 8 pages) but the supplementary material has various minor issues.
Smaller comments / questions:
- While I understand it is somewhat standard in optimization, I find the term “dimension-independent“ here somewhat misleading, as in many cases in practice (for instance, vanilla SGD on deep nets), the parameters L and kappa (not to mention L_1 and kappa_1) will grow with the dimension.
- Do these assumptions hold with good constants for training neural networks? I would be somewhat surprised if they did.
- Can one get dimension independent bounds for quantized gradients under these assumptions?
- The proofs after page 22 are all italicized.
- The brackets around expectations are too small in comparison to the rest of the expressions.