Dimension-Free Bounds for Low-Precision Training
Low-precision training is a promising way of decreasing the time and energy cost of training machine learning models. Previous work has analyzed low-precision training algorithms, such as low-precision stochastic gradient descent, and derived theoretical bounds on their convergence rates. These bounds tend to depend on the dimension of the model in that the number of bits needed to achieve a particular error bound increases as increases. This is undesirable because a motivating application for low-precision training is large-scale models, such as deep learning, where can be huge. In this paper, we prove dimension-independent bounds for low-precision training algorithms that use fixed-point arithmetic, which lets us better understand what affects the convergence of these algorithms as parameters scale. Our methods also generalize naturally to let us prove new convergence bounds on low-precision training with other quantization schemes, such as low-precision floating-point computation and logarithmic quantization.
 This paper provides an in-depth study of the quantization error in low-precision training and gives consequent bounds on the low-precision SGD (LP-SGD) algorithm for convex problems under various generic quantization schemes. [pros] This paper provides a lot of novel insights in low-precision training, for example, a convergence bound in terms of the L1 gradient Lipschitzness can potentially be better than its L2 counterpart (which is experimentally verified on specially designed problems). I also liked the discussions about non-linear quantization, how they can give a convergence bound, and even how one could optimally choose the quantization parameters, or the number of {exponent, significance} bits in floating-point style quantization, in order to minimize the convergence bound. The restriction to convex problems is fine for me, because otherwise essentially there is not a lot interesting things to say (for quantized problems it does not make sense to talk about “stationary points” as points are isolated.) This paper is very well-written and I enjoyed reading it. The authors are very precise and unpretentious about their contributions and have insightful discussions throughout the entire paper. [cons] My main concern is that of the significance: while it is certainly of interest to minimize the quantization error with a given number of bits as the budget (and that’s very important for the deployment side), it is unclear if such a *loss-unaware* theory really helps explain the success of low-precision training in practice. An alternative belief is that the success comes in a *loss-aware* fashion, that is, efficient feature extraction and supervised learning in general can be achieved by low-precision models, but the good quantization scheme comes in a way that depends on the particular problem which varies case by case. Admittedly, this is a more vague statement which may be harder to analyze or empirically study, but it sounds to me more reasonable for explaining successful low-precision training than the fact that we have certain tight bounds for quantized convex optimization. [a technical question] In the discussions following Theorem 2, the authors claim that the quantization parameters can be optimized to push the dependence on .sigma_1 into a log term -- this sounds a bit magical to me, because there is the assumption that .zeta < 1/.kappa, which restricts setting .zeta to be too large (and thus restricts the “acceleration” of strides from being too large) . I imagine the optimal bound only holds when the optimal choice of .zeta is indeed blow 1/.kappa?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Dimension-Free Bounds for Low-Precision Training
Low-precision training is a promising way of decreasing the time and energy cost of training machine learning models. Previous work has analyzed low-precision training algorithms, such as low-precision stochastic gradient descent, and derived theoretical bounds on their convergence rates. These bounds tend to depend on the dimension of the model in that the number of bits needed to achieve a particular error bound increases as increases. This is undesirable because a motivating application for low-precision training is large-scale models, such as deep learning, where can be huge. In this paper, we prove dimension-independent bounds for low-precision training algorithms that use fixed-point arithmetic, which lets us better understand what affects the convergence of these algorithms as parameters scale. Our methods also generalize naturally to let us prove new convergence bounds on low-precision training with other quantization schemes, such as low-precision floating-point computation and logarithmic quantization.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper provides an in-depth study of the quantization error in low-precision training and gives consequent bounds on the low-precision SGD (LP-SGD) algorithm for convex problems under various generic quantization schemes. [pros] This paper provides a lot of novel insights in low-precision training, for example, a convergence bound in terms of the L1 gradient Lipschitzness can potentially be better than its L2 counterpart (which is experimentally verified on specially designed problems). I also liked the discussions about non-linear quantization, how they can give a convergence bound, and even how one could optimally choose the quantization parameters, or the number of {exponent, significance} bits in floating-point style quantization, in order to minimize the convergence bound. The restriction to convex problems is fine for me, because otherwise essentially there is not a lot interesting things to say (for quantized problems it does not make sense to talk about “stationary points” as points are isolated.) This paper is very well-written and I enjoyed reading it. The authors are very precise and unpretentious about their contributions and have insightful discussions throughout the entire paper. [cons] My main concern is that of the significance: while it is certainly of interest to minimize the quantization error with a given number of bits as the budget (and that’s very important for the deployment side), it is unclear if such a *loss-unaware* theory really helps explain the success of low-precision training in practice. An alternative belief is that the success comes in a *loss-aware* fashion, that is, efficient feature extraction and supervised learning in general can be achieved by low-precision models, but the good quantization scheme comes in a way that depends on the particular problem which varies case by case. Admittedly, this is a more vague statement which may be harder to analyze or empirically study, but it sounds to me more reasonable for explaining successful low-precision training than the fact that we have certain tight bounds for quantized convex optimization. [a technical question] In the discussions following Theorem 2, the authors claim that the quantization parameters can be optimized to push the dependence on .sigma_1 into a log term -- this sounds a bit magical to me, because there is the assumption that .zeta < 1/.kappa, which restricts setting .zeta to be too large (and thus restricts the “acceleration” of strides from being too large) . I imagine the optimal bound only holds when the optimal choice of .zeta is indeed blow 1/.kappa?
assistant
 This paper provides an in-depth study of the quantization error in low-precision training and gives consequent bounds on the low-precision SGD (LP-SGD) algorithm for convex problems under various generic quantization schemes. [pros] This paper provides a lot of novel insights in low-precision training, for example, a convergence bound in terms of the L1 gradient Lipschitzness can potentially be better than its L2 counterpart (which is experimentally verified on specially designed problems). I also liked the discussions about non-linear quantization, how they can give a convergence bound, and even how one could optimally choose the quantization parameters, or the number of {exponent, significance} bits in floating-point style quantization, in order to minimize the convergence bound. The restriction to convex problems is fine for me, because otherwise essentially there is not a lot interesting things to say (for quantized problems it does not make sense to talk about “stationary points” as points are isolated.) This paper is very well-written and I enjoyed reading it. The authors are very precise and unpretentious about their contributions and have insightful discussions throughout the entire paper. [cons] My main concern is that of the significance: while it is certainly of interest to minimize the quantization error with a given number of bits as the budget (and that’s very important for the deployment side), it is unclear if such a *loss-unaware* theory really helps explain the success of low-precision training in practice. An alternative belief is that the success comes in a *loss-aware* fashion, that is, efficient feature extraction and supervised learning in general can be achieved by low-precision models, but the good quantization scheme comes in a way that depends on the particular problem which varies case by case. Admittedly, this is a more vague statement which may be harder to analyze or empirically study, but it sounds to me more reasonable for explaining successful low-precision training than the fact that we have certain tight bounds for quantized convex optimization. [a technical question] In the discussions following Theorem 2, the authors claim that the quantization parameters can be optimized to push the dependence on .sigma_1 into a log term -- this sounds a bit magical to me, because there is the assumption that .zeta < 1/.kappa, which restricts setting .zeta to be too large (and thus restricts the “acceleration” of strides from being too large) . I imagine the optimal bound only holds when the optimal choice of .zeta is indeed blow 1/.kappa?