AdaScale SGD: A Scale-Invariant Algorithm for Distributed Training
When using distributed training to speed up stochastic gradient descent, learning rates must adapt to new scales in order to maintain training effectiveness. Re-tuning these parameters is resource intensive, while fixed scaling rules often degrade model quality. We propose AdaScale SGD, a practical and principled algorithm that is approximately scale invariant. By continually adapting to the gradient’s variance, AdaScale often trains at a wide range of scales with nearly identical results. We describe this invariance formally through AdaScale’s convergence bounds. As the batch size increases, the bounds maintain final objective values, while smoothly transitioning away from linear speed-ups. In empirical comparisons, AdaScale trains well beyond the batch size limits of popular “linear learning rate scaling” rules. This includes large-scale training without model degradation for machine translation, image classification, object detection, and speech recognition tasks. The algorithm introduces negligible computational overhead and no tuning parameters, making AdaScale an attractive choice for large-scale training.
Authors use the PL condition to motivate a formula for learning rate selection. (they bound their loss by a quadratic with fixed curvature alpha, and use this to get learning rate). Their analysis very closely mirrors one presented in *Gradient Diversity* paper, it uses the same assumptions on the loss. IE compare A.1 in the paper to the B.5 in *Gradient Diversity* Gradient Diversity solves for batch size B after which linear learning rate scaling starts to break down, while this paper instead fixes B and solves for learning rate. Two results are comparable, if you take their learning rate formula in section 3.2 (need formula numbers) and solve for B which gives learning rate halfway between B=1 and B=inf, you get the same expression as *critical batch size* in equation 5 of gradient diversity paper. It*s not immediately obvious how to invert formula in Gradient Diversity paper to solve for learning rate, so I would consider their learning rate formula an interesting result. I also appreciate the breadth of experiments used for evaluation. The biggest issue I have with the paper is that I can*t tell if it*s better of worse than linear learning rate scaling from their experiment section. All of their experiments use more iterations for AS evaluation uses than for LSW evaluation. They demonstrate better training (and test) losses for AS, but because of extra iterations, I can*t tell that the improvement in training loss is due to number of iterations, or due to AS scaling.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
AdaScale SGD: A Scale-Invariant Algorithm for Distributed Training
When using distributed training to speed up stochastic gradient descent, learning rates must adapt to new scales in order to maintain training effectiveness. Re-tuning these parameters is resource intensive, while fixed scaling rules often degrade model quality. We propose AdaScale SGD, a practical and principled algorithm that is approximately scale invariant. By continually adapting to the gradient’s variance, AdaScale often trains at a wide range of scales with nearly identical results. We describe this invariance formally through AdaScale’s convergence bounds. As the batch size increases, the bounds maintain final objective values, while smoothly transitioning away from linear speed-ups. In empirical comparisons, AdaScale trains well beyond the batch size limits of popular “linear learning rate scaling” rules. This includes large-scale training without model degradation for machine translation, image classification, object detection, and speech recognition tasks. The algorithm introduces negligible computational overhead and no tuning parameters, making AdaScale an attractive choice for large-scale training.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Authors use the PL condition to motivate a formula for learning rate selection. (they bound their loss by a quadratic with fixed curvature alpha, and use this to get learning rate). Their analysis very closely mirrors one presented in *Gradient Diversity* paper, it uses the same assumptions on the loss. IE compare A.1 in the paper to the B.5 in *Gradient Diversity* Gradient Diversity solves for batch size B after which linear learning rate scaling starts to break down, while this paper instead fixes B and solves for learning rate. Two results are comparable, if you take their learning rate formula in section 3.2 (need formula numbers) and solve for B which gives learning rate halfway between B=1 and B=inf, you get the same expression as *critical batch size* in equation 5 of gradient diversity paper. It*s not immediately obvious how to invert formula in Gradient Diversity paper to solve for learning rate, so I would consider their learning rate formula an interesting result. I also appreciate the breadth of experiments used for evaluation. The biggest issue I have with the paper is that I can*t tell if it*s better of worse than linear learning rate scaling from their experiment section. All of their experiments use more iterations for AS evaluation uses than for LSW evaluation. They demonstrate better training (and test) losses for AS, but because of extra iterations, I can*t tell that the improvement in training loss is due to number of iterations, or due to AS scaling.
assistant
Authors use the PL condition to motivate a formula for learning rate selection. (they bound their loss by a quadratic with fixed curvature alpha, and use this to get learning rate). Their analysis very closely mirrors one presented in *Gradient Diversity* paper, it uses the same assumptions on the loss. IE compare A.1 in the paper to the B.5 in *Gradient Diversity* Gradient Diversity solves for batch size B after which linear learning rate scaling starts to break down, while this paper instead fixes B and solves for learning rate. Two results are comparable, if you take their learning rate formula in section 3.2 (need formula numbers) and solve for B which gives learning rate halfway between B=1 and B=inf, you get the same expression as *critical batch size* in equation 5 of gradient diversity paper. It*s not immediately obvious how to invert formula in Gradient Diversity paper to solve for learning rate, so I would consider their learning rate formula an interesting result. I also appreciate the breadth of experiments used for evaluation. The biggest issue I have with the paper is that I can*t tell if it*s better of worse than linear learning rate scaling from their experiment section. All of their experiments use more iterations for AS evaluation uses than for LSW evaluation. They demonstrate better training (and test) losses for AS, but because of extra iterations, I can*t tell that the improvement in training loss is due to number of iterations, or due to AS scaling.