Learning with Random Learning Rates.
Hyperparameter tuning is a bothersome step in the training of deep learning mod- els. One of the most sensitive hyperparameters is the learning rate of the gradient descent. We present the All Learning Rates At Once (Alrao) optimization method for neural networks: each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude. This comes at practically no computational cost. Perhaps surprisingly, stochastic gra- dient descent (SGD) with Alrao performs close to SGD with an optimally tuned learning rate, for various architectures and problems. Alrao could save time when testing deep learning models: a range of models could be quickly assessed with Alrao, and the most promising models could then be trained more extensively. This text comes with a PyTorch implementation of the method, which can be plugged on an existing PyTorch model.
 In this paper the authors propose a method called “All learning rates at once” (Alrao) which aims to save the time needed to tune learning rate for DNN models testing. The method sets individual learning rate to each feature in each layer of a network using the values sampled from truncated log-uniform distribution. The only cost of the method is the creation of several branches of the classifier layer. Each of the branches is trained with a predefined learning rate value, and the final predictions are obtained by model averaging. In the presented experiments Alrao demonstrates performance comparable to SGD with optimal learning rate and more stable results compared to Adam. The authors indicate limitations of Alrao caused by the overhead in the final layer which complicates the application of the method for models with large classifier layer. Overall, the paper is written clearly and organized well. However, Equation (2) needs to be corrected. The denominator in the normalizing constant of log-uniform distribution should be .log.eta_{max} - .log.eta_{min}. My main concern is related to the experimental evaluation of the method. I find the experimental evidence for the effectiveness of Alrao insufficient. As the authors propose to employ the method to quickly evaluate models and select best models to further training it would be beneficial to have more results in order to ensure that the method is reliable in this setting. Other demonstrations which would show possibly that the method enhances performance of architecture search methods may emphasize significance of the proposed method. Also, more experiments comparing Alrao against sampling learning rates per weight are needed. Given the current results, it is still unclear whether the proposed method performs better. Finally, I recommend to include comments explaining how much more time is needed in practice to train model with Alrao compared to SGD training.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning with Random Learning Rates.
Hyperparameter tuning is a bothersome step in the training of deep learning mod- els. One of the most sensitive hyperparameters is the learning rate of the gradient descent. We present the All Learning Rates At Once (Alrao) optimization method for neural networks: each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude. This comes at practically no computational cost. Perhaps surprisingly, stochastic gra- dient descent (SGD) with Alrao performs close to SGD with an optimally tuned learning rate, for various architectures and problems. Alrao could save time when testing deep learning models: a range of models could be quickly assessed with Alrao, and the most promising models could then be trained more extensively. This text comes with a PyTorch implementation of the method, which can be plugged on an existing PyTorch model.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 In this paper the authors propose a method called “All learning rates at once” (Alrao) which aims to save the time needed to tune learning rate for DNN models testing. The method sets individual learning rate to each feature in each layer of a network using the values sampled from truncated log-uniform distribution. The only cost of the method is the creation of several branches of the classifier layer. Each of the branches is trained with a predefined learning rate value, and the final predictions are obtained by model averaging. In the presented experiments Alrao demonstrates performance comparable to SGD with optimal learning rate and more stable results compared to Adam. The authors indicate limitations of Alrao caused by the overhead in the final layer which complicates the application of the method for models with large classifier layer. Overall, the paper is written clearly and organized well. However, Equation (2) needs to be corrected. The denominator in the normalizing constant of log-uniform distribution should be .log.eta_{max} - .log.eta_{min}. My main concern is related to the experimental evaluation of the method. I find the experimental evidence for the effectiveness of Alrao insufficient. As the authors propose to employ the method to quickly evaluate models and select best models to further training it would be beneficial to have more results in order to ensure that the method is reliable in this setting. Other demonstrations which would show possibly that the method enhances performance of architecture search methods may emphasize significance of the proposed method. Also, more experiments comparing Alrao against sampling learning rates per weight are needed. Given the current results, it is still unclear whether the proposed method performs better. Finally, I recommend to include comments explaining how much more time is needed in practice to train model with Alrao compared to SGD training.
assistant
 In this paper the authors propose a method called “All learning rates at once” (Alrao) which aims to save the time needed to tune learning rate for DNN models testing. The method sets individual learning rate to each feature in each layer of a network using the values sampled from truncated log-uniform distribution. The only cost of the method is the creation of several branches of the classifier layer. Each of the branches is trained with a predefined learning rate value, and the final predictions are obtained by model averaging. In the presented experiments Alrao demonstrates performance comparable to SGD with optimal learning rate and more stable results compared to Adam. The authors indicate limitations of Alrao caused by the overhead in the final layer which complicates the application of the method for models with large classifier layer. Overall, the paper is written clearly and organized well. However, Equation (2) needs to be corrected. The denominator in the normalizing constant of log-uniform distribution should be .log.eta_{max} - .log.eta_{min}. My main concern is related to the experimental evaluation of the method. I find the experimental evidence for the effectiveness of Alrao insufficient. As the authors propose to employ the method to quickly evaluate models and select best models to further training it would be beneficial to have more results in order to ensure that the method is reliable in this setting. Other demonstrations which would show possibly that the method enhances performance of architecture search methods may emphasize significance of the proposed method. Also, more experiments comparing Alrao against sampling learning rates per weight are needed. Given the current results, it is still unclear whether the proposed method performs better. Finally, I recommend to include comments explaining how much more time is needed in practice to train model with Alrao compared to SGD training.