Answers.Body: <p>The 2015 article <a href="https://arxiv.org/abs/1506.01186" rel="nofollow noreferrer">Cyclical Learning Rates for Training Neural Networks</a> by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate.</p>
  <p>The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed learning rate value. For this to work, you still need to select good lower and upper bounds, and Smith suggests training the model for a few epochs while increasing the learning rate between a large range of values. At first, the learning rate will be too small to make any progress at all. As the learning rate increases, eventually, the loss will begin to decrease, but, at some point, the learning rate will get too large, and the loss will stop decreasing and even begin increasing.  Your ideal range consists of the learning rate values where the loss was decreasing steeply.  After finding your range, you can reset the weights and biases on your model and restart training using whatever learning rate schedule you plan to use for training.</p>
  <p>Here is a concrete example from one of my experiments:</p>
  <p><a href="https://i.sstatic.net/m23Vt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m23Vt.png" alt="Learning Rate Finder Loss Plot" /></a></p>
  <p>In this case, I start my learning rate search at 1e-09 and plan to end with a learning rate of 0.99 (although I am actually able to stop sooner than that).  Your experiment may require different search bounds, but you could always start with that and adjust things as needed.  At first, the loss plot is flat, and then it begins to decrease, but is too gradual.  At the first red line, loss starts to decrease sharply, and once it reaches the second red line, the plot has begun to level off, so I can end my search.  For this particular experiment, my ideal learning rate range had a minimum of 4.01e-4 and a maximum of 2.58e-2.</p>
  <p>For more information, I suggest reading this <a href="https://www.pyimagesearch.com/2019/08/05/keras-learning-rate-finder/" rel="nofollow noreferrer">Keras Learning Rate Finder</a> post, which contains more information on how the process works and a tutorial for how to program it using Keras and Tensorflow.</p>
  <hr/>
  <p>Edit: I recently learned that in newer versions of Tensorflow / Keras, the <code>on_train_batch_end</code> callback now returns an aggregated average loss instead of the raw loss for the given batch, which gives poor results for the learning rate finder.  See this <a href="https://github.com/keras-team/keras/issues/17167" rel="nofollow noreferrer">Github Issue</a> for more information and the workaround I am currently using.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-22T15:53:16.247Z
Answers.LastActivityDate: 2022-11-28T15:23:32.857Z
Answers.LastEditDate: 2022-11-28T15:23:32.857Z
Answers.UserDisplayName: Leland Hepworth
Answers.Body: <p>The visualisation can be found in <a href="https://www.researchgate.net/publication/3907199_The_need_for_small_learning_rates_on_large_problems" rel="nofollow noreferrer"><em>The need for small learning rates on large problems</em></a>. This paper by D. Randall Wilson and Tony R. Martinez from 2001 investigates the role of learning rates in gradient descent algorithms.</p>
  <p>In general, different algorithms assign different meaning to the same word 'learning rate'. For example, the learning rate in a gradient descent algorithm is not comparable to the learning rate in a tabular reinforcement learning algorithm such as Q-learning.
  This means that at a particular 'best' does not exist considering the different concepts denoted with the term `learning rate' in different algorithms.</p>
  <p>Additionally, the learning rate is typically considered a part of the learning algorithm. The no free lunch theorem of machine learning tells us that no particular learning algorithm performs best across tasks. Because the learning rate is part of the solution, no particular learning rate is 'best' across tasks either.</p>
  <p>In practice, you should set the learning rate sufficiently low to not 'overshoot' the optimal solution which will be evidenced by oscillations in the error (no convergence). But you also should also set it high enough to obtain reasonable performance given the amount of available training time.</p>
  <p>What learning rate gives you the right trade-off typically requires a combination of domain knowledge and experimentation on the training set.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-22T14:48:56.520Z
Answers.Text: "In general, different algorithms assign different meaning to the same word 'learning rate'." This is wrong for your example. The meaning of "learning rate" in Q-learning is very similar to the meaning of "learning rate" in GD (if not the same thing): just look at the pseudocode!! In fact, in deep Q-learning, they are exactly the same thing. Maybe you need to clarify that the ideal range for the LR in one case may be different than in another case.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-22T14:51:20.507Z
Answers.Text: Moreover, note that now the question focuses on neural networks trained with gradient descent.  Moreover, this answer doesn't not answer the question at all. The question is "is there an ideal **range** of learning rate which always gives a good result almost in all problems?". The question is **not** "How do I set my ideal learning rate?"
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-22T09:46:19.530Z
Answers.LastActivityDate: 2021-02-22T13:18:52.420Z
Answers.LastEditDate: 2021-02-22T13:18:52.420Z
Answers.UserDisplayName: Floris den Hengst
Body: <p>I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph from the paper: <a href="https://www.researchgate.net/publication/3907199_The_need_for_small_learning_rates_on_large_problems" rel="noreferrer"><em>The need for small learning rates on large problems</em></a></p>
  <p><a href="https://i.sstatic.net/o7z4Y.jpg" rel="noreferrer"><img src="https://i.sstatic.net/o7z4Y.jpg" alt="enter image description here" /></a></p>
  <p>In the context of neural networks trained with gradient descent, is there a range of the learning rate, which should be used to reduce the training time and get a good performance in almost all problems?</p>
  
Tags: <neural-networks><deep-learning><reference-request><gradient-descent><learning-rate>
Title: Is there an ideal range of learning rate which always gives a good result almost in all problems?
UserDisplayName: Sadaf Shafi
ThreadText: # Is there an ideal range of learning rate which always gives a good result almost in all problems?
  
  (Asked by: username_0 on 2021-02-22)
  
  I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph from the paper: [*The need for small learning rates on large problems*](https://www.researchgate.net/publication/3907199_The_need_for_small_learning_rates_on_large_problems)
  
  [![enter image description here](https://i.sstatic.net/o7z4Y.jpg)](https://i.sstatic.net/o7z4Y.jpg)
  
  In the context of neural networks trained with gradient descent, is there a range of the learning rate, which should be used to reduce the training time and get a good performance in almost all problems?
  
  **Tags:** neural-networks, deep-learning, reference-request, gradient-descent, learning-rate
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-02-22. Score: 7
  
  The 2015 article [Cyclical Learning Rates for Training Neural Networks](https://arxiv.org/abs/1506.01186) by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate.
  
  The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed learning rate value. For this to work, you still need to select good lower and upper bounds, and Smith suggests training the model for a few epochs while increasing the learning rate between a large range of values. At first, the learning rate will be too small to make any progress at all. As the learning rate increases, eventually, the loss will begin to decrease, but, at some point, the learning rate will get too large, and the loss will stop decreasing and even begin increasing. Your ideal range consists of the learning rate values where the loss was decreasing steeply. After finding your range, you can reset the weights and biases on your model and restart training using whatever learning rate schedule you plan to use for training.
  
  Here is a concrete example from one of my experiments:
  
  [![Learning Rate Finder Loss Plot](https://i.sstatic.net/m23Vt.png)](https://i.sstatic.net/m23Vt.png)
  
  In this case, I start my learning rate search at 1e-09 and plan to end with a learning rate of 0.99 (although I am actually able to stop sooner than that). Your experiment may require different search bounds, but you could always start with that and adjust things as needed. At first, the loss plot is flat, and then it begins to decrease, but is too gradual. At the first red line, loss starts to decrease sharply, and once it reaches the second red line, the plot has begun to level off, so I can end my search. For this particular experiment, my ideal learning rate range had a minimum of 4.01e-4 and a maximum of 2.58e-2.
  
  For more information, I suggest reading this [Keras Learning Rate Finder](https://www.pyimagesearch.com/2019/08/05/keras-learning-rate-finder/) post, which contains more information on how the process works and a tutorial for how to program it using Keras and Tensorflow.
  
  ---
  
  Edit: I recently learned that in newer versions of Tensorflow / Keras, the `on_train_batch_end` callback now returns an aggregated average loss instead of the raw loss for the given batch, which gives poor results for the learning rate finder. See this [Github Issue](https://github.com/keras-team/keras/issues/17167) for more information and the workaround I am currently using.
  
  
  ---
  
  ### Answer by username_2 on 2021-02-22. Score: 4
  
  The visualisation can be found in [*The need for small learning rates on large problems*](https://www.researchgate.net/publication/3907199_The_need_for_small_learning_rates_on_large_problems). This paper by D. Randall Wilson and Tony R. Martinez from 2001 investigates the role of learning rates in gradient descent algorithms.
  
  In general, different algorithms assign different meaning to the same word 'learning rate'. For example, the learning rate in a gradient descent algorithm is not comparable to the learning rate in a tabular reinforcement learning algorithm such as Q-learning.
  This means that at a particular 'best' does not exist considering the different concepts denoted with the term `learning rate' in different algorithms.
  
  Additionally, the learning rate is typically considered a part of the learning algorithm. The no free lunch theorem of machine learning tells us that no particular learning algorithm performs best across tasks. Because the learning rate is part of the solution, no particular learning rate is 'best' across tasks either.
  
  In practice, you should set the learning rate sufficiently low to not 'overshoot' the optimal solution which will be evidenced by oscillations in the error (no convergence). But you also should also set it high enough to obtain reasonable performance given the amount of available training time.
  
  What learning rate gives you the right trade-off typically requires a combination of domain knowledge and experimentation on the training set.
  
  > **Comments:**
  > **username_3**: "In general, different algorithms assign different meaning to the same word 'learning rate'." This is wrong for your example. The meaning of "learning rate" in Q-learning is very similar to the meaning of "learning rate" in GD (if not the same thing): just look at the pseudocode!! In fact, in deep Q-learning, they are exactly the same thing. Maybe you need to clarify that the ideal range for the LR in one case may be different than in another case.  
  > **username_3**: Moreover, note that now the question focuses on neural networks trained with gradient descent.  Moreover, this answer doesn't not answer the question at all. The question is "is there an ideal **range** of learning rate which always gives a good result almost in all problems?". The question is **not** "How do I set my ideal learning rate?"  
  
  ---
  
  