Answers.Body: <p>Yes, the optimal learning rate will differ for every change you make in the network. In fact finding the optimal learning rate is very computationally expensive, so you will normally only get a rough guess anyway.</p>
  <p>The learning rate is used to traverse an N dimensional loss landscape that changes drastically with even the smallest differences. If you add one more training data point, the optimal learning rate will change. If you add one more neuron, it will change.</p>
  <p>What you tune first is up to you, but once you have settled on an architecture, you should tune the learning rate first because it will have the largest effect on other hyper-parameters (normally, not always). For example tuning the number of epochs to train for first will be worthless if you start changing the learning rate because the network will learn at a different speed. That's why the learning rate is usually tuned first (after the architecture).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-29T12:46:18.490Z
Answers.Text: so for example when I do grid search of optimal hyperparameters search, for every set of values I should additionally tune the learning rate?
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-30T12:46:36.703Z
Answers.Text: Your explanation of why the optimal learning rate would change seems reasonable to me, but it may be a good idea to provide a link to some research paper or maybe an animation that shows some evidence for this.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-29T02:11:35.647Z
Answers.LastActivityDate: 2021-07-29T02:11:35.647Z
Answers.UserDisplayName: Recessive
Body: <p>In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in <a href="https://www.youtube.com/watch?v=h7iBpEHGVNc" rel="nofollow noreferrer">this lecture</a>
  (minute 59:55), the lecturer says that the learning rate is the first hyper-parameter that he tunes.</p>
  <p>However, is it possible that the optimal learning rate is different for different architectures (for example, a different number of layers and neurons)? Or maybe the LR is architecture-independent and it depends only on the characteristics of the particular dataset we train our model on?</p>
  <p>Moreover, should the LR be searched in the same process (e.g. grid-search) as the other hyper-parameters?</p>
  
Tags: <neural-networks><deep-learning><training><hyperparameter-optimization><learning-rate>
Title: Can the optimal learning rate differ for different architectures?
UserDisplayName: GKozinski
ThreadText: # Can the optimal learning rate differ for different architectures?
  
  (Asked by: username_0 on 2021-07-28)
  
  In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in [this lecture](https://www.youtube.com/watch?v=h7iBpEHGVNc)
  (minute 59:55), the lecturer says that the learning rate is the first hyper-parameter that he tunes.
  
  However, is it possible that the optimal learning rate is different for different architectures (for example, a different number of layers and neurons)? Or maybe the LR is architecture-independent and it depends only on the characteristics of the particular dataset we train our model on?
  
  Moreover, should the LR be searched in the same process (e.g. grid-search) as the other hyper-parameters?
  
  **Tags:** neural-networks, deep-learning, training, hyperparameter-optimization, learning-rate
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-29. Score: 1
  
  Yes, the optimal learning rate will differ for every change you make in the network. In fact finding the optimal learning rate is very computationally expensive, so you will normally only get a rough guess anyway.
  
  The learning rate is used to traverse an N dimensional loss landscape that changes drastically with even the smallest differences. If you add one more training data point, the optimal learning rate will change. If you add one more neuron, it will change.
  
  What you tune first is up to you, but once you have settled on an architecture, you should tune the learning rate first because it will have the largest effect on other hyper-parameters (normally, not always). For example tuning the number of epochs to train for first will be worthless if you start changing the learning rate because the network will learn at a different speed. That's why the learning rate is usually tuned first (after the architecture).
  
  > **Comments:**
  > **username_0**: so for example when I do grid search of optimal hyperparameters search, for every set of values I should additionally tune the learning rate?  
  > **username_2**: Your explanation of why the optimal learning rate would change seems reasonable to me, but it may be a good idea to provide a link to some research paper or maybe an animation that shows some evidence for this.  
  
  ---
  
  