Answers.Body: <p>Setting too high a learning rate will extend the time to get a good result.</p>
  
  <p>In my opinion, it is better to set not too big a learning rate but to use learning with momentum. When the learning starts to be ineffective, increase the learning rate to find a better optimal result. It seems to me that this allows to get very good results in a faster time than setting a large value of the learning rate from the very beginning. At least in cases I've dealt with.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T08:27:30.983Z
Answers.LastActivityDate: 2020-06-16T08:27:30.983Z
Answers.UserDisplayName: Dominiksr
Body: <p>It is known that if <span class="math-container">$\alpha$</span> is set to high, then the cost function of the model may not converge. <a href="https://i.sstatic.net/KszUC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KszUC.png" alt="enter image description here"></a> 
  However, would a decaying of the learning rate provide some "tuning" of the <span class="math-container">$\alpha$</span> value during training ? In the sense that if you set a high learning rate but you also have some form of learning rate decay, then eventually <span class="math-container">$\alpha$</span> value would fall within the "just right" and "too low" range eventually. Is it better to then set an initial learning rate that is more "flexible" in the higher ranges rather than a learning rate that is too low ?</p>
  
Tags: <machine-learning><deep-learning><learning-rate>
Title: Is it harmful to set the learning rate of training a model to be too high if there is some decay function for the learning rate?
UserDisplayName: calveeen
ThreadText: # Is it harmful to set the learning rate of training a model to be too high if there is some decay function for the learning rate?
  
  (Asked by: username_0 on 2020-06-16)
  
  It is known that if $\alpha$ is set to high, then the cost function of the model may not converge. [![enter image description here](https://i.sstatic.net/KszUC.png)](https://i.sstatic.net/KszUC.png)
  However, would a decaying of the learning rate provide some "tuning" of the $\alpha$ value during training ? In the sense that if you set a high learning rate but you also have some form of learning rate decay, then eventually $\alpha$ value would fall within the "just right" and "too low" range eventually. Is it better to then set an initial learning rate that is more "flexible" in the higher ranges rather than a learning rate that is too low ?
  
  **Tags:** machine-learning, deep-learning, learning-rate
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-16. Score: 1
  
  Setting too high a learning rate will extend the time to get a good result.
  
  In my opinion, it is better to set not too big a learning rate but to use learning with momentum. When the learning starts to be ineffective, increase the learning rate to find a better optimal result. It seems to me that this allows to get very good results in a faster time than setting a large value of the learning rate from the very beginning. At least in cases I've dealt with.
  
  
  ---
  
  