Answers.Body: <p>Apart from the already mentioned methods there is one which I came across recently is the Cyclic Learning Rates. </p>
  
  <p>Mainstream methods usually reduce the learning rates monotonically. In most of the cases they work well. Yet they do not do well with handling saddle points and local minima. CLR method claims to solve the problem by effectively handling the case of saddle points. </p>
  
  <p>Ref: <a href="https://arxiv.org/abs/1506.01186" rel="nofollow noreferrer">https://arxiv.org/abs/1506.01186</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-09T14:46:35.047Z
Answers.LastActivityDate: 2017-09-09T14:46:35.047Z
Answers.UserDisplayName: Yeshwanth Venkatesha
Answers.Body: <p>While adjusting the learning rate during the training is certainly interesting, so is finding of a good initial learning rate, too.</p>
  
  <p><a href="https://arxiv.org/abs/1506.01186" rel="nofollow noreferrer">Cyclical Learning Rates for Training Neural Networks</a>, <a href="https://docs.fast.ai/basic_train.html#lr_find" rel="nofollow noreferrer">fast.ai (pytorch) implementation</a>.</p>
  
  <p>Here's a good practitioner's <a href="https://github.com/fastai/fastai/blob/master/courses/dl2/training_phase.ipynb" rel="nofollow noreferrer">overview of learning rate schedules</a> (python, but rather readable and with nice plots).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-22T08:54:00.887Z
Answers.LastActivityDate: 2019-02-22T08:54:00.887Z
Answers.UserDisplayName: Xpector
Answers.Body: <p>Adjusting for learning rate is a common scenario in machine learning. There is rich literature about it, countless papers. The most common implementation:</p>
  
  <ul>
  <li>AdaGrad</li>
  <li>RMSProp</li>
  <li>Adam</li>
  </ul>
  
  <p>There are many more variants. You'll need to do some research. Please take a look at:</p>
  
  <blockquote>
    <p><a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Stochastic_gradient_descent</a></p>
  </blockquote>
  
  <p><a href="https://i.sstatic.net/NhrR7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NhrR7.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-01T01:38:33.693Z
Answers.LastActivityDate: 2017-08-01T01:38:33.693Z
Answers.UserDisplayName: SmallChess
Body: <p>I'm learning Neural Networks, and everything works as planned but, like humans do, adjusting themselves to learn more efficiently, I'm trying to understand conceptually how one might implement an auto adjusting learning rate for a Neural Network.</p>
  
  <p>I have tried to make it based on error, something like how bigger is error learning rate is getting bigger as well. <sub><em>[Could use some clarification here--not entirely sure what you're saying.  If can clarify, I'm happy to clean up the English. -DukeZhou]</em></sub></p>
  
  <p>*If you want give me an example give it on a C based language or math because I don't have experience with Python or Pascal. </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-07-31T01:10:56.703Z
Comments.Text: Google "adaptive learning rate"
Comments.UserDisplayName: SmallChess
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-07-31T01:11:05.050Z
Comments.Text: http://cs231n.github.io/neural-networks-3/#ada
Comments.UserDisplayName: SmallChess
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-07-31T10:49:55.763Z
Comments.Text: thx, you can make this comment an answer to mark it as best answer?
Comments.UserDisplayName: Laceanu George
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-07-31T18:50:18.843Z
Comments.Text: @SmallChess very few people seem to be voting right now, but I'd definitely upvote a formal answer!
Comments.UserDisplayName: DukeZhou
Tags: <neural-networks>
Title: How to implement an Automatic Learning Rate for a Neural Network?
UserDisplayName: Laceanu George
ThreadText: # How to implement an Automatic Learning Rate for a Neural Network?
  
  (Asked by: username_0 on 2017-07-30)
  
  I'm learning Neural Networks, and everything works as planned but, like humans do, adjusting themselves to learn more efficiently, I'm trying to understand conceptually how one might implement an auto adjusting learning rate for a Neural Network.
  
  I have tried to make it based on error, something like how bigger is error learning rate is getting bigger as well. *[Could use some clarification here--not entirely sure what you're saying. If can clarify, I'm happy to clean up the English. -username_2]*
  
  \*If you want give me an example give it on a C based language or math because I don't have experience with Python or Pascal.
  
  **Tags:** neural-networks
  
  > **Comments:**
  > **username_1**: Google "adaptive learning rate"  
  > **username_1**: http://cs231n.github.io/neural-networks-3/#ada  
  > **username_0**: thx, you can make this comment an answer to mark it as best answer?  
  > **username_2**: @username_1 very few people seem to be voting right now, but I'd definitely upvote a formal answer!  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2017-08-01. Score: 3 | Accepted answer
  
  Adjusting for learning rate is a common scenario in machine learning. There is rich literature about it, countless papers. The most common implementation:
  
  * AdaGrad
  * RMSProp
  * Adam
  
  There are many more variants. You'll need to do some research. Please take a look at:
  
  > <https://en.wikipedia.org/wiki/Stochastic_gradient_descent>
  
  [![enter image description here](https://i.sstatic.net/NhrR7.png)](https://i.sstatic.net/NhrR7.png)
  
  
  ---
  
  ### Answer by username_3 on 2017-09-09. Score: 3
  
  Apart from the already mentioned methods there is one which I came across recently is the Cyclic Learning Rates.
  
  Mainstream methods usually reduce the learning rates monotonically. In most of the cases they work well. Yet they do not do well with handling saddle points and local minima. CLR method claims to solve the problem by effectively handling the case of saddle points.
  
  Ref: <https://arxiv.org/abs/1506.01186>
  
  
  ---
  
  ### Answer by username_4 on 2019-02-22. Score: 0
  
  While adjusting the learning rate during the training is certainly interesting, so is finding of a good initial learning rate, too.
  
  [Cyclical Learning Rates for Training Neural Networks](https://arxiv.org/abs/1506.01186), [fast.ai (pytorch) implementation](https://docs.fast.ai/basic_train.html#lr_find).
  
  Here's a good practitioner's [overview of learning rate schedules](https://github.com/fastai/fastai/blob/master/courses/dl2/training_phase.ipynb) (python, but rather readable and with nice plots).
  
  
  ---
  
  