Body: <p>Is it a good idea to change the learning rate at each training step as a function of the loss? i.e. for points with high loss value, put a high learning rate and for low loss value a low learning rate (using a tailored function)?</p>
  <p>I know that the update of the parameters is done via <span class="math-container">$\gamma \nabla L$</span>, where <span class="math-container">$\nabla L$</span> is the gradient and <span class="math-container">$\gamma$</span> the learning rate, and that points with high loss should correspond to a high gradient. Hence the dependency of the update of the parameters on the value of the loss should be already contained, although in a more indirect way. Is doing what I propose dangerous and/or useless?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T11:36:36.347Z
Comments.Text: Have you ever heard of _adaptive learning rate_? I think that's what you are looking for.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T11:40:24.557Z
Comments.Text: Yes, but I've never read of a learning rate scheduler that updates the learning rate explicitly depending on the value of the loss at each step. Usually, for example, you halven the learning rate at a plateau, or something similar. I want to change the learning rate at each step
Comments.UserDisplayName: Giulio Ortali
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T11:44:19.080Z
Comments.Text: When you adjust your learning rate based on the loss, you have more chance to get stuck in local minima more often. It could be more efficient, but in general, I don't feel it has a positive effect in more complex situations.
Comments.UserDisplayName: Steven01123581321
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T11:49:41.050Z
Comments.Text: https://machinelearningmastery.com/learning-rate-for-deep-learning-neural-networks/ maybe this is a good resource to go to.
Comments.UserDisplayName: Steven01123581321
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T12:02:16.977Z
Comments.Text: Why you think you have more chances of getting a local minima? experience?
Comments.UserDisplayName: Giulio Ortali
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T12:36:36.867Z
Comments.Text: @JuliaOrtali Yes, but also from a theoretical point of view perhaps. I'm not saying it is always the case, but many times, losses around the minima are quite small, so the result will be that you keep taking smaller steps and eventually will never escape them. I do think that adapting your learning rate in a 'smarter' way will help. That's why optimizers like Adam and RMSprops are used more often instead of GD.So I guess that when adapting your learning rate for better outcomes will eventually lead you to an Adam optimizer.
Comments.UserDisplayName: Steven01123581321
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T02:07:14.947Z
Comments.Text: its an interesting idea, but i dont see an obvious implementation. these very clever variations of vanilla backprop usually involve an "obvious formulation", such as SGD's loss function, or the update value for Newton's method of optimization. i dont see an obvious way of changing the learning rate asides from what is already implicit in the standard formulation, but it can be open to formulation
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T02:08:09.130Z
Comments.Text: but as always, perhaps the best way to test this hypothesis is by testing it on somthing like MNIST
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Tags: <machine-learning><learning-rate>
Title: Is it a good idea to change the learning rate at each training step as a function of the loss?
UserDisplayName: Giulio Ortali
ThreadText: # Is it a good idea to change the learning rate at each training step as a function of the loss?
  
  (Asked by: username_0 on 2020-08-06)
  
  Is it a good idea to change the learning rate at each training step as a function of the loss? i.e. for points with high loss value, put a high learning rate and for low loss value a low learning rate (using a tailored function)?
  
  I know that the update of the parameters is done via $\gamma \nabla L$, where $\nabla L$ is the gradient and $\gamma$ the learning rate, and that points with high loss should correspond to a high gradient. Hence the dependency of the update of the parameters on the value of the loss should be already contained, although in a more indirect way. Is doing what I propose dangerous and/or useless?
  
  **Tags:** machine-learning, learning-rate
  
  > **Comments:**
  > **username_1**: Have you ever heard of _adaptive learning rate_? I think that's what you are looking for.  
  > **username_0**: Yes, but I've never read of a learning rate scheduler that updates the learning rate explicitly depending on the value of the loss at each step. Usually, for example, you halven the learning rate at a plateau, or something similar. I want to change the learning rate at each step  
  > **username_2**: When you adjust your learning rate based on the loss, you have more chance to get stuck in local minima more often. It could be more efficient, but in general, I don't feel it has a positive effect in more complex situations.  
  > **username_2**: https://machinelearningmastery.com/learning-rate-for-deep-learning-neural-networks/ maybe this is a good resource to go to.  
  > **username_0**: Why you think you have more chances of getting a local minima? experience?  
  > **username_2**: @JuliaOrtali Yes, but also from a theoretical point of view perhaps. I'm not saying it is always the case, but many times, losses around the minima are quite small, so the result will be that you keep taking smaller steps and eventually will never escape them. I do think that adapting your learning rate in a 'smarter' way will help. That's why optimizers like Adam and RMSprops are used more often instead of GD.So I guess that when adapting your learning rate for better outcomes will eventually lead you to an Adam optimizer.  
  > **username_3**: its an interesting idea, but i dont see an obvious implementation. these very clever variations of vanilla backprop usually involve an "obvious formulation", such as SGD's loss function, or the update value for Newton's method of optimization. i dont see an obvious way of changing the learning rate asides from what is already implicit in the standard formulation, but it can be open to formulation  
  > **username_3**: but as always, perhaps the best way to test this hypothesis is by testing it on somthing like MNIST  
  
  ---
  
  ## 0 answers
  
  