Answers.Body: <p>There is an approach to machine learning, called <a href="https://en.wikipedia.org/wiki/Simulated_annealing" rel="nofollow noreferrer">Simulated Annealing</a>, which varies the rate: starting from a large rate, it is slowly reduced over time. The general idea is that the initial larger rate will cover a broader range, while the increasingly lower rate then produces a less 'erratic' climb towards a maximum.</p>
  <p>If you only use a low rate, you risk getting stuck in a local maximum, while too large a rate will not find the best solution but might end up close to one. Adjusting the rate gives you the best of both.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-30T13:33:14.737Z
Answers.Text: Thank you Oliver. It hadn't occurred to me that having too low a learning rate could result in getting stuck in a local maximum. I'm aware of learning decay in Tensorflow and have been using it but not aware how important it is to use both appropriately. Am I right in saying we would want big jumps early on and then a stable learning later as it optimises on hopefully the best maxima?
Answers.UserDisplayName: Oliver P
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-30T15:35:37.493Z
Answers.Text: @OliverP Yes, exactly.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-23T14:32:27.157Z
Answers.Text: That should have been local minimum, @OliverP
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-25T11:36:02.693Z
Answers.Text: @DavidHoelzer No, local maximum is correct.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-30T13:04:53.857Z
Answers.LastActivityDate: 2020-09-30T13:04:53.857Z
Answers.UserDisplayName: Oliver Mason
Answers.Body: <p>If you have an erratic loss landscape, it can lead to an unstable learning curve. Thus, it's always better to choose a simpler function which creates a simple landscape. Sometimes even due to uneven dataset distribution, we can observe those jumps/irregularities in the training curve.</p>
  <p>And yes, those jumps do mean it might've found something significant in the landscape. Those jumps can arise while the model is exploring the multiple local minima of the landscape.</p>
  <p>During Machine Learning Optimization, we usually use algorithms like <strong>Stochastic Gradient Descent</strong> and <strong>Adam</strong> to find <em>Local Minima's</em> whereas approaches like <strong>Simulated Annealing</strong> find <em>global minima</em>. There have been multiple discussions around why to use local minima's instead of global minima. Some argue that local minima are just as useful as global minima in case of machine learning problems.</p>
  <p>Thus, Stable Learning is preferable as it symbolizes that the model is converging to local minima.</p>
  <h1>References</h1>
  <hr />
  <p>You can read <a href="https://arxiv.org/pdf/1906.06821.pdf" rel="nofollow noreferrer">A Survey of Optimization Methods from
  a Machine Learning Perspective</a> by Shiliang Sun, Zehui Cao, Han Zhu, and Jing Zhaoto et al. and read about all the optimization functions commonly used in Machine Learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-30T15:41:21.117Z
Answers.LastActivityDate: 2020-09-30T15:41:21.117Z
Answers.UserDisplayName: Saurav Maheshkar
Body: <p>A stable/smooth learning validation curve often seems to keep improving over more epochs than an unstable learning curve. My intuition is that dropping the learning rate and increasing the patience of a model that produces a stable learning curve could lead to better validation fit.</p>
  <p>The counter argument is that jumps in the curve could mean that the model has just learned something significant, but they often jump back down or tail off after that.</p>
  <p>Is one better than the other? Is it possible to take aspects of both to improve learning?</p>
  
Tags: <machine-learning><training><learning-rate>
Title: Is stable learning preferable to jumps in accuracy/loss
UserDisplayName: Oliver P
ThreadText: # Is stable learning preferable to jumps in accuracy/loss
  
  (Asked by: username_0 on 2020-09-30)
  
  A stable/smooth learning validation curve often seems to keep improving over more epochs than an unstable learning curve. My intuition is that dropping the learning rate and increasing the patience of a model that produces a stable learning curve could lead to better validation fit.
  
  The counter argument is that jumps in the curve could mean that the model has just learned something significant, but they often jump back down or tail off after that.
  
  Is one better than the other? Is it possible to take aspects of both to improve learning?
  
  **Tags:** machine-learning, training, learning-rate
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-09-30. Score: 2
  
  There is an approach to machine learning, called [Simulated Annealing](https://en.wikipedia.org/wiki/Simulated_annealing), which varies the rate: starting from a large rate, it is slowly reduced over time. The general idea is that the initial larger rate will cover a broader range, while the increasingly lower rate then produces a less 'erratic' climb towards a maximum.
  
  If you only use a low rate, you risk getting stuck in a local maximum, while too large a rate will not find the best solution but might end up close to one. Adjusting the rate gives you the best of both.
  
  > **Comments:**
  > **username_0**: Thank you Oliver. It hadn't occurred to me that having too low a learning rate could result in getting stuck in a local maximum. I'm aware of learning decay in Tensorflow and have been using it but not aware how important it is to use both appropriately. Am I right in saying we would want big jumps early on and then a stable learning later as it optimises on hopefully the best maxima?  
  > **username_1**: @OliverP Yes, exactly.  
  > **username_2**: That should have been local minimum, @OliverP  
  > **username_1**: @DavidHoelzer No, local maximum is correct.  
  
  ---
  
  ### Answer by username_3 on 2020-09-30. Score: 1
  
  If you have an erratic loss landscape, it can lead to an unstable learning curve. Thus, it's always better to choose a simpler function which creates a simple landscape. Sometimes even due to uneven dataset distribution, we can observe those jumps/irregularities in the training curve.
  
  And yes, those jumps do mean it might've found something significant in the landscape. Those jumps can arise while the model is exploring the multiple local minima of the landscape.
  
  During Machine Learning Optimization, we usually use algorithms like **Stochastic Gradient Descent** and **Adam** to find *Local Minima's* whereas approaches like **Simulated Annealing** find *global minima*. There have been multiple discussions around why to use local minima's instead of global minima. Some argue that local minima are just as useful as global minima in case of machine learning problems.
  
  Thus, Stable Learning is preferable as it symbolizes that the model is converging to local minima.
  
  # References
  
  ---
  
  You can read [A Survey of Optimization Methods from
  a Machine Learning Perspective](https://arxiv.org/pdf/1906.06821.pdf) by Shiliang Sun, Zehui Cao, Han Zhu, and Jing Zhaoto et al. and read about all the optimization functions commonly used in Machine Learning.
  
  
  ---
  
  