Answers.Body: <p>In one hand you have the fact that if the initial step-size is a bit too optimistic, decay will at some point lead to a step-size that works</p>
  <p>However, there is a much more theoretical reason for it.<br />
  Indeed if you consider a non-smooth function, it's necessary sometimes to converge to decay your stepsize...</p>
  <p>As an example, consider <span class="math-container">$f(x) = |x|$</span>, with an initial guess of <span class="math-container">$x = 2.5$</span> and a stepsize of <span class="math-container">$\alpha = 1$</span></p>
  <p>Well, the gradient of such function is <span class="math-container">$1$</span> if <span class="math-container">$x &gt; 0$</span> and <span class="math-container">$-1$</span> if <span class="math-container">$x &lt; 0$</span></p>
  <p>Now, apply GD: <span class="math-container">$x = x - \alpha \nabla_x f(x) \rightarrow x = x - \nabla_x f(x)$</span>, and considering that the gradient on <span class="math-container">$x = 2.5$</span> is also 1, becomes <span class="math-container">$x = x - 1$</span></p>
  <p>Therefore, you will start from 2.5, then next time go to 1.5, then again to 0.5, and at that point, you'll jump to -0.5, and from there jump back to 0.5</p>
  <p>As you can see, you will never converge to the optimal <span class="math-container">$x = 0$</span>, except if you decay your stepsize</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-23T08:36:33.000Z
Answers.LastActivityDate: 2024-01-23T08:36:33.000Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>The keyword here is <em>can</em>. Deep learning models have millions of parameters making the loss landscape incredibly complex. Although everything in the slide still applies -- GD will take smaller steps as you get closer to a local minimum and you <em>could</em> theoretically converge using a fixed learning rate -- you almost always need extra tricks for training a neural network.</p>
  <p>For example, you might want to set the <a href="https://cs231n.github.io/neural-networks-3/?ref=jeremyjordan.me#annealing-the-learning-rate" rel="nofollow noreferrer">initial learning rate of your model higher and decrease it as training progresses</a> because the loss landscape for neural networks has many local minima and you don't want to converge prematurely into a suboptimal one. (Although, even this is a huge oversimplification of deep learning optimization.)</p>
  <p>Contrast that with a much simpler (e.g., linear) model. You won't see learning rate schedules being used to train these models because the loss landscape will be very simple (in fact, for a 1D linear model, it'll look exactly like the diagram in the slide). In this situation, the learning rate schedule <em>would</em> probably be redundant.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-23T05:13:35.970Z
Answers.LastActivityDate: 2024-01-23T05:34:08.693Z
Answers.LastEditDate: 2024-01-23T05:34:08.693Z
Answers.UserDisplayName: Alexander Wan
Body: <p>Andrew Ng said in his slide that:
  <a href="https://i.sstatic.net/ImraM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ImraM.png" alt="enter image description here" /></a>
  However, there are numerous types of 'learning rate schedules' in TensorFlow that change the learning rate profile as training progresses.</p>
  <p>If it's true that these adjustments are redundant then why do we need learning rate schedules?</p>
  
Tags: <gradient-descent>
Title: Why use learning rate schedules if weight updates automatically decrease when approaching local optimal?
UserDisplayName: Wong
ThreadText: # Why use learning rate schedules if weight updates automatically decrease when approaching local optimal?
  
  (Asked by: username_0 on 2024-01-23)
  
  Andrew Ng said in his slide that:
  [![enter image description here](https://i.sstatic.net/ImraM.png)](https://i.sstatic.net/ImraM.png)
  However, there are numerous types of 'learning rate schedules' in TensorFlow that change the learning rate profile as training progresses.
  
  If it's true that these adjustments are redundant then why do we need learning rate schedules?
  
  **Tags:** gradient-descent
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-01-23. Score: 2
  
  In one hand you have the fact that if the initial step-size is a bit too optimistic, decay will at some point lead to a step-size that works
  
  However, there is a much more theoretical reason for it.  
  Indeed if you consider a non-smooth function, it's necessary sometimes to converge to decay your stepsize...
  
  As an example, consider $f(x) = |x|$, with an initial guess of $x = 2.5$ and a stepsize of $\alpha = 1$
  
  Well, the gradient of such function is $1$ if $x > 0$ and $-1$ if $x < 0$
  
  Now, apply GD: $x = x - \alpha \nabla\_x f(x) \rightarrow x = x - \nabla\_x f(x)$, and considering that the gradient on $x = 2.5$ is also 1, becomes $x = x - 1$
  
  Therefore, you will start from 2.5, then next time go to 1.5, then again to 0.5, and at that point, you'll jump to -0.5, and from there jump back to 0.5
  
  As you can see, you will never converge to the optimal $x = 0$, except if you decay your stepsize
  
  
  ---
  
  ### Answer by username_2 on 2024-01-23. Score: 1
  
  The keyword here is *can*. Deep learning models have millions of parameters making the loss landscape incredibly complex. Although everything in the slide still applies -- GD will take smaller steps as you get closer to a local minimum and you *could* theoretically converge using a fixed learning rate -- you almost always need extra tricks for training a neural network.
  
  For example, you might want to set the [initial learning rate of your model higher and decrease it as training progresses](https://cs231n.github.io/neural-networks-3/?ref=jeremyjordan.me#annealing-the-learning-rate) because the loss landscape for neural networks has many local minima and you don't want to converge prematurely into a suboptimal one. (Although, even this is a huge oversimplification of deep learning optimization.)
  
  Contrast that with a much simpler (e.g., linear) model. You won't see learning rate schedules being used to train these models because the loss landscape will be very simple (in fact, for a 1D linear model, it'll look exactly like the diagram in the slide). In this situation, the learning rate schedule *would* probably be redundant.
  
  
  ---
  
  