Body: <p>In this blog post: <a href="http://www.argmin.net/2016/04/18/bottoming-out/" rel="nofollow noreferrer">http://www.argmin.net/2016/04/18/bottoming-out/</a></p>
  <p>Prof Recht shows two plots:</p>
  <p><a href="https://i.sstatic.net/IKNxh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IKNxh.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/T0njq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/T0njq.png" alt="enter image description here" /></a></p>
  <p>He says one of the reasons the plot below has a lower train-test gap is because that model was trained with a lower learning rate (and he also manually drops the learning rate at 120 epoch).</p>
  <p>Why would a lower learning rate reduce overfitting?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-14T09:59:24.283Z
Comments.Text: bigger learning rate makes the loss curve oscillation high, and it works for training data only
Comments.UserDisplayName: Dan D.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-14T12:39:04.340Z
Comments.Text: BTW, two represented figures have not the same y-scale and it can be confusing and deceptive!
Comments.UserDisplayName: OmG
Tags: <neural-networks><machine-learning><generalization><learning-rate>
Title: Why does learning rate reduce train-test generalization gap?
UserDisplayName: user3180
ThreadText: # Why does learning rate reduce train-test generalization gap?
  
  (Asked by: username_0 on 2020-07-14)
  
  In this blog post: <http://www.argmin.net/2016/04/18/bottoming-out/>
  
  Prof Recht shows two plots:
  
  [![enter image description here](https://i.sstatic.net/IKNxh.png)](https://i.sstatic.net/IKNxh.png)
  
  [![enter image description here](https://i.sstatic.net/T0njq.png)](https://i.sstatic.net/T0njq.png)
  
  He says one of the reasons the plot below has a lower train-test gap is because that model was trained with a lower learning rate (and he also manually drops the learning rate at 120 epoch).
  
  Why would a lower learning rate reduce overfitting?
  
  **Tags:** neural-networks, machine-learning, generalization, learning-rate
  > **Comments:**
  > **username_1**: bigger learning rate makes the loss curve oscillation high, and it works for training data only  
  > **username_2**: BTW, two represented figures have not the same y-scale and it can be confusing and deceptive!  
  
  ---
  
  ## 0 answers
  
  