<p>In this blog post: <a href="http://www.argmin.net/2016/04/18/bottoming-out/" rel="nofollow noreferrer">http://www.argmin.net/2016/04/18/bottoming-out/</a></p>
<p>Prof Recht shows two plots:</p>
<p><a href="https://i.sstatic.net/IKNxh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IKNxh.png" alt="enter image description here" /></a></p>
<p><a href="https://i.sstatic.net/T0njq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/T0njq.png" alt="enter image description here" /></a></p>
<p>He says one of the reasons the plot below has a lower train-test gap is because that model was trained with a lower learning rate (and he also manually drops the learning rate at 120 epoch).</p>
<p>Why would a lower learning rate reduce overfitting?</p>

CC BY-SA 4.0
2020-07-14T09:59:24.283Z
bigger learning rate makes the loss curve oscillation high, and it works for training data only
Dan D.
CC BY-SA 4.0
2020-07-14T12:39:04.340Z
BTW, two represented figures have not the same y-scale and it can be confusing and deceptive!
OmG
<neural-networks><machine-learning><generalization><learning-rate>
Why does learning rate reduce train-test generalization gap?
user3180
# Why does learning rate reduce train-test generalization gap?

(Asked by: username_0 on 2020-07-14)

In this blog post: <http://www.argmin.net/2016/04/18/bottoming-out/>

Prof Recht shows two plots:

[![enter image description here](https://i.sstatic.net/IKNxh.png)](https://i.sstatic.net/IKNxh.png)

[![enter image description here](https://i.sstatic.net/T0njq.png)](https://i.sstatic.net/T0njq.png)

He says one of the reasons the plot below has a lower train-test gap is because that model was trained with a lower learning rate (and he also manually drops the learning rate at 120 epoch).

Why would a lower learning rate reduce overfitting?

**Tags:** neural-networks, machine-learning, generalization, learning-rate

> **Comments:**
> **username_1**: bigger learning rate makes the loss curve oscillation high, and it works for training data only  
> **username_2**: BTW, two represented figures have not the same y-scale and it can be confusing and deceptive!  

---

## 0 answers

