Machine Learning, Deep Learning, Gradient Descent, Neural Networks, Learning Rate.
ability to react to new data. In order to find a more stable local minimum, we can increase the learning rate from time to time, encouraging the model to “jump” from one local minimum to another if it is in a steep trough. This is the “restarts” in SGDR. Image 9: Increasing the learning rates every