Machine Learning, Deep Learning, Gradient Descent, Neural Networks, Learning Rate.
In the second “half cosine” period, we descend into another local minimum. Image 12: Descending into a more stable local minimum And then we sharply increase the learning rate again. Only this time, because we are in a more stable region of the loss function, this “restart” does not take us out of