Machine Learning, Deep Learning, Gradient Descent, Neural Networks, Learning Rate.
few iterations to “restart” the gradient descent // Source In the first “half cosine” period in the graph above, we descend into a local minimum, like so: Image 10: Descending into local minimum Suddenly, we increase the learning rate, taking a big step in the next iteration: Image 11: “Restarting”