Machine Learning, Deep Learning, Gradient Descent, Neural Networks, Learning Rate.
iteration. Left: Decrease learning rate by steps; Right: Linearly decrease learning rate SGDR uses cosine annealing, which decreases learning rate in the form of half a cosine curve, like so: Image 4: Cosine Annealing This is a good method because we can start out with relatively high learning