Machine Learning, Artificial Intelligence, Neural Networks, Data Science, Towards Data Science.
minima (overfitting). Saddle points are abundant in high dimensions, and convergence becomes very slow, if not impossible. Thus, if we use a purely decreasing learning rate it is easy to get stuck in a single location, especially in higher dimensions. Cyclic learning rates raise the learning rate