Machine Learning, Artificial Intelligence, Neural Networks, Data Science, Towards Data Science.
First, we can start with a small learning rate and increase it on every batch exponentially. Simultaneously, we can compute the loss function on a validation set. This also works for finding bounds for cyclic learning rates. The plot below shows the exponentially-increasing learning rate as a