Deep Learning, Artificial Intelligence, Convolutional Network, Image Classification, Resnet.
values and therefore far away from the final solution. Using a too large learning rate may result in numerical instability. In the warmup heuristic, a small learning rate is used at the beginning and then the initial learning rate is switched back later on. The first m batches (e.g. 5 data epochs)