Deep Learning, Artificial Intelligence, Convolutional Network, Image Classification, Resnet.
stage is ignored), then at batch t, the learning rate ηt is computed as: where η is the initial learning rate. This is called “cosine” decay. Visualization of learning rate schedules with warm-up. As can be seen, the cosine decay decreases the learning rate slowly at the beginning, and then becomes