@ptrblck Thanks for replying! Right now, I am doing it a bit differently (I will try your method right after). My initial learning rate was 1, so I’ve set the learning rate to 0.5, and called optimizer.step() 8 times, each time calculating the loss. However, when I try to step “back” by setting the learning rate to -3 and calling optimizer.step(), I don’t get the same loss that I had calculated in the previous step. Do you know why that would be the case?
Step 0, Accuracy 0.7734375, Loss 1.1175949573516846
Step 1, Accuracy 0.8515625, Loss 0.6774735450744629
Step 2, Accuracy 0.85546875, Loss 0.5671432614326477
Step 3, Accuracy 0.841796875, Loss 0.5469013452529907
Step 4, Accuracy 0.83203125, Loss 0.559360921382904
Step 5, Accuracy 0.80078125, Loss 0.5869375467300415
Step 6, Accuracy 0.783203125, Loss 0.6219274401664734
Step 7, Accuracy 0.76953125, Loss 0.6623443365097046
After stepping back with -3 lr: Loss: 0.7005575299263 (it should have been 0.6774735450744629)
I assumed that since I did not call loss.backwards(), the same gradients would have been applied each time. Is that incorrect?