Hey,
I train a RESNET34 with angular softmax loss for speaker identification.
a very strange effect happens while I stop the training and save the already trained model using “torch.save” and load the saved model to continue training (with the same parameters and continue with the last LR with the same optimizer). The test loss and train loss are decreasing significantly (so it’s not an overfitting) just for the first epoch of the second training session and then increase on the second and third epochs until it stabilises and then continues decreasing in the way I expect.
After the decreasing of the first epoch the model, the test accuracy is much better and so the model (If I stop it and check on the test test) better then even 5 or 6 epoch after the decreasing.
attached is a print screen of the train and val loss. I stopped and save the training at the end of epoch 12 and then I load it and continue to epoch 13 (you can see the drop). also I stop it on epoch 42 and the continue on epoch 43 with the same effect.
image990×726 55.8 KB
Do you have an idea why is this drop happens?