When I trained resnet18 on ImageNet, I stop it at epoch 30. Then in a later period, i train it again resuming from the pretrained model(epoch 30). However, during the training process, the training loss always turns to nan half way. And I try three times, it always has the same problem(loss nan at different iteration of epoch 30)
The training code is the official training example at https://github.com/pytorch/examples/blob/master/imagenet/main.py 2
I don’t know how the solve the problem. If someone can help me, I will be quite grateful!
loss_nan_pic1431×237 21.6 KB