I tried the new fp16 in native torch. However, when I continue my model training for my segmentation task I get loss as NaNs. With the same script, if I initialize the same model architecture from scratch then it works fine.
I set adam’s ‘ep’ to 1e-4 as well but it made no difference. I get NaN loss from the first batch continuing my trained model.
The model that I am continuing is already quite good at the task. I am guessing it is due to very low loss value but am not sure. How can I fix this?
Python 3.6
Pytorch 1.6
Linux
pip installed lightning
CUDA Version: 10.1
Nvidia-V100