I am training a self supervised learning model.
The learning rate, loss goes from learning rate = 0.00073495, loss = 310.6790 to learning rate = 0.00073412, loss = nan, in the middle of the 52th epoch
I have read earlier suggestions as well, but since my loss in not exploding could there be other issue?
Since, the error occurs after 50th epoch, and my each epoch takes upto 30min, the issue can not be reproduced immediately. I am using pytorch nightly binaries 1.6+dev