In different runs it’s happening in different moments, so it is not about samples because I turned off shuffling the data.
I’m using pretrained ULMFiT, I’m encoding text with sentencepiece.
Is it possible that after optimize.step(), weights in the model are getting NaN values? Hmm, but why is it possible if I set requires_grad = False in this text model? Maybe it’s not that.
What would you suggest next?