I’m shocked.
I tried same thing with torch.no_grad() and text model in evaluation mode and I got something like this:
2020-08-16 05:13:52 UTC -- 5435
2020-08-16 05:13:52 UTC -- tensor(2.2348, device='cuda:0', grad_fn=<AddBackward0>)
text_output
2020-08-16 05:13:52 UTC -- tensor([nan, nan, nan, nan, nan, nan, nan, nan, (...)], device='cuda:0')
(...)
2020-08-16 05:13:53 UTC -- 5436
2020-08-16 05:13:53 UTC -- tensor(nan, device='cuda:0', grad_fn=<AddBackward0>)

The number 5435 is an iterator. So we can see that first there were NaNs in the text output and after that NaN in the loss value! If it was made with torch.no_grad() so it has to be something wrong with text model. But what could it be if I trained it using the same data?
I’m loading state dict to this text model, so maybe it is corrupted? I’ll try to train once again this text model and check if it will help. If not, maybe there is something wrong with sentencepiece model?
And one more thing - I’ve also tried to run model with training and once again I see that this problem happens in different places in dataset (during different iterations) - I’m running without shuffe, so I think that it’s not connected to data.
Best regards