This might possibly be due to exploding gradients. You should try to clip the value of gradient using torch.nn.utils.clip_grad_value or torch.nn.utils.clip_grad_norm.
To verify this, you can print the value of gradients during training and see if they go to infinity.