Are the inputs you feed to the model on the same device (cuda:0) when you run the training loop?
Also, would it be possible for you to come up with a small example that reproduces the problem you’re seeing? It would be easier to debug the issue that way.