It looks like the cuda code is doing stuff out of order and not properly coping with the synchronisation.
You can try adding torch.cuda.synchronize() before the training loop. I’m only guessing, I don’t have a GPU so I can’t say whether it will help.