I don’t know if this is the cause of your issue, but I noticed you have model.cuda() inside the for loop. And the same for the cross_entropy_loss. You can send the model and loss to the gpu outside the loop once. You can also set the loss to zero at the beginning of second for loop. Hope this helps