return loss I think this is the problem. Can you try with return loss.detach().
Problems like these are generally caused when you training loop is holding on to some things that it shouldn’t. Also, make sure that you are not storing some temporary computations in an ever growing list without deleting them.
To track this down, you can also get timings for different parts separately: data loading, network forward, loss computation, backward pass and parameter update. Hopefully just one will increase and you will be able to see better what is going on.