Thank you for your comment. However, when I did have a loss function and perform backprop the RAM also exploded. I was only trying to isolate the issue by removing the loss.
You know, I went to get my code with the loss still in it and ended up finding the real bug! I was only calling zero_grad() on my optimizer every epoch rather than every minibatch… Dumb mistake.
But it makes sense why keeping track of so many little gradients from a stepwise approach minibatch after minibatch made the RAM explode before the next epoch could reset it.
Its actually fortunate this happened as otherwise I wouldn’t have spotted the bug and would have been doing backprop every minibatch on gradients calculated from previous minibatches I had already backpropped on!
Thank you for your help getting me to spot what was actually wrong!