You should not save from one iteration to the other a Tensor that has requires_grad=True. If you want to save it for later inspection (or accumulating the loss), you should .detach() it before. So that pytorch knows you won’t try and backpropagate through it.