This most likely happen because you either store things in a list that ever grows at each iteration or if you hold onto the computational graph of the whole history.
You should be able to check the first one by printing list size in your code (in particular, buffer)
For the second one, you want to make sure that if you store things for which you only want the value, and not gradients to be backpropagated, use .detach().
You can you use torchviz 4 to print the computational graph associated with your loss to make sure it does not grow at every iteration. Otherwise you need to identify where it links to the previous operations, and use .detach() to break it at this point.