Based on your description it seems that you might be (accidentally) storing tensors (with the computation graph), which would increase the memory usage in each iteration.
Try to isolate the code which would be responsible for it by removing other code parts and by printing the memory usage in each iteration. Once you have isolated a subset of potential methods, check if they are storing any object in a list or create references to it in any other way.