If you are seeing an increase in the memory usage in each iteration, check if you are storing any tensors, which might be attached to the computation graph (such as the model output), in e.g. a list. This would not only store the tensor, but also the entire computation graph.