Based on the description it seems that you are storing tensors in e.g. a list, which are still attached to the computation graph. Thus you wouldn’t only store the tensors, but the entire graph with all intermediate tensors in each iteration.
If you don’t need to call backward() on the stored tensor or any tensor created from it, you could detach() the tensor before storing it in the list (or any other container).