If you append a tensor with an attached computation graph (valid .grad_fn), the computation graph will be stored with it and should could see an increased memory usage in each iteration.
Is this the case for you or are you seeing any other issue?