In my case, before the iteration I got several tensor which will be calculated through the whole training stage. As I noticed, the memory keeps increasing, even though I use .cpu(). The code structure is like allowings:
tensor_for_accumulate = [ ]
for iter in range(10):
…
tensor_current = …
tensor_for_accumulate += tensor_current
I found that the length and dtype is the same during the training. Thus, I can’t understand why the memory is increasing all the time.