I think my comment above is relevant:


Doing
del input
torch.cuda.empty_cache()
gc.collect()

in your code has very little impact as the Tensor pointed by input is still referenced in the  forward  call of the nn Module and so has not gone out of scope yet. So nothing can really be cleaned up.


Here, the input to your model, data is the same Tensor as input that you get during forward. But even if you don’t store input, you still hold onto data in your training loop, so it cannot be freed.
Changing to
output = mode(data)
del data

should help.
Also if and only if you introduce reference cycles, you need to gc.collect() to make sure things are freed.
Let me know if it works the way you want