I would really want to understand this properly. There are many many threads about GPU memory problems, but it is hard to get a proper understanding of the details behind how Pytorch manages the memory one the GPU.
I have a problem where I run out of memory when doing crossvalidation, where for each fold, I load and train a new model and then evaluate the model before the same Python variable is used to hold a new model, train that etc.
Initially I thought that if a Python variable references a tensor on the GPU and that Python variable then gets re-assigned, that the memory on the GPU would get released because no variable is pointing to it any more. But this does not seem to be the case: the GPU memory needed by the model seems to accumulate over each fold even though the python variable that holds the model gets re-assigned.
However, if I move the model to the cpu and do torch.cuda.empty_cache() before reassigning the variable, the memory does not seem to accumulate or at least not as much.
Is this intended behaviour? Should GPU memory deallocation happen automatically or is a specific procedure needed? What is going on?