Hello,
I am training a model using 1 of my 2 GPUs and wanted to ask something about the mechanics of GPU memory usage of PyTorch.
After I trained the model and evaluated and got results, there is finally no continuing process inside my ipython notebook. However, when I check the GPU memory, I see that a huge chunk of memory is being used off of the GPU.
Why is happening? The same thing sometimes happen in Google Colab where I have to find some process and kill it manually. Why wouldn’t it let go of my GPU memory so I can quickly try some other experiment and see the results of some other configuration?
(Also what do you recommend I do in this situation, how do I solve this and do quick experiments one after another when I am done with evaluation/training?)
Thank you very much in advance for your help.