I just had a memory leak that persisted with 90% of the GPU memory occupied AFTER I closed python and did killall python + killall jupyter for good measure. I also checked nvidia-smi and saw the 90% GPU memory usage but no corresponding processes actually using it…
How are things like this possible? How do I prevent them? And how do I diagnose them?
The only recourse I had that I knew of at the time was to reboot. But I really need some context to this problem…
P.S. I just got this to happen again by interrupting a jupyter notebook cell during the training process. So that may be related. But still I’m not sure how to deal with these issues in the long run.