I noticed that 99% of the GPU RAM is still being used after and no process is listed by nvidia-smi after PyTorch aborts with an
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1518244421288/work/torch/lib/THC/generic/THCStorage.see you:58

One could run sudo nvidia-smi --gpu-reset -i 0 as a remedy, however,  it would be nice if PyTorch could free the memory after such a “crash” – not sure if this is only a problem in my particular case, so I wanted to ask if others have the same issue (before I post a GitHub issue)
EDIT 1: I should mention that the memory is freed if I e.g., interrupt or kill a process, it’s just the RuntimeError: cuda runtime error (2) : out of memory which doesn’t seem to free the memory
EDIT 2: It does not seem to be just a reporting issue by nvidia-smi as I can’t run even smaller networks on that card after I encountered that issue