I now have the exact same issue:
cuda runtime error (2) : out of memory at /home/gpu/dev/rt/pytorch/torch/lib/THC/generic/THCStorage.cu:66
Thrown after repeated training during which I stopped the kernel and started training again.
Of-course PyTorch returns False for torch.cuda.is_available()
Neither torch nor Jupyter are able to recover from this. The only solution for me was to restart the computer since the memory is not released otherwise.
BTW I am using this command line for monitoring:
watch -n 0.1 'ps f -o user,pgrp,pid,pcpu,pmem,start,time,command -psudo lsof -n -w -t /dev/nvidia*'
(source: https://stackoverflow.com/questions/8223811/top-command-for-gpus-using-cuda 22)