Sometimes the same issue occured, when I killed a script with multiple DataLoader workers with CTRL+C.
It seems to me, killing the process like this, sometimes does not kill all child processes, so that these processes still use the GPU memory.
If no other python scripts are running on your machine, try killing them with killall python in your terminal.
This usually works for me.