I also had the same problem. The admin had to reboot every time this happened. Something is quite wrong in the DataParallel.
Also I’ve often experienced the following:
running on multiple GPUs with DataParallel and DataLoader (num_worker = 16), try to terminate the process with ctrl+c then program freezes. Gpus hang, including nvidia-smi or other GPU related work.