Definitely sounds similar. I’m not manually starting any threads, just using num_workers in dataloader.
Sounds like an NVCC version mismatch can cause things to hang? Nvidia SMI shows
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   35C    P0    27W / 250W |      0MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+


But I am using
torch                      1.6.0+cu101
torchvision                0.6.0+cu101

Think that could be the issue? There is some other talk about NCCL in that issue but I don’t think that applies since I’m using a single GPU.