Hi,
The code I’m working on randomly used to get stuck. I was able to come up with a minimal example 15 that I found had similar behavior. (ran a loop of 100 runs and it got stuck at some point; In the example, I used the Office-Home dataset, but I suppose the specific dataset doesn’t matter)
Here’s the stack trace when I Ctrl+c’ed :
Starting training
[15:32 26-08-2020] Step [0] Loss : 12.5213
[15:32 26-08-2020] Step [1] Loss : 12.5203
[15:32 26-08-2020] Step [2] Loss : 12.5186
[15:32 26-08-2020] Step [3] Loss : 12.5178
[15:32 26-08-2020] Step [4] Loss : 12.5234
[15:32 26-08-2020] Step [5] Loss : 12.5203
^CTraceback (most recent call last):
  File "tests/test_dl_wo_wandb.py", line 65, in <module>
    imgs3 = imgs3.cuda(non_blocking=True)
  File "/home/samarth/anaconda3/envs/pytorch3conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/home/samarth/anaconda3/envs/pytorch3conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 974, in _next_data
    idx, data = self._get_data()
  File "/home/samarth/anaconda3/envs/pytorch3conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 931, in _get_data
    success, data = self._try_get_data()
  File "/home/samarth/anaconda3/envs/pytorch3conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 779, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/samarth/anaconda3/envs/pytorch3conda/lib/python3.7/queue.py", line 179, in get
    self.not_empty.wait(remaining)
  File "/home/samarth/anaconda3/envs/pytorch3conda/lib/python3.7/threading.py", line 300, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/samarth/anaconda3/envs/pytorch3conda/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/home/samarth/anaconda3/envs/pytorch3conda/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 3508) is killed by signal: Terminated.

Is there something I’m doing in the example that is known to possibly deadlock?
Much appreciated!