I’m using a DataLoader with 28 workers. After training for 3.5k iterations (~ 6 hours), I got the following error:
    idx, data = self._get_data()
  File "...site-packages/torch/utils/data/dataloader.py", line 848, in _get_data
    success, data = self._try_get_data()
  File "...site-packages/torch/utils/data/dataloader.py", line 811, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "...python3.6/queue.py", line 173, in get
    self.not_empty.wait(remaining)
  File ".../python3.6/threading.py", line 299, in wait
    gotit = waiter.acquire(True, timeout)
  File "...site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 487) is killed by signal: Hangup. 

Has anyone had DataLoader worker killed by SIGHUP?
There are a number of issues focused on DataLoader workers being killed by signal BusError and Killed. However, I can’t find any mention of DataLoader workers being killed by SIGHUP. My understanding of SIGHUP is that it is a signal sent to processes when their terminal is closed, so it strikes me as an odd signal for a worker process to be killed by.
Has anyone run into this before / have any insight into what kind of issue the SIGHUP signal would suggest?
Thanks!