I’m just running the script in the terminal, do i have to worry about exception being swallowed up?
Also I’m not sure about the way I coded try except, Is there a specific exception I should handle to avoid other important exception to be missed?
Edit: the error below is probably just due to not having enough ram. I will see if some error happens with 0 workers.
With 8 workers the error below happens within 10 seconds, probably the time required to load a single batch, not sure if is the same error as the previous one.
With 0 workers I got no error within 16 minutes of execution. I will leave it running and see if something happens.
I don’t feel like It is guaranteed to never happen even after hours and hours of training, since I previously managed to run the code for several hours with 4 workers.
I haven’t experimented enough with this, but It is clear that I will never fetch the first batch with 8 workers, so it looks strongly dependent with the number of workers.
Traceback (most recent call last):
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 872, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 88, in rebuild_tensor
    t = torch._utils._rebuild_tensor(storage, storage_offset, size, stride)
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/_utils.py", line 133, in _rebuild_tensor
    return t.set_(storage, storage_offset, size, stride)
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 4157) is killed by signal: Killed. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train_AD.py", line 333, in <module>
    train_net(args,cfg, DEBUG, model)
  File "train_AD.py", line 211, in train_net
    for data in train_dataloader:
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1068, in _next_data
    idx, data = self._get_data()
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1034, in _get_data
    success, data = self._try_get_data()
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 885, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 4157) exited unexpectedly


Below with a timeout different from zero:
Traceback (most recent call last):
  File "train_AD.py", line 334, in <module>
    train_net(args,cfg, DEBUG, model)
  File "train_AD.py", line 212, in train_net
    for data in train_dataloader:
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1068, in _next_data
    idx, data = self._get_data()
  File "/home/lorenzo/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1021, in _get_data
    raise RuntimeError('DataLoader timed out after {} seconds'.format(self._timeout))
RuntimeError: DataLoader timed out after 1 seconds