I don’t want to make a new entry for this so just replying to this thread.
MultiGPU keeps crashing with an OSError.
File “ExploreArch.py”, line 245, in 
for bt, data in enumerate(trainloader):
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py”, line 819, in next
return self._process_data(data)
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py”, line 846, in _process_data
data.reraise()
File “/usr/local/lib/python3.6/dist-packages/torch/_utils.py”, line 369, in reraise
raise self.exc_type(msg)
OSError: Caught OSError in DataLoader worker process 1.
Original Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py”, line 178, in _worker_loop
data = fetcher.fetch(index)
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in 
data = [self.dataset[idx] for idx in possibly_batched_index]
File “ExploreArch.py”, line 102, in getitem
OSError: [Errno 5] Input/output error: 
This system has 10 P4 cards. I’ve made the following observations:
With num_workers=0, there are no crashes. However, it slows down the processing.
With num_workers=20, I get the exact same error but on a much earlier epoch (say, 2)
With num_workers=10, the code ran smoothly until epoch 34 or so and then crashed.
I’ve seen a few links but the solution isn’t readily obvious? Any suggestions? If not, I’ll move this with num_workers = 0.