Hey there,
so I’ve been using PyTorch (0.3.1) for a while now and have tried to fix the following issue for some time now:
When I run a training script simulatenously (i.e. in two separate terminal instances) with different parameters, but same overall structure, on two separate 1080Tis, I get a full system freeze under Ubuntu 16.04 which I can’t leave without a force shutdown. Alse, there is no prior error message.
This doesn’t happen when I utilize either one of the GPUs alone.
Since I have encountered this issue, I have tried multiple things to fix this:
[1] Upgraded PSU to 1200W
[2] Set shared memory to high values.
[3] Use torch.multiprocessing.set_start_method(‘spawn’)
[4] Varied # of workers. Except for 0 workers, the error still occurs. Unfortunately, using the 0 worker option is not a feasible option for me. This most likely points to a multiprocessing issue; but I don’t know enough in that regard to investigate deeper. Also, its hard to tell since, again, no error message is thrown.
[5] Tested Ubuntu 18.04 - Same issue
[6] Tested PyTorch 0.4 - Same issue
[7] Worked my way through most of the nvidia-drivers.
Since this is most likely (although not necessarily) a specific issue either with my PC or my training script, I don’t expect full solutions in that regard. But pointers to options that could be helpful and that I haven’t looked into would be nice :).
Note:
The training script structure is fairly straightforward: Initialize Dataloader with paths to image files -> Iterate over them one by one.