I had a working model. It ran and was very consistent with its result regarding the number of GPUs.
This means it already gave similar results when running on 1-4 GPUs (all combinations had been tested).
Now a came across some weird issue, Everything works fine on single or double GPUs (done with ““model = nn.DataParallel(model, device_ids=devices)””).
If I try to use 3 GPUs it crashes with  NCCL Error 2: unhandled system error.
Moreover, if I use 4 GPUs it just gets stuck, ctrl+c doesn’t help of course (luckily I’m working via terminal and ssh so I just break the pipe to kill the run).