I did the following for PyTorch 0.4.1:
model = nn.DataParallel(model, device_ids=[3, 0, 1, 2])
And got:
torch/cuda/comm.py", line 40, in broadcast_coalesced
return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
RuntimeError: all tensors must be on devices[0]