I met the same problem in my program yesterday. It seems that this error only happens when device_ids[0] is not 0.
I tried to debug my code with pdb and found the DataParallel.forward may fail to replicate the original model’s parameters into gpu device_ids[0] if device_ids[0]!=0. You can check it in line 33 of torch/nn/parallel/replicate.py, param_copies[0].get_device()  is always 0 after executing param_copies = Broadcast(device_ids)(param) no matter what device_ids[0] is.
(Pdb) l
 29         for param in module.parameters():
 30             if param in seen_params:
 31                 continue
 32             seen_params.add(param)
 33 B           param_copies = Broadcast(device_ids)(param)
 34  ->         for param_copy, remap in zip(param_copies, param_remap):
 35                 remap[param] = param_copy
(Pdb) p param_copies[0].get_device()
0
(Pdb) p param_copies[1].get_device()
3
(Pdb) p device_ids
[2, 3]

I am still working on this and have not got any solutions so far.