Tensors are on different GPUS
<SCODE>img = Variable(img).cuda()
label = Variable(label).cuda()
<ECODE> <SCODE>Traceback (most recent call last):
  File "trainer.py", line 53, in <module>
    outputs = resnet(rmap)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 202, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 92, in forward
    outputs = self.parallel_apply(replicas, scattered, gpu_dicts)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 102, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py", line 50, in parallel_apply
    raise output
RuntimeError: tensors are on different GPUs
<ECODE>
ycszen
smth
ycszen
I met the same problem in my program yesterday. It seems that this error only happens when device_ids[0] is not 0. <SCODE>(Pdb) l
 29         for param in module.parameters():
 30             if param in seen_params:
 31                 continue
 32             seen_params.add(param)
 33 B           param_copies = Broadcast(device_ids)(param)
 34  ->         for param_copy, remap in zip(param_copies, param_remap):
 35                 remap[param] = param_copy
(Pdb) p param_copies[0].get_device()
0
(Pdb) p param_copies[1].get_device()
3
(Pdb) p device_ids
[2, 3]
<ECODE> I am still working on this and have not got any solutions so far.
nicklhy
I got the same problem. Is it a bug of PyTorch?
melody-rain
nicklhy
mehdi-shiba
Please change your code to xx.cuda(device=gpus[0]) instead of xx.cuda(device=0).
nicklhy
By moving all variable tensors and models into gpus[0](not gpu 0), the first model replica will be on the right device you selected(gpus=[xx, xx, xx]) even though the broadcast function will still ignore it.
nicklhy
mehdi-shiba
has this been fixed in PyTorch?
lakehanne
What’s wrong??
hyqneuron
smth
stacked.twix
In case if I have, what is the solution?
botcs
My 2 cents worth. I’m used to creating lists of Conv objects as I did in Keras and placing them into a class attribute. Turns out this doesn’t work if you’re using PyTorch and Data parallel as it has to be added using add_module.
jarrelscy
smth
Thanks! That is a great tip.
jarrelscy
I am also curious what to do if my model has tensor and variables used in self?
kits
furlat