Hi, when I use DataParallel on multi-gpu, I got the following error message recently:
PS: all the codes run normally using single GPU.

RuntimeError: Expected tensor for argument #1 ‘input’ to have the same device as tensor for argument #2 ‘weight’; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)

It seems my data are on cuda1 but network paramters are on cuda0.
I google the error message and want to figure out what happened, but I cannot find a specific solution and still confused, so I post it and really appreciate for your help.
I want to train a custom encoder-decoder model namely EncoderDecoder, and it consist of multiple custom modules, e.g., custom_module1, custom_module2…, each module only have a __init__  and forward function.
and basically I use it like:
device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')
...
class Model:
    def __init__(self, config: Configuration, device: torch.device):
        super(Model, self).__init__()    
        ...
        self.encoder_decoder = EncoderDecoder(config, device).to(device)
        self.encoder_decoder = torch.nn.DataParallel(self.encoder_decoder) 
    
    def train_on_batch(self, ...):
        ...
        output = self.encoder_decoder(input)

I am using pytorch:1.3.
And I am not sure why this happened. so I check the devices used both outside and inside the models, in the outside (the class Model), the input data and the self.encoder_decoder are all on cuda:0 (the device). And issue happened inside the model (self.encoder_decoder).
In the custom_module1 of self.encoder_decoder.
I checked the input data are:

audio:  torch.Size([5, 1, 257, 376]) cuda:0 torch.float16
audio:  torch.Size([5, 1, 257, 376]) cuda:1 torch.float16

which should be correct right? data are both on cuda0 and cuda1.
But it seems conflict with the model weights, which obviously should be stored only on cuda0? I guess this is why this bug happened?
and I checked the device inside the EncoderDecoder(config, device):

device in EncoderDecoder (an DataParallel model’s device):  cuda:0
device in EncoderDecoder (an DataParallel model’s device):  cuda:0

Is this why this bug happened? but I still don’t know how to handle it? Or, the previous information are normal and correct when using DataParallel? if then, I don’t know why this issue appears?
Thanks for your help!