I am trying to use torch.nn.DataParallel to train a model in multiple gpus in 0.4.1, but I find some different solution from google, which makes me confused. Following is the basic configuration.

model = resnet50()
device_0 = torch.device(‘cuda:0’)
device_ids = [0, 1, 2, 3]

I want to know:


which order below is correct?



model.to(device_0)
model = torch.nn.DataParallel(model, device_ids=device_id)

or



model = torch.nn.DataParallel(model, device_ids=device_id)
model.to(device_0)





for optimizer ,

what parameter should I give: the model before torch.nn.DataParallel or after that.
whether it is necessary to wrap the optimizer using torch.nn.DataParallel



In DataParallel 6 source code, the outputs from different gpus is gathered to output device, and loss is computed in output device, then backward is done in output devices or all other devices and how?