Hi everyone.
I am having some trouble with data parallelism on a system with two gpus. After reading the tutorial it seems it is easy but I cannot get it running.
The point is I have two models, say model1 and model2 and I want to train both on both gpus. The point is, as i have understood, that pytorch get a minibatch let say 100 samples, and split it equally into both gpus and the average the weights. What i do is:
model1=torch.nn.DataParallel(model1).cuda()
model2=torch.nn.DataParallel(model2).cuda()
Then tipe
for i in trainLoader: o=model1(i) output=model2(o)
I get the error:
raise runtime error(‘all tensors must be on devices[0]’)
What is the good way of doing this?. For the moment what I do is put model1 in gpu 0 and model2 in gpu 1 and transfer the output from model1 to gpu1.
I do not want to merge the models.
Thanks in advance.