Hi,
I have a class for my model that uses some other model in it. For example:
class Mynet1(nn.Module):
def init(self):
super(NN_model, self).init()
self.fc1 = nn.Linear(4160, 500)
class Mynet2(nn.Module):
def init(self, Mynet):
super(NN_model, self).init()
self.fc1 = nn.Linear(4160, 500)
self.layer = Mynet(…)
mynet1=Mynet1()
model = torch.nn.Dataparallel(Mynet2(mynet1))
model.cuda()
So if I want to run Mynet2 on multiple devices. I will do torch.nn.Dataparallel(Mynet2). Do I also need to take Mynet1 to multiple devices as in do torch.nn.Dataparallel(Mynet1) and then send it to Mynet2?
Or it will automatically be on multiple devices since I take Mynet2 to multiple devices.
One more question related to dataparallel. Let’s say I have my custom conv2d class that uses torch variables like torch.mul, torch.sub etc. So when I was running this code on a single device I used to do “.to(device)” for all the intermediate variable in my custom conv2d class. So now when I run a model using this custom conv2d class on multiple devices using dataparallel I keep on getting this error: “RuntimeError: Expected tensor for argument #1 ‘input’ to have the same device as tensor for argument #2 ‘weight’; but device 0 does not equal 1 (while checking arguments for cudnn_batch_norm)”.
I have taken my model that uses this custom conv2d class to multiple devices using torch.nn.dataparallel. So I don’t understand why I keep on getting this error. Is this because the intermediate variable that I have inside my custom conv2d class are on single device?
If so how can I take this intermediate variable to multiple devices?
Should I define them as model parameters using torch.nn.Parameter?
Thanks