s_n:

So if I want to run Mynet2 on multiple devices. I will do torch.nn.Dataparallel(Mynet2). Do I also need to take Mynet1 to multiple devices as in do torch.nn.Dataparallel(Mynet1) and then send it to Mynet2?
Or it will automatically be on multiple devices since I take Mynet2 to multiple devices.


nn.DataParallel will automatically create the model replica with all internal submodules, parameters, and buffers on the specified devices, so you can wrap the main model only into nn.DataParallel.



 s_n:

Is this because the intermediate variable that I have inside my custom conv2d class are on single device?
If so how can I take this intermediate variable to multiple devices?
Should I define them as model parameters using torch.nn.Parameter?


It depends, how you’ve created these tensors.
All registered nn.Parameters and buffers (via self.register_buffer('name', torch.randn(1))) will be pushed automatically to the corresponding device.
However, if you have created a plain tensor and pushed it to a device via to('cuda'), you would either have to use the device argument of another parameter or register it as an nn.Parameter (if it needs gradients) of buffer.