s_n:

So I won’t even have to take mynet1 instance to the device before passing it as an argument to Mynet2?


That is correct. Your mynet1 would act as a “standard” module, e.g. similar to nn.Linear or nn.Conv2d, so you can just pass it into the parent model (Mynet2) and call to() or cuda() on the parent.



 s_n:

Can you please explain a bit more on what you mean by


All nn.Modules, nn.Parameters and buffers will be properly moved to the device, if to() or cuda()/cpu() is called on the parent device, while plain tensors will not. This code snippet demonstrates the behavior:
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.module = nn.Linear(1, 1)
        self.param = nn.Parameter(torch.randn(1, 1))
        self.register_buffer('buf', torch.randn(1, 1))
        self.ten = torch.randn(1, 1)

    def forward(self, x):
        return self.module(x) + self.param + self.buf + self.ten
        
model = MyModel()
print(model.module.weight.device)
print(model.param.device)
print(model.buf.device)
print(model.ten.device)
out = model(torch.randn(1, 1)) # works
> cpu
cpu
cpu
cpu

model.to('cuda')
print(model.module.weight.device)
print(model.param.device)
print(model.buf.device)
print(model.ten.device)
out = model(torch.randn(1, 1).to('cuda')) # fails, as self.ten is still on the CPU
> cuda:0
cuda:0
cuda:0
cpu
RuntimeError: expected device cuda:0 but got device cpu




 s_n:

One solution to this problem I feel would be to define all my intermediate variables as nn.Parameters and make require_grad=False. Also, I can define them self.register_buffer and make grad=False. Will both solutions be similar?


If you don’t want to train this tensor, use a buffer. Using a parameter with requires_grad=False would also work, but a buffer would be the cleaner approach.