The following code is runnable and bug-free. I am puzzled why the self.relu dosen’t need to be cast to GPU device while the self.net1 and self.net1 is cast to GPU.  In my considering,  the default device of self.relu should be CPU if there is no explict casting.  If this is correct, the output of self.net1 is on gpu, which mismatch the self.relu device type.  Why this is bug-free?
import torch
import torch.nn as nn
import torch.optim as optim


class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = torch.nn.Linear(10, 10).to('cuda:0')
        self.relu = torch.nn.ReLU()
        self.net2 = torch.nn.Linear(10, 5).to('cuda:0')

    def forward(self, x):
        x = self.relu(self.net1(x.to('cuda:0')))
        return self.net2(x.to('cuda:0'))


model = ToyModel()
loss_fn = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.001)

optimizer.zero_grad()
outputs = model(torch.randn(20, 10))
labels = torch.randn(20, 5).to('cuda:0')
loss_fn(outputs, labels).backward()
optimizer.step()