class Net(nn.Module):
    ...

net = Net()
input = torch.Tensor()
output = net(input.cuda())
loss = output - label
loss.backward()
optimizer.step()

If I give a Tensor.cuda() to the net, will be the optim of the Net in the GPU?