No, you shouldn’t. You have to register the variable as a Parameter if it requires grad or as a buffer if it does not. I modified your toy example to show both cases:
class Toy(torch.nn.Module):
    def __init__(self):
        super(Toy, self).__init__()
        self.register_parameter("var_param", torch.nn.Parameter(torch.ones([], requires_grad=True)))

        self.register_buffer("var_buffer", torch.ones([], requires_grad=False))


    def forward(self, x):
        print(self.var_param.device)
        return self.var_param * x + self.var_buffer


toy = Toy()
print(toy.var_param.device) 
a = torch.ones([], device='cuda')
print(toy(a).device)  
toy.to('cuda')
print(toy(a).device)   
toy.cuda()
print(toy(a).device)  


Note: I changed the line a = torch.ones([], device='gpu') since "gpu" is not a valid device specifier but "cuda" is.

For further information have a a look at the documentation of register_buffer 15, register_parameter 6 and torch.nn.Parameter