Solved by justusschock in post #7 
              
              
                No, you shouldn’t. You have to register the variable as a Parameter if it requires grad or as a buffer if it does not. I modified your toy example to show both cases: 
class Toy(torch.nn.Module):
    def __init__(self):
        super(Toy, self).__init__()
        self.register_parameter("var_param",…