Solved by ptrblck in post #2 
              
              
                If self.log_alpha is trainable (requires gradients), you should wrap it in an nn.Parameter, which will also make sure to move the tensor to the device.