I was doing something like this:

    self.L_1 = Parameter(torch.randn(dim, dim), requires_grad=True)
    self.L_1.data = torch.tril(self.L_1.data)
    self.log_diag = Parameter(torch.diag(self.L_1.data), requires_grad=True)
    self.log_diag.data = torch.exp(self.log_diag.data)
    self.mask = Parameter(torch.diag(torch.ones_like(self.log_diag.data)))
    self.L = Parameter(self.mask.data * torch.diag(self.log_diag.data) + (1. - self.mask.data) * self.L_1.data, requires_grad=True).cuda()