If the latter, you can use that too, but, as the doc says, the lower triangular matrix needs to have positive diagonal elements:
import torch
from torch.distributions.multivariate_normal import MultivariateNormal
from torch.distributions import kl_divergence

p = MultivariateNormal(torch.zeros(5), torch.eye(5))
print('p.sample()', p.sample())

q_mean = torch.randn(1, 5)
print('q_mean', q_mean)
L = torch.tril(torch.randn(5,5).abs() + 0.1)
print('L', L)
q = MultivariateNormal(q_mean, scale_tril=L)
print('q.sample()', q.sample())

kl = kl_divergence(p,q)
print('kl', kl)

output:
p.sample() tensor([ 0.6368, -0.0909,  0.0822,  0.1273,  1.1169])
q_mean tensor([[ 1.1417, -0.2070, -1.2527,  0.4430, -1.5522]])
L tensor([[ 0.6695,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 1.4359,  0.7194,  0.0000,  0.0000,  0.0000],
        [ 1.4802,  0.7240,  0.3760,  0.0000,  0.0000],
        [ 1.2003,  0.3702,  0.6561,  1.0024,  0.0000],
        [ 1.4357,  0.4746,  0.1379,  1.3532,  1.9632]])
q.sample() tensor([[ 1.4201, -0.7672, -1.6811,  1.1081, -1.4706]])
kl tensor([ 30.6731])

(Yes, I’m learning as I go along too. That’s kind of the point of why answering questions is fun  )