I cannot reproduce this issue on an RTX3090 using the PyTorch 1.7.1 conda binaries with cudatoolkit=11.0:
# tmp.py
from torch.distributions.multivariate_normal import MultivariateNormal
import torch
print(torch.cuda.get_device_name(0))
print(torch.__version__)
print(torch.version.cuda)
mean =  torch.ones(3).cuda()
scale = torch.ones(3).cuda()
mvn = MultivariateNormal(mean, torch.diag(scale))
print(mvn)

Output:
GeForce RTX 3090
1.7.1
11.0
MultivariateNormal(loc: torch.Size([3]), covariance_matrix: torch.Size([3, 3]))