Thank @GlassyWing for your suggestion to work around this issue.
I was indeed curious about this bug, as switching to CPU or the concatenation would anyway slow down the code. In our project, the same error shows up from time to time in a very unpredictable way when torchdist.MultivariateNormal() is called. This is very frustrating as using CPU would be painfully slow, and nobody knows when the training will be stuck (usually good in the early stages). We see the same issue in multiple devices, including RTX 2080 Ti, RTX 3090, Tesla T100 in PyTorch 1.4.0, 1.6.0, and 1.8.0.