Thanks for the update.
The workaround is invalid, as you would call the underlying model on the default device and would skip the nn.DataParallel wrapper.
The device mismatch is raised, since torch.distributions do not have a to() method and are not registered as modules.
Here is a minimal code snippet to reproduce this issue:
class MyModel(nn.Module):
    def __init__(self, prior):
        super(MyModel, self).__init__()
        self.prior = prior

    def forward(self, x):
        y = self.prior.log_prob(x)
        return y

img_size = 1
device = 'cuda:0'
prior = torch.distributions.MultivariateNormal(torch.zeros(img_size).to(device), torch.eye(img_size).to(device))

model = MyModel(prior)
print(model.prior)
model.to('cuda:1')
print(model.prior)

x = torch.randn(1).to('cuda:1')
out = model(x)
print(out)

As you can see, even after calling model.to('cuda:1') model.prior is still on cuda:0 and the forward pass will raise the same error.
It also seems to be a known issue so feel free to add your use case to it.
I wanted to suggest the same workaround from this post, i.e. to register the loc and scale as buffers and recreate the distribution in the forward method.