Thanks again for the code snippet as well as the great debugging!
I was able to narrow it down to a sync issue using this “minimal” code snippet:
x = torch.randn(1, 2, 4, 4, device='cuda')
v = 2
m = torch.randn(1024, 1024, device='cuda')

res = []
for _ in range(10):
    psv_tgt = x[:, 0:1].repeat([1, v, 1, 1]).reshape([-1, 4, 4])
    res.append(torch.inverse(psv_tgt))
    for _ in range(5):
        torch.matmul(m, m)

res = torch.stack(res)
print(res)

The pseudo workload is necessary to show the sync issue, otherwise you’ll get valid outputs.
The issue is created by a wrong sync pattern in apply_batched_inverse_lib, which was already fixed in this PR 2.
It’s good to know that the code works fine with 11.2, but I assume you’ve built PyTorch from source or installed the 1.8rc for it (which would thus already include the fix).
I’ve also verified that your code works fine using the nightly binary with CUDA10.1, so I’m quite sure you were also hitting the aforementioned issue.