Hey, I’m running PyTorch 1.6 and I may have a pretty simple related repro for this, and the error is kind of weird. Basically if you run this
torch.matmul(torch.ones(3,1), torch.ones(1,3).to('cuda'))

It will work but  return some garbage the first time:
tensor([[0.0000e+00, 0.0000e+00, 6.9633e-36],
        [0.0000e+00, 6.9633e-36, 0.0000e+00],
        [6.9634e-36, 0.0000e+00, 0.0000e+00]])

, then the second time it will return RuntimeError: CUDA error: an illegal memory access was encountered.
Needless to say, sending the second ones tensor to the appropriate device fixes this. Running a similar piece of code was also returning an “out of memory” error, but I can’t repro this as easily.
Hope this helps.