Understood, I can see now how the asynchronicity can make it considerably more difficult to track down these sorts of errors. I was also unaware that linalg was so new, it makes sense issues like this would come up! I will definitely keep track of PyTorch changes on the MAGMA vs cuSOLVER front
In my case, do you have any advice for avoiding this particular exception? Alternatives to torch.linalg.inv, ways of manipulating the GPU cache that might help, delaying or syncing in certain parts of the program? I know you might not have an exact answer, but any potential avenues that might solve this illegal memory access would be great to get me started. It’d be great to get some version of this loss function working even if these issues can’t be resolved cleanly until later changes to torch.linalg