Yes, I am only seeing this illegal memory access exception raised once the singular matrix exception is caught. When I happen to weight this custom loss function lower in comparison to my reconstruction loss (this program is for a regularized autoencoder) my matrices never become singular and training proceeds as normal. I can tell since I print a small warning statement to the console when the exception is triggered
However, I have noticed that typically several exceptions will be caught before the memory access error is raised. Somewhat strange also that the two debug print statements mentioned earlier mitigated the issue (I even saw several warnings for the exception being raised within the log file, but training continued), I would think once I catch this exception the rest is deterministic and the CUDA exception would definitely be raised. Is there potentially some delay in communication between Python try-catch statements and the GPU? I’m not sure how far in advance CUDA tends to queue up operations
It’s also interesting that the function causing the issue is typically called only for batch inputs, since the for loop in the original code snippet is looping through each batch of inputs and calling torch.linalg.inv separately. In fact, each call to inverse which raises the RuntimeError that must be caught is only upon a single matrix of size [20, 20]. Is the cuSOLVER change a future switch or should this already be in effect for PyTorch 1.9?
Just want to take another opportunity to again say thank you for your help, and thank you for your patience in dealing with a primarily plain PyTorch user. Very very much appreciated!