@ptrblck unsure if this clarifies anything, but after my loss accumulation loop (the one that includes the try…catch statement that I showed above) I added a call to torch.cuda.synchronize and received an identical error but with a stacktrace that pointed now to the call to synchronize:
Traceback (most recent call last):
  File "/panfs/roc/groups/13/suo-yang/dikem003/DimensionReductionNLE/auto_ode/AETrainingConditionNum.py", line 238, in <module>
    stiffness_loss = AE_stiffness_loss(latent_predictions) * EPOCH_SCALER
  File "/home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/panfs/roc/groups/13/suo-yang/dikem003/DimensionReductionNLE/auto_ode/CustomLossFunctions.py", line 226, in forward
    torch.cuda.synchronize(self.DEVICE)
  File "/home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/cuda/__init__.py", line 446, in synchronize
    return torch._C._cuda_synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Also, after exporting flags for not caching memory and the debugging flag
export PYTORCH_NO_CUDA_MEMORY_CACHING=1
export CUDA_LAUNCH_BLOCKING=1
I got a more helpful stacktrace
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: an illegal memory access was encountered
Exception raised from uncached_delete at /pytorch/c10/cuda/CUDACachingAllocator.cpp:1307 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f09bf565a22 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xfa5e (0x7f09bf7c5a5e in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x54 (0x7f09bf54f5a4 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0xe5e3e9 (0x7f09c08413e9 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x16b4842 (0x7f09c1097842 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #5: at::native::_linalg_inv_out_helper_cuda_lib(at::Tensor&, at::Tensor&, at::Tensor&) + 0x374 (0x7f09c109e244 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xf22ffa (0x7f09c0905ffa in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #7: at::_linalg_inv_out_helper_(at::Tensor&, at::Tensor&, at::Tensor&) + 0x133 (0x7f0a03625093 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x1001850 (0x7f0a02ee8850 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::native::linalg_inv_ex_out(at::Tensor const&, bool, at::Tensor&, at::Tensor&) + 0xc3 (0x7f0a02ee9103 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::linalg_inv_ex(at::Tensor const&, bool) + 0xe6 (0x7f0a02ee9366 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x1af7e40 (0x7f0a039dee40 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::redispatch::linalg_inv_ex(c10::DispatchKeySet, at::Tensor const&, bool) + 0xb1 (0x7f0a036fefc1 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x2f90d8b (0x7f0a04e77d8b in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x2f91223 (0x7f0a04e78223 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #15: at::linalg_inv_ex(at::Tensor const&, bool) + 0x11b (0x7f0a03546dbb in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #16: at::native::linalg_inv(at::Tensor const&) + 0x30 (0x7f0a02edfbf0 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #17: <unknown function> + 0x1b71fcc (0x7f0a03a58fcc in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #18: at::linalg_inv(at::Tensor const&) + 0x111 (0x7f0a03409271 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #19: <unknown function> + 0x8dd4a4 (0x7f0a15f734a4 in /home/suo-yang/dikem003/.conda/envs/torchcombust/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #41: __libc_start_main + 0xf5 (0x7f0a18b57555 in /lib64/libc.so.6)

/var/spool/slurmd/job5169384/slurm_script: line 27: 2939038 Aborted                 python3 -u AETrainingConditionNum.py -an=softplus_ratiolowersing_high -aa=softplus -ad=2 -rn=null -cs=500 -cw=0.1 --condlosstype=ratiolowersingular > lowersing2.out

I see mention of the linalg.inv function, which is the one I am catching the exception from (when a given matrix is not invertible). So seems to be some issue with trying to delete a cached file from the exception that may not exist?
Would the implication of this be that there is no way to currently catch an exception from a inversion operation while using CUDA