I think I may have found a fix.
After reading
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars 2
I tried setting
CUDA_LAUNCH_BLOCKING =1
And I completed my training without any issue! Hope I can help anyone having the same issue.