Could you rerun the code with CUDA_LAUNCH_BLOCKING=1 python script.py args and post the stack trace again, please?
Since CUDA operations are asynchronous, the current stack trace might point to a wrong location.
Was this device working fine before and did you update anything in your setup?
Also, you could try to use anomaly detection to check, if something in your model goes wrong, although I wouldn’t expect a launch failure in this case.