Thanks for the code so far.
Could you also post the code you are using to initialize the model, optimizer, and scheduler?
Also, could you try to run the code on the CPU only and check, if you see the same error?
If not, could you rerun the GPU code using CUDA_LAUNCH_BLOCKING=1 python script.py args and post the stack trace again?