The stacktrace points to an invalid indexing operation.
You could try to rerun your script with CUDA_LAUNCH_BLOCKING=1, which should print the operation using the invalid index in the stacktrace directly. If that doesn’t work, try to use this env variable with a single GPU or run the script on the CPU to isolate the issue further.