Hi,
This error is because some function failed on the GPU.
To know which function was problematic, you can use CUDA_LAUNCH_BLOCKING=1.
This error almost always is caused by a label out of range given to your loss function. Make sure that all your labels are correct and passed properly to the loss function.