Based on the error message you are running into a device assert so you could either run the script via CUDA_LAUNCH_BLOCKING=1 python script.py args from the terminal to isolate the failing operation or run it on the CPU.