CUDA operations are asynchronous so the illegal memory access might be created in the model and the next CUDA operation would run into this error and raise it.
Could you rerun your script with CUDA_LAUNCH_BLOCKING=1 python scripy.py args and post the stack trace here, please?