Are you using a custom extension, or did you just run cuda-memcheck on the complete model?
Due to the asynchronous execution of CUDA code, you could rerun the code with CUDA_LAUNCH_BLOCKING=1 python script.py args and post the stack trace here, so that we could have a look.