Can you please try running your code with env variable CUDA_LAUNCH_BLOCKING=1 or with cuda-memcheck (cuda-memcheck python my_code.py)? Warning - to run with cuda-memcheck you have to trim down your code to bare minimum, otherwise it would be very slow. cublas error is likely happening because earlier cudnn call returned an error, but it’s hard to say what is happening without additional debug info.