I found out that the issue was something else. When CUDA_LAUNCH_BLOCKING=1 flag is not used with python script, it showed incorrect error CUDNN_STATUS_MAPPING_ERROR (in my case, it said batchnorm failed). However, when the flag is used, I get “RuntimeError: CUDA error: device-side assert triggered” at line X. I manually checked that there was a index out of bounds problem at line X. Correcting it solved the problem.
The real issue is that the code never tells that the problem is “index out of bounds”, but instead says CUDNN_STATUS_MAPPING_ERROR or “RuntimeError: CUDA error: device-side assert triggered”, depending on if you are using CUDA_LAUNCH_BLOCKING=1 or not.