Hello!
I am University Student who studying in Speech Recognition in Korea.
Me and my team is working on a project now, but I got a problem.
When my model starts training,
“cuDNN error : CUDNN_STATUS_EXECUTION_FAILED”
comes out at random batch and kills the training sequence.
Not because it dies at exactly same batch, I don’t know what occurs this error.
I asked for advice in Korea Pytorch Forum, and many of them replied for me to upgrade CUDA version. Is that the only option I can do to fix this situation?
Here is settings and tools I used:
device               : GTX 1080 Ti
CUDA version   : 9.2
cuDNN version  : 7.6.4
torch version     : 1.3.1
Command used to install Pytorch :
conda install pytorch torchvision cudatoolkit=9.2 -c pytorch -c defaults -c numba/label/dev
It will be lovely if someone’s gonna help me.
Thanks! 

cuDNN_error1440×1080 869 KB