Solved by ptrblck in post #5 
              
              
                The bias is handled outside of the cudnn call, as it’s faster to let PyTorch add it explicitly and calculate its gradients. 
If you want to use the cudnn backward call, you can still use cudnnConvolutionBackwardFilter and cudnnConvolutionBackwardData manually.