Thanks for your help! I want to ask one more thing: how about the backward of bias?
The bias should also be updated in the backward pass. In former implementation, PyTorch used cudnn_convolution_backward_bias , but now they deleted.