Unfortunately, double backward for cudnn rnn is not supported, there’s an upstream issue tracking this https://github.com/pytorch/pytorch/issues/5261 157. Recommended way of doing what you want to do is writing custom rnns with torch script https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/ 155