cuDNN provides very fast primitives, but without access to the internals.
If you were to implement the RNN in plain pytorch (see tutorials & example scripts) pytorch has access to the internal buffers which are needed for using the gradients in the objective.