Yeah, cuDNN is a library that (among other things) lets you perform the forward or backward pass of an entire multi-layer, multi-timestep LSTM/GRU/RNN with one function call, dramatically reducing overhead relative to calling each constituent mathematical operation separately. LSTMCell is the module that defines the set of mathematical operations that makes up an LSTM, so using a series of LSTMCells (as done in AutogradRNN) will have the same result as CudnnRNN but be much slower)
One thing you can do is use cuDNN (by calling nn.LSTM rather than nn.LSTMCell) repeatedly with inputs of one timestep, but that will be only slightly faster at best than using LSTMCell.
Most of the time you should just use nn.LSTM and PyTorch will take care of things. If you want to do something unusual inside your LSTM, you won’t be able to use cuDNN anyway, so you should base your code on the code in AutogradRNN.