So the stock LSTM will use CuDNN which has been heavily optimized but is inflexible.
But the JIT can make custom RNNs almost as fast: Some of this is discussed in the LSTM with JIT blogpost and we also have optimizations for the backward.
You can look at the benchmark code to get ideas how to use them.
Best regards
Thomas