So I discussed optimization in general a bit in a talk I gave last december 10. Whenever I looked at it, the difference between Python and “same thing in C” was about 10%. That is obviously not true universally, but it seems to be a realistic ballpark number in many cases. The true gain is

when you “fuse” (pointwise, mainly) operations, i.e. have them in a single cuda kernel, but that is rather time-consuming to implement,
when you minimize the number of operations (e.g. by applying weight_ih on all inputs in one go) - this is what the premul variants in the PyTorch benchmark repository does.

To me this hints a bit to the for loop itself in Python or C++ not being the crucial bit, but rather whether you do things in parallel before and after and whether you’re efficient inside the for loop.
NVidia has a great writeup about how they optimized LSTM 43.
In the current PyTorch master, scripted LSTM is of similar speed than the ATen implementation.
Maybe you can get best results on current PyTorch by manually defining a backward in a , but using @torch.jit.scripted functions for both forward and backward. I must admit I haven’t looked into this too much, though, because I mainly approached things from a can we improve PyTorch to make this better perspective.
I’ll probably try to write a few things about my new LSTM optimizations 37 - they actually get a bit better than advertised there to about 1.33x CuDNN wall clock time for the jit-premul LSTM backward - not too bad for less than 200 lines of code diff.
Best regards
Thomas