At first glance, the code you link assembles the LSTM from typical NN-functions like linear. Autograd will provide gradients, but they will be slower than a custom-made gradient – in the PyTorch C++ extension tutorial, you get a ~20% performance boost from moving from Autograd components to a custom backward for the LLTM model (it advertises a 30% speedup for moving from Python + automatic gradients to C++ + custom backward, my experience is that ~10%-pts are for moving to C++ and ~20%-pts are for the backward).
You wouldn’t need to change weight update and optimizer steps.
Best regards
Thomas