Following this thread because I’m also interested in doing modifications to the original LSTM and benchmarked with my results on reddit 154. For example:

Layer normalization
Multiplicative Integration
Additive Residual Connections between Layers

Is there anyway to maintain the speed that the cudnn LSTM has while adding these new features in? I don’t know if autograd has been refactored yet.