if your RNN is written in terms of RNNCell/LSTMCell/GRUCell + for loops and it is a fairly small RNN, then maybe you are suffering from autograd overhead. We’re working on this. See: https://github.com/pytorch/pytorch/issues/2518#issuecomment-327835296 209