This problem is inherently serial if you use LSTM/GRU.
There is an alternative which is claimed to be much faster: SRU.
Training RNNs as Fast as CNNs (https://arxiv.org/abs/1709.02755 3)
PyTorch code: https://github.com/taolei87/sru 5