Solved by blackberry in post #2 
              
              
                This problem is inherently serial if you use LSTM/GRU. 
There is an alternative which is claimed to be much faster: SRU. 
Training RNNs as Fast as CNNs (https://arxiv.org/abs/1709.02755) 
PyTorch code: https://github.com/taolei87/sru