Well, because you have always have dependencies on the previous time step in recurrent networks, there isn’t a magic way to fully untangle t-1 from t.
There are recurrent models that restructure the computation to try to do as much as possible on GPU.
Such as the simple recurrent unit.

  
      
      GitHub
  
  
    

taolei87/sru 29

sru - Training RNNs as Fast as CNNs (https://arxiv.org/abs/1709.02755)


  
  
    
    
  
  


It does speed up computation, but it’s also a bit more buggy   Once I get my seq2seq model working with standard LSTMCell, I’m going to try it again.