Moreover, since you are essentially using it as a RNN cell rather than complete RNN, you may want to look into LSTMCell. You may need to experiment to see which is faster, since LSTM is implemented in cudNN but has overhead.