Thanks for the information.
One difference between both implementations is the usage of BasicLSTMCell and MultiRNNCell in your TF code, while you are using an nn.LSTM in the PyTorch approach.
Maybe it would be easier to stick to the cell approach and use nn.LSTMCell instead?
This would also make is easier to compare the intermediate outputs between both models.