Thanks for the information!
Comparing both approaches, I’ve noticed that the initializations are still a bit different.
Since I don’t have the dataset, could you change it to:
def initialize_weights(model):
    if type(model) in [nn.Linear]:
        nn.init.xavier_uniform_(model.weight)
        nn.init.zeros_(model.bias)
    elif type(model) in [nn.LSTM, nn.RNN, nn.GRU]:
        nn.init.orthogonal_(model.weight_hh_l0)
        nn.init.xavier_uniform_(model.weight_ih_l0)
        nn.init.zeros_(model.bias_hh_l0)
        nn.init.zeros_(model.bias_ih_l0)

and compare it again? (I’m not sure, if Keras uses a different gain for xavier_uniform_ as I couldn’t find anything in the code)
Let me know, if it helped a bit and I can have a further look.