Once the model architecture is fixed, I would recommend to check the weight initializations, as they might be different between both frameworks.
Some PyTorch init methods were directly taken from Torch7 (Lua), which might not be the state of the art anymore. I don’t know, if Keras changes the default inits regularly.