Two observations.

Given that training loss decreases, but validation loss does not hints that you are overfitting.
Unless Keras model is trained on CPU and Pytorch model on GPU, it does not make sense that Pytorch is so much faster.

Looking these observations together I would say that there is some problem with training data or data loader. Perhaps only a single example is used not all the data? I would start from debugging or printing out what is given as input to the model.