Could you provide a link to (or post) your Keras model here, so that we could compare both training codes, please?
Just skimming through the code, it seems the learning rate is quite high, but since I cannot compare it to your Keras implementation, that’s just a wild guess.