Did you check the initialization and the loss, too?
I think keras uses Xavier Glorot’s method by default for some layers.
You probably could extend your comparison to the gradient and possible regularizer and then look at the optimizer step itself.
The good news is that you can probably just “bisect” the backward pass if you find that the gradients differ.
Best regards
Thomas
(who is also trying to stare down a model where he cannot make sense of the apparent training deficiency)