I’ve gone down the rabbit hole of differences between Keras and PyTorch and learned about how each package initializes weights differently. I’ve changed my code to use the same initialization methods and got MUCH better results.
Anyone who gets different results between the packages when their model looks correct should try this.