One difference would be the explicit initialization of the first conv layers with he_uniform in Keras and xavier_uniform (also knows as Glorot) in PyTorch, so you might want to use the same parameter initializations.