Now this might have something to do with how the initializations have been implemented in PyTorch and Tensorflow. To experiment you can try to train a small mlp, and override all initializations with say xavier or even all 0.5 etc. Would love to dig further if even then the results differ…