Actually, I do not know exactly why keras does not the issue. I assume that the only difference is the data preprocessing. E.g. in inception v3, in Keras the preprocessed range are [-1, 1], instead pytorch [-2.1179, 2.64]. In the training process, even though both have the same optimization strategy, they could end up with different parameters. Hence, they could have different effects on the issue. It is not a systematic error, just some randomness. That is my personal opinion!