Hi Siyu, they were mostly implementation differences in the train and test pipeline which resulted in such issues. Spotting these were difficult but what pointed me towards checking some old code was a simple experiment:

I used the script given above to transfer weights from a Keras model to a PyTorch model.
Then from the PyTorch model back to the Keras model.
I continued this activity for more than 10000 steps via simple for loop.

If my conjecture that that we lose accuracy while transferring weights is correct, this whole moving weights from one platform to another would leave the model in almost a random state. But, the result was that the only accuracy I lost was in the first iteration. So that made me look in my implementation of the testing pipeline where I found my bug.
No, I didn’t use MMdnn to do my conversion as my model wasn’t that complex and the naming scheme made it easier to write up a script to transfer weights across the platform. All one requires is to have exactly the same model in the two platforms. The difference comes in what orientation (height of the image first or the width for convolutions?) the two platforms store the tensors. And thus, exploiting the naming scheme and tensor orientations I created a script to transfer weights: Transferring weights from Keras to PyTorch