Thank you for you reply.
You are right. But I have found these points too.
The bn momentum and bn epsilon are different.
I also write my own padding like that in TF before conv layer with stride=2.
I have test the two models with the same weights and the same input tensor.
I follow each model layer by layer in the forward pass. All tensors are the same.
For the backward pass, I only checked one step of update and find that the first conv layer after update are still the same.
Until now, I have also tested to use the pytorch dataset to feed data into tensorflow model (a wired thing). After training, the evaluation result is good. So dataset is not the culprit.
Besides, I use SGD with momentum =0.9 and weight decay factor=2e-4.
I really don’t know what else hyperparameters I can set.