@shicai @fmassa thanks for the suggestions!
I think it makes sense that the two reasons make the difference:

different data augmentations or image transforms
“hard” fine-tuning and “soft” fine-tuning

From my dataset experiment result, I think maybe “soft” fine-tuning is better than “hard” fine-tuning.
@apaszke from my dataset experiment result, maybe these transforms have effect indeed