Great to hear that you found the problem!
The reaming difference can be different initialization, but also due to different train/test split or learning rate schedule…
have you gone through the list of @rwightman?

ensure image pre-processing is exactly the same (aside from color channel transpose)
ensure batch sizes are the same
ensure test/val splits are the same
ensure model mode is being switched between train() and evail() and back otherwise dropout remains active during eval
ensure any learning rate schedules are the same