I put together a “Minimal Working Example” to reproduce my results here: GitHub - inzouzouwetrust/resnet50-imagenet-baseline: Image classification baseline using ResNet50 on ImageNet 6.
Let me know if that helps!
UPDATE:
I may have found the root cause for the test performance discrepancy.
In my implementation, I happened to use a total batch size equal to 1024 as each process used a batch size of 256 and 4 processes were spawned. In the official PyTorch example, each process use bs=256/N where N is the number of processes (4 here). It means that I had to either adjust the batch size (i.e. set it to 64 per process) or tune the learning rate accordingly (i.e. set it higher initially, e.g. 0.4 when using 256 images per process).
I will keep this post updated once I get the final results.