Thanks for your question!
In the code you linked, a DistributedSampler is used to ensure that each GPU is fed different data which is the typical scenario in DDP training.

or that the accuracy on GPU0 is somehow representative of entire accuracy
In general, we take the model on an arbitrary GPU and use that to measure the accuracy, in this case, GPU 0. The actual model is the same across all GPUs though since gradients are synchronized.


The accuracy() function divides by the batch_size , so returning acc_meter.avg divides by the batch_size again. The return value should be acc_meter.sum .

Dones’t acc_meter.avg take the average accuracy across all the updates? Or does it actually also divide by the batch size? If so, this seems incorrect and an issue should be filed in the pytorch/examples repo.
Your approach if you would like to get the accuracy across all ranks seems correct. In general, maybe it would be good to have something like DistributedAverageMeter. Although in most training scenarios I’ve come across, it’s enough to evaluate the model on only one rank during training and then the entire dataset during eval. This of course assumes the data is shuffled properly and one rank doesn’t get all positive/negative samples, for example.