Hi @StuChen,

The losses are different because different processes see different inputs with different labels and therefore produce different losses. If you’re looking for a global top1 or top5, you can use distributed primitives from torch.distributed to average them.
The model in each process will be optimized in the same way, because after calling loss.backward() the resulting gradients are identical across processes. In combination with the initial weights being identical, the resulting weights after optimization are also identical.