@GeoffreyChen777
Really thank you! Briefly speaking, Dataparallel firstly sum and then average, because each GPU calculates part of the 128 batch, then they must send loss to master GPU to update parameters. Distributed Dataparallel has independent model and parameters in each GPU, so the loss calculated on one GPU has been the average of batch_size 32, then we should average loss between different GPU. That is in Distributed Dataparallel , we do average inner model and then average between different GPU. If I do not understand correctly, please let me known, thank you again!