Okay Let me try to rephrase my question, gather has given me loss calculated by each GPU. My batch size for model being trained on 2 GPUs is 16 which means at a time 32 images will be processed. The gradient step that was taken was with respect to 32 images. torch.distributed.gather returns loss with respect to 16 images on each GPU in a list. I want to get the loss with respect to 32 images that were processed in one iteration in total.