Since I am using multiple GPUs with nccl backend gather does not work so I tried using all_gather and below is a part of my code
code:
loss = criterion(output, targets)
tmp = [torch.empty_like(loss).cuda(rank) for _ in range(2)]
dist.all_gather(tmp, loss)
print(tmp[0].data.item())
loss.backward()
optimizer.step()
print(f'Model on GPU {rank}, Epoch: {epoch}--{i}/{len(dataloader)}],  Loss: {loss.data.item()}')

output:
0.9214748740196228
0.9214748740196228
Model on GPU 0, Epoch: 1--0/28125],  Loss: 0.9214748740196228,  Batch_time_Average - 4.840
Model on GPU 1, Epoch: 1--0/28125],  Loss: 0.9064291715621948,  Batch_time_Average - 4.781
0.7848501801490784
0.7848501801490784
Model on GPU 0, Epoch: 1--1/28125],  Loss: 0.7848501801490784,  Batch_time_Average - 6.893
Model on GPU 1, Epoch: 1--1/28125],  Loss: 0.6567432880401611,  Batch_time_Average - 6.798
0.7931838035583496
0.7931838035583496
Model on GPU 0, Epoch: 1--2/28125],  Loss: 0.7931838035583496,  Batch_time_Average - 8.924
Model on GPU 1, Epoch: 1--2/28125],  Loss: 0.825346052646637,  Batch_time_Average - 8.835
1.0175780057907104
1.0175780057907104
Model on GPU 0, Epoch: 1--3/28125],  Loss: 1.0175780057907104,  Batch_time_Average - 10.966
Model on GPU 1, Epoch: 1--3/28125],  Loss: 0.5258045196533203,  Batch_time_Average - 10.868

tmp is just printing loss value at GPU 0. I am using Categorical cross entropy. I am not sure if this is the required output.