Thank you Shen Li for your detail explanation. It is very helpful, and now I understand what’s going on in DP and DDP. I modified your codes to see the order of data so that I can make sure that the output is correctly compared to corresponding labels in loss function.
import torch
import torch.nn as nn

device = "cuda:0"

class Model(nn.Module):

    def __init__(self):
        super(Model, self).__init__()
    
    # forward() outputs the input as it is. 
    def forward(self, input):
        output = input
        print("per-GPU output ", output)
        return output

model = Model()
model = nn.DataParallel(model)
model.to(device)

# input is a sequence of integer in 2D shape.
input = torch.arange(20 * 5).reshape(20, 5)
input = input.to(device)
print("total input ", input)
output = model(input)
print("total output ", output)

I was not sure about the “tag” that you pointed out, but I added “distributed” to “Categories”.
I still have a related question about DDP.
In my understanding, the gradient is a vector that points a direction where the loss increases the most. I learned from your explanation that we don’t have the “total” loss until we “gather”, “all_gather”, or “all_reduce” the loss computed in each GPU. If we use a loss in each process instead of total loss to compute each gradient and average all the gradients, will it be a correct “total” gradient of the total loss?
In other words, I wonder if it is mathematically correct that averaging all gradients that increase each of respective loss produces a total gradient that increases the averaged loss.
If it is not correct, I think it means that we need to do all_reduce of the loss before we do loss.backward in order to hand total loss information to each process for computing correct gradients. Is my thinking correct?
Thank you again for your kind assistance.