Big thanks to your reply! So hogwild is not a must for distributed data parallel, even if the models are declared in different processes, they can synchronize their parameters and gradients in some way. Is it right?
I tried saving the models at the end of these processes using the following code, but I found the parameters of the models are different. Does this mean that I made something wrong?
def run(rank, size):
    ...
    model = Net()
    ...
    for epoch in range(args.epochs):
        for batch_idx, (input, target) in enumerate(train_loader):
            output = model(input)
            loss = criterion(output, target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

    ...
    model.save({...}, f'checkpoint-process-{rank}.pth')