I tried saving the model just after the forward, the parameters of the saved models are the same.
def run(rank, size):
    ...
    for epoch in range(args.epochs):
        output = model(input)
        loss = criterion(output, target)

        torch.save({...}, f'checkpoint-process-{rank}-epoch-{epoch}.pth')

        ...

    ...

Still checking the sync of parameters and gradients in the source code.