@mrshenli Great. Thank you so much for the explanations!
I hope @fmassa could join the discussion and clarify the points for which you mentioned him earlier, especially the one related to evaluate(). I tested the code and it seems that this function is called only once across the processes.
The training loop looks like this:
for epoch in range(args.start_epoch, args.epochs):
    if args.distributed:
        train_sampler.set_epoch(epoch)
    train_one_epoch(...)
    lr_scheduler.step()
    evaluate(model, criterion, data_loader_test, device=device)

where the evaluation function looks like:
def evaluate(model, criterion, data_loader, device, print_freq=100):
    model.eval()
    metric_logger = utils.MetricLogger(delimiter="  ")
    header = 'Test:'
    with torch.no_grad():
        for image, target in metric_logger.log_every(data_loader, print_freq, header):
            ...
    # gather the stats from all processes
    metric_logger.synchronize_between_processes()

    print(' * Acc@1 {top1.global_avg:.3f} Acc@5 {top5.global_avg:.3f}'
          .format(top1=metric_logger.acc1, top5=metric_logger.acc5))
    return metric_logger.acc1.global_avg

This function has some print() to display the accuracy. In my experiment, that string is only displayed once, which means the function is called only once. Why? There is no is_main_process() check. Why isn’t this function called on all processes? I’m confused…