@mrshenli Thanks for your prompt reply!

IIUC, the torchvision example only saves the checkpoint to file in epoch but is not reading from it unless it is recovering from a crash?

Well, the script has a resume code as follows:
if args.resume:
    checkpoint = torch.load(args.resume, map_location='cpu')
    model_without_ddp.load_state_dict(checkpoint['model'])
    optimizer.load_state_dict(checkpoint['optimizer'])
    lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])
    args.start_epoch = checkpoint['epoch'] + 1

so I guess it does what you’ve described, which is a usual scenario. But then are you saying that the demo_checkpoint() example given in the tutorial 1 handles a different scenario (other than resuming a training)?

The gradients are synchronized in DDP backward using AllReduce operations. So, there is no need to add another barrier here to do that. As soon as  loss.backward()  returns, the local gradients should be representing the global average.

Thanks! This has cleared up a lot of things for me. I guess the gradients are averaged? In this case, it seems that the learning rate should be scaled up by the number of GPUs.
Besides, the tutorial also notes that

if training starts from random parameters, you might want to make sure that all DDP processes use the same initial values. Otherwise, global gradient synchronizes will not make sense.

but I don’t see this being taken into account anywhere in the reference script.