@hhxx

(1) In  demo_checkpoint , all processes need to be synchronized by loading the same checkponts saved by process-0. If it is necessary when we train model with DistributedDataParallel.?

No, this is not necessary. This is only useful when your training job take very long and can crash in the middle. You can then use the checkpoint to recover instead of starting over from scratch.

(2) If it is necessary to synchronize model across multi nodes and gpus by loading the same checkpoint, how can node-B load checkpoints saved in node-A?
I don’t know if I understand  demo_checkpoint  in a right way. Could you please help answer this question? Thanks !

The recovery scheme should be application-dependent. That tutorial demonstrates single-machine multi-GPU DDP with checkpointing. So, all DDP processes can read from the same file. If you need checkpoint, and if your training spans multiple machines, you can load it from rank0 and then broadcast it to other ranks using torch.distributed.braodcast.
BTW, we probably should call that tutorial “intermediate”, and use this one 44 as a starting example.