Well, the script has a resume code as follows:

Not 100% sure about how this example would be used. @fmassa would know more.  Given the code, it looks the resume mode is designed for starting from pre-trained models or resume from crash. In these cases, the checkpoint file is ready before launching the script, so it should be fine.

you saying that the  demo_checkpoint()  example given in the tutorial 4 handles a different scenario

It is not targeting any specific use case, just wanted to make sure the example code can run as is. The main information it tries to convey is that applications need to make sure checkpoints are ready before loading them. We previous saw users running into weird errors caused by reading too soon.

if training starts from random parameters, you might want to make sure that all DDP processes use the same initial values. Otherwise, global gradient synchronizes will not make sense.

DDP handles this by broadcasting model weights from rank 0 to others at construction time. However, if the application modified model weights after constructing DDP and if that resulted in inconsistent weights across processes, DDP won’t be able to recover, as the broadcast only happens once in ctor.