Solved by mrshenli in post #2 
              
              
                If the last line is used to ensure all processes finish reading, why does not it directly follow  ddp_model.load_state_dict ? 

Good catch! The original reason for adding that barrier is to guard the file deletion below: 
    if rank == 0:
        os.remove(CHECKPOINT_PATH)

But looking at it again…