If the last line is used to ensure all processes finish reading, why does not it directly follow  ddp_model.load_state_dict ?

Good catch! The original reason for adding that barrier is to guard the file deletion below:
    if rank == 0:
        os.remove(CHECKPOINT_PATH)

But looking at it again, this barrier is not necessary. Because the backward() on the DDP model is also a synchronization point as it calls AllReduce internally. Let me remove that.

For each iteration, do we need to call  dist.barrier() ?

No. Two common reasons of using a barrier are

to avoid AllReduce timeout caused by skewed workloads across DDP processes
code after barrier() on rank A depends on the completion of code before barrier() on node B.

If none of the bot is a concern in your use case, then barrier shouldn’t be necessary.