hmm, this is weird. One possibility could be the print buffer from the main process wasn’t full at the beginning and then was only flushed on exit, leading to the message to be shown in the end. But I am not sure if this is case.
One reason for this behavior might be some DDP process hits OOM (or some other error) after a while and crashed, causing other DDP processes to hang. Did you try to do any try-except around DDP in main.py? If so, you might want to try https://pytorch.org/elastic 7, as try-except in one process could lead to DDP communication de-sync/hang/timeout.