the program is first run dist.init_process_group() and then sync between works by dist.barriers(). After these, it will create the data loader, model and then do the training. However, it crashes at dist.barrier() with random error message. Here is a few error. Most of the cases, the job can run well but this happens now and then.


out of memory message
image2190×911 76.4 KB


NCCL error
image1926×811 61.6 KB


illegal access
image1896×828 63.8 KB