Also seen similar issues. After init_process_group, i run dist.barrier before creating data loader, creating model, … It crashes with OOM at dist.barrier, which is quite strange. Normally i resubmit the job, and the new job might be allocated to some new nodes, and then it can run smoothly sometimes. But this problem happens almost every day. GPU memory is 0 before it hits init_process_group(). pytorch 1.6.