I’m using the file method, on a parallel file system:

    jobid = os.environ['SLURM_JOB_ID']
    world_size = int(os.environ['SLURM_NTASKS'])
    rank = int(os.environ['SLURM_PROCID'])
    dist.init_process_group(backend='gloo', init_method='file:///scratch/gpfs/me/main_'+jobid+'.txt',
                            world_size=world_size, rank=rank)


The errors don’t occur one on each machine, rather if I try running this various times, one of those errors will be thrown (but not both at the same time).