When you run this multiple times, do those runs use the same SLURM_JOB_ID? PyTorch makes an attempt to remove the file that is used for initialization at exit, but if any of the processes crashes it may stick around, and cause problems. You can “fix” this by force removing the file before starting a new run.