My Pytorch distributed code (similar to ImageNet example, but without multiprocessing) was working on distributed nodes, using the GLOO backend (Python 3.7, Pytorch 1.0.1). There was some system updates, and now all of a sudden I have intermittent success with the code, with the init_process_group throwing various errors often. I can usually get it work on 2 nodes (4 GPU’s each node), but when I try > 2 compute nodes, almost always there is an error thrown in the init_process_group call (see below for some of the errors).
I put “python -X faulthandler”, and it tells me for the Segmentation faults, that the error is in ProcessGroupGloo in distributed_c10d.py (line 360).
I’m a bit at a loss of how to debug this, and what to check. I have reinstalled the Pytorch packages after the system updates, and tried going to Python 3.6 also, but no luck. I haven’t tried compiling from source yet. I started looking at the GLOO repo, and saw there are some tests, not sure if they would help pinpoint the cause.
Error #1:

srun: error: tiger-i21g6: task 0: Segmentation fault

Error #2:

Traceback (most recent call last):
File “disruptcnn/main.py”, line 595, in <module>
main()
File “disruptcnn/main.py”, line 161, in main
main_worker(args.gpu, ngpus_per_node, args)
File “disruptcnn/main.py”, line 177, in main_worker
world_size=args.world_size, rank=args.rank)
File “~/.conda/envs/python3a/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 360, in init_process_group
timeout=timeout)
RuntimeError: read: Bad address
terminate called after throwing an instance of 'std::system_error’
what():  read: Bad address

Error #3:

*** Error in `~/.conda/envs/python36/bin/python’: free(): invalid next size (fast): 0x000055b8f75ea9b0 ***