Gotcha! Thanks for the tips!
After reading more into how things are implemented, I think now I know what is going on.
We are integrating pytorch into a system where we spawn multiple processes, and the training happens within one of the sub-processes on one single machine.  pytorch package imports get leaked into multiple processes including the parent, thus for each time “Using backend: pytorch” is shown, there is one c10d/rendezvous setup/registration happening automatically. Therefore the distributed modules got confused and in consequence connection issues.
What is the easy way to tell a process when we spawning it, saying “ok, you are (or not) the one to initialize distributed modules”?   I guess I can hack into pytorch for now, but I am not sure about side-effects doing so.  If there is already a mechanism, that would be awesome.
Now I guess I get the reason why this module was originally provided only as a script.
Thank you so much everyone!