So yeah, unless I’m blind, I don’t see any docs saying that don’t use --rdzv_endpoint=$ENDPOINT on the actual endpoint node, now that’s out of the way I’m kind of a step closer to things working. However now either the host or the slave complains ValueError: host not found: Temporary failure in name resolution. I have tried launch in two different ways by either using --network host or exposing the default port 29400. Either the host or the slave crashes with the error below.
Traceback (most recent call last):
  File "train_detached.py", line 19, in <module>
    run_training()
  File "train_detached.py", line 11, in run_training
    config_dict = initialise_training_modules()
  File "/workspace/nnet_training/training_init.py", line 208, in initialise_training_modules
    dist.init_process_group(backend=args.backend, init_method=args.dist_method)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 559, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 201, in _env_rendezvous_handler
    store = TCPStore(  # type: ignore[call-arg]
ValueError: host not found: Temporary failure in name resolution

When I use   docker run ... --network host ... the first rendezvous is below where both the master and slave agree. However the computer name of the master_addr is that of the slave which is at 130.194.133.55??? You will note in the slave launch args it has the intended endpoint and the master launch args has no endpoint.
[INFO] 2021-08-02 12:23:42,863 api: [default] Rendezvous complete for workers. Result:
  restart_count=0
  master_addr=mu00186362
  master_port=43937
  group_rank=1
  group_world_size=2
  local_ranks=[0, 1]
  role_ranks=[2, 3]
  global_ranks=[2, 3]
  role_world_sizes=[4, 4]
  global_world_sizes=[4, 4]

If I instead expose the default port only docker run ... -p 29400:29400 ..., the first rendezvous is agreed upon as
[INFO] 2021-08-02 12:38:32,335 api: [default] Rendezvous complete for workers. Result:
  restart_count=0
  master_addr=4ebda5ed7386
  master_port=52585
  group_rank=1
  group_world_size=2
  local_ranks=[0, 1]
  role_ranks=[2, 3]
  global_ranks=[2, 3]
  role_world_sizes=[4, 4]
  global_world_sizes=[4, 4]

Where the slave has the launch config
[INFO] 2021-08-02 12:23:40,064 api: Starting elastic_operator with launch configs:
  entrypoint       : train_detached.py
  min_nodes        : 2
  max_nodes        : 2
  nproc_per_node   : 2
  run_id           : 1234
  rdzv_backend     : c10d
  rdzv_endpoint    : 130.194.129.201
  rdzv_configs     : {'timeout': 900}
  max_restarts     : 3
  monitor_interval : 5
  log_dir          : None
  metrics_cfg      : {}

and the master has the launch config
[INFO] 2021-08-02 12:23:41,712 api: Starting elastic_operator with launch configs:
  entrypoint       : train_detached.py
  min_nodes        : 2
  max_nodes        : 2
  nproc_per_node   : 2
  run_id           : 1234
  rdzv_backend     : c10d
  rdzv_endpoint    : 
  rdzv_configs     : {'timeout': 900}
  max_restarts     : 3
  monitor_interval : 5
  log_dir          : None
  metrics_cfg      : {}