Can you please try to repro the issue directly on the host?
For example, you can use the following commands:
pip install torch==1.9.0

#simple.py
print("hi")

# run on host:
python -m torch.distributed.run --rdzv_id 555 --rdzv_backend c10d --rdzv_endpoint IP_OF_MACHINE_0:29400 --nnodes 2 --nproc_per_node 2 simple.py


If you were able to repro the issue, can you do the following:
Run python -m torch.distributed.run --rdzv_id 555 --rdzv_backend c10d --rdzv_endpoint IP_OF_MACHINE_0:29400 --nnodes 2 --nproc_per_node 2 simple.py on training_machine0, then on the second host use the following cmd:
traceroute -T -p 29400 10.100.4.219 
to check whether  another host can access training_machine0.
Also, if you want faster iteration on this issue, you can DM me in slack: csivanou at gmail dot com