Hi!

Currently torch doesn’t handle node failures well.
We rely on TCP reliability for node connectivity, but we also have some robustness mechanisms like rpc retries

You can also look into TorchElastic — PyTorch/Elastic master documentation 3 to handle node failures.