Hello,
I am trying to deploy a crash-resilient distributed deployment in PyTorch. Assume that node of rank 0 controls the learning process. Now, if this node fails/crashes, I want node 1 to continue the job of node 0 and somehow notifies other nodes of the change. One way to do this is to assign the rank 0 to node 1 so that all nodes can communicate directly as usual with node 0 (which was node 1 before the change). Is there any way to do this?
Thanks