hmm how do you kill the agent? using ctrl+c? if so can you try to kill by sending a SIGTERM instead? Might be related to [distributed elastic] How to tolerate agent failures with etcd rendezvous backend? · Issue #67616 · pytorch/pytorch · GitHub 3
Looks like in python SIGINT (send when ctrl + c from terminal) produces a KeyboardInterrupt error which results in the shutdown() method of rendezvous being called in the finally block. The shutdown() should only get called on an orderly shutdown not a “failure” hence closes the rendezvous permanently failing all other nodes with a RendezvousClosedException
Elastic was built to handle real life faults which would just SIGTERM or SIGKILL or in the worst case just cause the node itself to just crash and disappear. All of which doesn’t produce a SIGINT.