iffiX:

since you are blocking all processes on init and exchange their connection information, it would be overly complex to try and recover these information on crash.


The rendezvous in RPC init will be gone in future releases (not sure which release yet  ). We are thinking about using c10d::Store + listeners to support dynamic worker join/leave.

My solution basically implements a stable distributed election process & perfect failure detector based on rpc, and let each worker in the rpc group takes one/many role(s), when a worker failes (hard/soft), its role will be reassigned to other workers and this worker is marked as tainted permanently. Users only needs to declare the number of total workers on start and specify what a role have to do when they are: initialized, running, and stopped. Adds a little bit overhead on start & worker failure, but near zero overhead during a normal rpc call.

This is awesome! Are these in an open-source repo? It will be great if we could have the opportunity to learn more about it.  cc @Kiuk_Chung