Yes, I was actually thinking of using rpc. My only concern is about its performance, compared to the other collective primitives (e.g., gather, broadcast…etc.). Can you comment on this performance comparison?
The problem of consistency among PSes could be solved using checkpoints. Assume there is one primary PS, I’d let this PS only to update the model and periodically checkpoint it to some persistent database. If it crashes, the backup PS first loads the latest checkpoint and then continues training normally. How does that sound?