@Kiuk_Chung  Can your elaborate more on the last part?
My goal is to keep the training process going with minimum withdraw from failed synchronization.
I assume your elastic feature offers the flexibility of leaving + rejoining the sync pool at any given point?
If one failed worker = all process have to restart from last checkpoint, it’s basically the same as I ran a background monitor process constantly checks->kills->restarts the whole process, which is not very efficient…