Hi, every one.
I read some documentation and found that many ML systems implement distributed checkpoints in a synchronous way, i.e. for every n steps, all nodes stop and dump the state.
Then I am curious how should PyTorch implements distributed checkpoint when it is doing asynchronous training. e.g. Node 1 is at iteration 10, Node 2 is at Iteration 15, etc
Does anyone know how PyTorch checkpoints the state for asynchronous training?
Thanks!