I was referring to this part of the PipeDream paper:


Each backward pass in a stage results
in weight updates; the next forward pass uses the latest version
of weights available, and “stashes" a copy of these weights to use
during the corresponding backward pass. Although the forward
pass will not see updates from incomplete in-flight mini-batches,
learning is still effective because model weights change relatively
slowly and bounded staleness has been found effective in improving training speeds

And the Hogwild paper linked above also mentioned sth with similar no-lock spirit:

In this work, we propose a simple strategy for eliminating the overhead associated with locking:
run SGD in parallel without locks, a strategy that we call Hogwild!. In Hogwild!, processors are
allowed equal access to shared memory and are able to update individual components of memory
at will. Such a lock-free scheme might appear doomed to fail as processors could overwrite each
other’s progress. However, when the data access is sparse, meaning that individual SGD steps only
modify a small part of the decision variable, we show that memory overwrites are rare and that
they introduce barely any error into the computation when they do occur. We demonstrate both
theoretically and experimentally a near linear speedup with the number of processors on commonly
occurring sparse learning problems.

In general, if lock is necessary (e.g., due to unacceptable accuracy drop), applications can do so by explicitly acquiring locks, but this will certainly have impact on training speed. So it is up to the application to decide how to play with the trade-off.





Is ring allreduce better than parameter server all the time?

No, the merit of allreduce is that it can (actually depend on the loss function) achieve mathematical equivalence with local training. But whether synchronous training is better than asynchronous training is an open question. DistributedDataParallel in PyTorch is using allreduce, but there are also that types of DDP, e.g., SlowMo 2 is using gossip.

But can the DDP use parameter server strategy instead?

Yes, it certainly can. The holgwild paper linked above can be one example of using parameter-server-based data parallel on the “decision variable”, e.g., an embedding table.




Currently the RPC package does not support priority-based communication/execution yet, but this might be doable from the application side. The callee can maintain priority queues to order incoming tasks and use an RPC argument to indicate the priority. The WIP async user function 2 will help to reduce the callee-side overhead for this use case.