Hey @ZiyiZhu
That’s a great question! I have two comments on this concurrent param updating approach:

Each trainer has its dedicated gradients on the parameter server. When using distributed autograd, the computed gradients are stored in the autograd context (instead of in param.grad), which can be identified by the unique cid. So there is no concern on race errors for gradient computation.
It is true that there can be multiple trainers updating the same parameter concurrently, and it is true that the parameter might change after a trainer computes the gradients and before it applies the gradients to the parameter. This idea is partially borrowed from the Hogwild! paper 1. And the approach of using “not perfectly up-to-date gradients” can also be found in other projects (e.g., PipeDream 2).

In general, this is a trade-off between model accuracy and training speed. And in practice, we saw several use cases where this helped to accelerate training a lot with little or none accuracy penalty.
cc @rvarm1