Hey @rahul003
This is for the use case where you have multiple independent trainers accessing the same parameter server (PS), where each training runs its own forward-backward-optimize in every iteration. With this setup, the parameters on the PS needs to remember gradients for different trainers, as they would run optimizer step independently. Hence, we cannot directly accumulate grads into param.grad field.
This is one example: https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html 3
The idea is partially borrowed from Hogwild training: https://people.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf
BTW, could you please add a “distributed-rpc” tag for RPC-related questions so that the team can get back to you promptly, thanks!