"I might miss something. Looks like you want to  manually  update a parameter, but still want DDP to help synchronize the parameter across the GPUs/processes"’
Yes. This is what I would like to achieve.
For example
class model():
def init(self):
a = torch.zeros((3, 1))
self.register_buffer(“a”, a)
def update_a(self, b):
self.a.add_(b)
b is a vector that is dynamic with respect to the input data. So I could not manually set it to the same value.



 mrshenli:

If this parameter is manually updated, would it be possible to let all processes to set it to the same value?


Let me know whether I describe the problem clearly? Thanks a lot for your help.