As mentioned in my previous post, I don’t think this will work out of the box without reducing the value somehow manually.
Hooks would be registered on each replica and thus would only be valid for this model only.
I don’t think that nn.DataParallel or DDP reduces hook by default, but might be wrong, so I still think that your best bet would be to use e.g. torch.nn.parallel.gather.