Consider a model training with DDP, all parameters are updated with gradient, except one that is only updated to keep recording the moving average of a value (generated after each forward pass) during training.
Is there an easy way to implemented it in DDP?
Thanks!