Hi,
Do you mean a separate parameter that is not part of model.paramters() which needs to keep track of a moving average cross all workers?
If so, you should be able to just use all_reduce to get the average value of this variable across all workers, and use any typical running mean algorithm to update this on the desired worker.