No, I don’t think batchnorm layers have this functionality added (in case you have multiple GPUs, you could try SyncBatchNorm). For a single GPU, you could try to change the momentum and see, if you could smooth the running estimates.