Hi,
Exponential Moving Average (EMA) is an important feature in state-of-the-art research, in Tensorflow they already implemented it with tf.train.ExponentialMovingAverage 13. I wonder why the Pytorch team has not released an official version of EMA.
In others related question, there is no expert confirm that it is the correct implementation:

  
    
    
    Exponential Moving Average in PyTorch, for weights and gradients nlp
  
  
    Do we need to apply exponential moving average to weights during training when we use Adam (or other optimizers)? 
My EMA is defined as: 
class EMA(object):

def __init__(self, mu):
    self.mu = mu
    self.shadow = {}

def register(self, name, val):
    self.shadow[name] = val.clone()

def __call__(self, name, x):
    assert name in self.shadow
    new_average = (1.0 - self.mu) * x + self.mu * self.shadow[name]
    self.shadow[name] = new_average.clone()
    return new_average

My code is like…
  



  
    
    
    How to apply exponential moving average decay for variables? 
  
  
    I am reading following paper. And it uses EMA decay for variables. 
BI-DIRECTIONAL ATTENTION FLOW FOR MACHINE COMPREHENSION 

During training, the moving averages of all weights of the model are 
maintained with the exponential decay rate of 0.999. 

They use TensorFlow and I found the related code of EMA. 
 
 
In PyTorch, how do I apply EMA to Variables? In TensorFlow, there is tf.train.ExponentialMovingAverage class. 
https://www.tensorflow.org/versions/r0.12/api_docs/python/train/moving_avera…
  


Or in this repo 28, the issue said there was some problem with the code.