I am reading following paper. And it uses EMA decay for variables.
BI-DIRECTIONAL ATTENTION FLOW FOR MACHINE COMPREHENSION 155

During training, the moving averages of all weights of the model are
maintained with the exponential decay rate of 0.999.

They use TensorFlow and I found the related code of EMA.

  
      github.com
  
  
    allenai/bi-att-flow/blob/master/basic/model.py#L229 36

    for var in tf.get_collection("ema/scalar", scope=self.scope):
        ema_var = ema.average(var)
        tf.scalar_summary(ema_var.op.name, ema_var)
    for var in tf.get_collection("ema/vector", scope=self.scope):
        ema_var = ema.average(var)
        tf.histogram_summary(ema_var.op.name, ema_var)


    with tf.control_dependencies([ema_op]):
        self.loss = tf.identity(self.loss)


def _build_var_ema(self):
    self.var_ema = tf.train.ExponentialMovingAverage(self.config.var_decay)
    ema = self.var_ema
    ema_op = ema.apply(tf.trainable_variables())
    with tf.control_dependencies([ema_op]):
        self.loss = tf.identity(self.loss)


def get_loss(self):
    return self.loss


def get_global_step(self):



  
  
    
    
  
  


In PyTorch, how do I apply EMA to Variables? In TensorFlow, there is tf.train.ExponentialMovingAverage class.
https://www.tensorflow.org/versions/r0.12/api_docs/python/train/moving_averages 77