title: What is the proper way to weight decay for Adam Optimizer
body: <p>Since Adam Optimizer keeps an pair of running averages like mean/variance for the gradients, I wonder how it should properly handle weight decay. I have seen two ways of implementing it.</p>
  
  <ol>
  <li><p>Only update mean/variance from the gradients based on the objective loss, decay weight explicitly at each mini-batch.  (the following code is taken from <a href="https://github.com/dmlc/mxnet/blob/v0.7.0/python/mxnet/optimizer.py" rel="noreferrer">https://github.com/dmlc/mxnet/blob/v0.7.0/python/mxnet/optimizer.py</a>)</p>
  
  <pre><code>weight[:] -= lr*mean/(sqrt(variance) + self.epsilon)
  
  wd = self._get_wd(index)
  if wd &gt; 0.:
      weight[:] -= (lr * wd) * weight
  </code></pre></li>
  <li><p>Update mean/variance from the gradients based on the objective loss + regularization loss, and update weights like usual. (the following code is taken from <a href="https://github.com/dmlc/mxnet/blob/master/src/operator/optimizer_op-inl.h#L210" rel="noreferrer">https://github.com/dmlc/mxnet/blob/master/src/operator/optimizer_op-inl.h#L210</a>)</p>
  
  <pre><code>grad = scalar&lt;DType&gt;(param.rescale_grad) * grad +
  scalar&lt;DType&gt;(param.wd) * weight;
  // stuff
  Assign(out, req[0],
     weight -
     scalar&lt;DType&gt;(param.lr) * mean /
     (F&lt;square_root&gt;(var) + scalar&lt;DType&gt;(param.epsilon)));
  </code></pre></li>
  </ol>
  
  <p>These two approaches sometimes show significant difference in training results. And I actually think the first one makes more sense (and find it gives better results time to time).  Caffe and old version of mxnet follow the first approach, while torch, tensorflow and new version of mxnet follow the second one. </p>
  
  <p>Really appreciate your help！</p>