Answers.Body: <p>The regularization terms are applied to the loss functions by default.
  However, their gradients do appear in the update step as the gradient of loss appears in the update step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-09T08:19:25.953Z
Answers.Text: Thank you for the clarification, but when you say "gradients appear in the update step", shouldn't we have the derivative of R(W) in the weight update equation, but instead we have the R(W) as it is.
Answers.UserDisplayName: Hrushi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-10T03:00:44.213Z
Answers.Text: That seems to be the case on the 3rd equation (most bottom). I think there's simply a typo on 2nd equation (upper-right). We should have gradient of R(w) there.
Answers.UserDisplayName: SpiderRico
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-11T06:21:14.897Z
Answers.Text: Yes, I believe it's a mistake from my part, [as per this credible resource ] (http://neuralnetworksanddeeplearning.com/chap3.html#regularization)
Answers.UserDisplayName: Hrushi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-08T19:12:05.910Z
Answers.LastActivityDate: 2020-07-08T19:12:05.910Z
Answers.UserDisplayName: SpiderRico
Body: <p>I have a confusion on where exactly is the L2 regularization (weight decay) is added.</p>
  <p>In various resources I have come across, I find two equations where L2 regularization is applied.</p>
  <p>Adding R(W) to loss function makes sense because it tries decrease large weights. Also, I have seen equations where we add R(W) to the weight update term, 2nd equation in 2nd line as shown in this image:</p>
  <p><a href="https://i.sstatic.net/n0EdM.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/n0EdM.jpg" alt="enter image description here" /></a></p>
  <p>In the above image, using the weight update rule that</p>
  <pre><code>W(final) = W(initial) + (alpha) * (Gradient of W),
  </code></pre>
  <p>I obtain a different equation as compared to the other equation which is commonly written in various resources.</p>
  <p>Where exactly is the regularization term added, I previously thought it was only added in Loss function but that gives me a different weight update equation from what is commonly presented in resources.( Or is my interpretation of the equation wrong? )</p>
  <p>I presume it is also added in weight update equation because while constructing models, we add regularization term.</p>
  <pre><code>model.add(Conv2D(256, (5,5), padding=&quot;same&quot;, kernel_regularizer=l2(reg))) 
  </code></pre>
  <p>Would be grateful for any help.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-11T12:08:36.220Z
Comments.Text: As per the keras documentation, "These penalties are summed into the loss function that the network optimizes. "
Comments.UserDisplayName: Hrushi
Tags: <regularization>
Title: Where is L2-regularization term applied
UserDisplayName: Hrushi
ThreadText: # Where is L2-regularization term applied
  
  (Asked by: username_0 on 2020-07-06)
  
  I have a confusion on where exactly is the L2 regularization (weight decay) is added.
  
  In various resources I have come across, I find two equations where L2 regularization is applied.
  
  Adding R(W) to loss function makes sense because it tries decrease large weights. Also, I have seen equations where we add R(W) to the weight update term, 2nd equation in 2nd line as shown in this image:
  
  [![enter image description here](https://i.sstatic.net/n0EdM.jpg)](https://i.sstatic.net/n0EdM.jpg)
  
  In the above image, using the weight update rule that
  
  ```
  W(final) = W(initial) + (alpha) * (Gradient of W),
  
  ```
  
  I obtain a different equation as compared to the other equation which is commonly written in various resources.
  
  Where exactly is the regularization term added, I previously thought it was only added in Loss function but that gives me a different weight update equation from what is commonly presented in resources.( Or is my interpretation of the equation wrong? )
  
  I presume it is also added in weight update equation because while constructing models, we add regularization term.
  
  ```
  model.add(Conv2D(256, (5,5), padding="same", kernel_regularizer=l2(reg))) 
  
  ```
  
  Would be grateful for any help.
  
  **Tags:** regularization
  
  > **Comments:**
  > **username_0**: As per the keras documentation, "These penalties are summed into the loss function that the network optimizes. "  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-08. Score: 1 | Accepted answer
  
  The regularization terms are applied to the loss functions by default.
  However, their gradients do appear in the update step as the gradient of loss appears in the update step.
  
  > **Comments:**
  > **username_0**: Thank you for the clarification, but when you say "gradients appear in the update step", shouldn't we have the derivative of R(W) in the weight update equation, but instead we have the R(W) as it is.  
  > **username_1**: That seems to be the case on the 3rd equation (most bottom). I think there's simply a typo on 2nd equation (upper-right). We should have gradient of R(w) there.  
  > **username_0**: Yes, I believe it's a mistake from my part, [as per this credible resource ] (http://neuralnetworksanddeeplearning.com/chap3.html#regularization)  
  
  ---
  
  