Answers.Body: <p>Usually, when talking about regularization for neural networks there are 3 main types:
  L1, L2 and dropout. All affect the gradient descent procedure.</p>
  
  <p>L1 and L2 regularization is implemented in the loss function, and therefore are part of gradient descent directly by altering the derivatives of the loss function thereby altering the weight update rules of the network during gradient descent. </p>
  
  <p>For L1 you add a penalty based on the <span class="math-container">$\mathcal L^1$</span> norm of the weight vector, while for L2 you add a penalty based on the <span class="math-container">$\mathcal L^2$</span>  norm. </p>
  
  <p>For dropout, there is no direct impact on the loss function, but you are still interfering in the gradient descent procedure indirectly by masking nodes to alter the forward and backward propagation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-01T07:44:41.637Z
Answers.LastActivityDate: 2020-04-02T22:40:52.190Z
Answers.LastEditDate: 2020-04-02T22:40:52.190Z
Answers.UserDisplayName: user10283726
Body: <p>Gradient descent is used to reduce the loss and regularization is used to fight over-fitting. </p>
  
  <p>Is there any relation between gradient descent and regularization, or both are independent of each other?</p>
  
Tags: <deep-learning><comparison><gradient-descent><regularization><hyper-parameters>
Title: What is relation between gradient descent and regularization in deep learning?
UserDisplayName: DRV
ThreadText: # What is relation between gradient descent and regularization in deep learning?
  
  (Asked by: username_0 on 2020-04-01)
  
  Gradient descent is used to reduce the loss and regularization is used to fight over-fitting.
  
  Is there any relation between gradient descent and regularization, or both are independent of each other?
  
  **Tags:** deep-learning, comparison, gradient-descent, regularization, hyper-parameters
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-01. Score: 4 | Accepted answer
  
  Usually, when talking about regularization for neural networks there are 3 main types:
  L1, L2 and dropout. All affect the gradient descent procedure.
  
  L1 and L2 regularization is implemented in the loss function, and therefore are part of gradient descent directly by altering the derivatives of the loss function thereby altering the weight update rules of the network during gradient descent.
  
  For L1 you add a penalty based on the $\mathcal L^1$ norm of the weight vector, while for L2 you add a penalty based on the $\mathcal L^2$ norm.
  
  For dropout, there is no direct impact on the loss function, but you are still interfering in the gradient descent procedure indirectly by masking nodes to alter the forward and backward propagation.
  
  
  ---
  
  