Answers.Body: <p>Here is my take.</p>
  <p>The larger the <span class="math-container">$\lambda$</span>, the more the corresponding regularization term for a coefficient will be big, so when minimizing the cost function, the coefficient will be reduced by a bigger factor, you can see this effect in the derivation of the update rule for gradient descent for example:
  <span class="math-container">\begin{align*} \theta_j := \theta_j - \alpha\ \left[ \left( \frac{1}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} \right) + \frac{\lambda}{m}\theta_j \right] &amp;\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline &amp; \end{align*}</span></p>
  <p><span class="math-container">\begin{align*} \theta_j := \theta_j (1- \alpha \frac{\lambda}{m})\left( \frac{\alpha}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} \right) &amp;\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline &amp; \end{align*}</span></p>
  <p>From this derivation, it is clear that at every update the coefficients get <strong>reduced</strong> by a factor that is usually a little less than 1 and <strong>directly proportional to <span class="math-container">$\lambda$</span></strong>, so, as <span class="math-container">$\lambda$</span> gets bigger, the weights get reduced more and more; eventually, for very big values of <span class="math-container">$\lambda$</span>, we risk to totally underfit the data, since only the regularization term will remain in the cost function and all the weights will go to zero.</p>
  <p>This is for linear regression, but it is essentially the same logic also for logistic regression.</p>
  <p>This is taken from <a href="https://www.coursera.org/learn/machine-learning/supplement/pKAsc/regularized-linear-regression" rel="nofollow noreferrer">Andrew Ng's course on Coursera</a>. A more mathematical precise (and complex) traction of the problem can be found in <a href="https://bloomberg.github.io/foml/#lecture-6-l1-and-l2-regularization" rel="nofollow noreferrer">the Bloomberg machine learning course material</a>.</p>
  <p>PS: in the derivation of the update rule for gradient descent, the <span class="math-container">$\lambda$</span> should be divided by the number of training examples, this is important in choosing the right <span class="math-container">$\lambda$</span> because we want this relationship to be inversely proportional, otherwise the coefficient won't be decreased.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-25T15:13:33.483Z
Answers.LastActivityDate: 2021-01-30T17:31:17.043Z
Answers.LastEditDate: 2021-01-30T17:31:17.043Z
Answers.UserDisplayName: m.rp
Body: <p>I'm learning logistic regression and <span class="math-container">$L_2$</span> regularization.
  The cost function looks like below.</p>
  <p><span class="math-container">$$J(w) = -\displaystyle\sum_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)})))$$</span></p>
  <p>And the regularization term is added. (<span class="math-container">$\lambda$</span> is a regularization strength)</p>
  <p><span class="math-container">$$J(w) = -\displaystyle\sum_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)}))) + \frac{\lambda}{2}\| w \|$$</span></p>
  <p>Intuitively, I know that if <span class="math-container">$\lambda$</span> becomes bigger, extreme weights are penalized and weights become closer to zero. However, I'm having a hard time to prove this mathematically.</p>
  <p><span class="math-container">$$\Delta{w} = -\eta\nabla{J(w)}$$</span>
  <span class="math-container">$$\frac{\partial}{\partial{w_j}}J(w) = (-y+\phi(z))x_j + \lambda{w_j}$$</span>
  <span class="math-container">$$\Delta{w} = \eta(\displaystyle\sum_{i=1}^{n}(y^{(i)}-\phi(z^{(i)}))x^{(i)} - \lambda{w_j})$$</span></p>
  <p>This doesn't show the reason why incrementing <span class="math-container">$\lambda$</span> makes weight become closer to zero. It is not intuitive.</p>
  
Tags: <machine-learning><proofs><hyper-parameters><regularization><l2-regularization>
Title: How does L2 regularization make weights smaller?
UserDisplayName: Riddle Aaron
ThreadText: # How does L2 regularization make weights smaller?
  
  (Asked by: username_0 on 2018-09-23)
  
  I'm learning logistic regression and $L\_2$ regularization.
  The cost function looks like below.
  
  $$J(w) = -\displaystyle\sum\_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)})))$$
  
  And the regularization term is added. ($\lambda$ is a regularization strength)
  
  $$J(w) = -\displaystyle\sum\_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)}))) + \frac{\lambda}{2}\| w \|$$
  
  Intuitively, I know that if $\lambda$ becomes bigger, extreme weights are penalized and weights become closer to zero. However, I'm having a hard time to prove this mathematically.
  
  $$\Delta{w} = -\eta\nabla{J(w)}$$
  $$\frac{\partial}{\partial{w\_j}}J(w) = (-y+\phi(z))x\_j + \lambda{w\_j}$$
  $$\Delta{w} = \eta(\displaystyle\sum\_{i=1}^{n}(y^{(i)}-\phi(z^{(i)}))x^{(i)} - \lambda{w\_j})$$
  
  This doesn't show the reason why incrementing $\lambda$ makes weight become closer to zero. It is not intuitive.
  
  **Tags:** machine-learning, proofs, hyper-parameters, regularization, l2-regularization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-25. Score: 2 | Accepted answer
  
  Here is my take.
  
  The larger the $\lambda$, the more the corresponding regularization term for a coefficient will be big, so when minimizing the cost function, the coefficient will be reduced by a bigger factor, you can see this effect in the derivation of the update rule for gradient descent for example:
  \begin{align\*} \theta\_j := \theta\_j - \alpha\ \left[ \left( \frac{1}{m}\ \sum\_{i=1}^m (h\_\theta(x^{(i)}) - y^{(i)})x\_j^{(i)} \right) + \frac{\lambda}{m}\theta\_j \right] &\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline & \end{align\*}
  
  \begin{align\*} \theta\_j := \theta\_j (1- \alpha \frac{\lambda}{m})\left( \frac{\alpha}{m}\ \sum\_{i=1}^m (h\_\theta(x^{(i)}) - y^{(i)})x\_j^{(i)} \right) &\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline & \end{align\*}
  
  From this derivation, it is clear that at every update the coefficients get **reduced** by a factor that is usually a little less than 1 and **directly proportional to $\lambda$**, so, as $\lambda$ gets bigger, the weights get reduced more and more; eventually, for very big values of $\lambda$, we risk to totally underfit the data, since only the regularization term will remain in the cost function and all the weights will go to zero.
  
  This is for linear regression, but it is essentially the same logic also for logistic regression.
  
  This is taken from [Andrew Ng's course on Coursera](https://www.coursera.org/learn/machine-learning/supplement/pKAsc/regularized-linear-regression). A more mathematical precise (and complex) traction of the problem can be found in [the Bloomberg machine learning course material](https://bloomberg.github.io/foml/#lecture-6-l1-and-l2-regularization).
  
  PS: in the derivation of the update rule for gradient descent, the $\lambda$ should be divided by the number of training examples, this is important in choosing the right $\lambda$ because we want this relationship to be inversely proportional, otherwise the coefficient won't be decreased.
  
  
  ---
  
  