Answers.Body: <p>Let us think about a very simple exemple, polynomial regression. It is not logistic regression nor neural network, but it should help to get the feel of regularization and dataset size.</p>
  <p>Imagine we have a typical straight line relationship between <span class="math-container">$y$</span> and <span class="math-container">$x$</span> with some random noise added. A balanced model would get an intercept, a slope and very low coefficients for higher order powers of the input variable, right?</p>
  <p>In our model <span class="math-container">$y=b+w_1 x+w_2 x^2+...$</span>, <span class="math-container">$w_i$</span> should be small for <span class="math-container">$i&gt;1$</span>.</p>
  <p>Now consider extreme cases for the size of the training dataset. If we have a dataset of three points, a parabola will be fitted to those three points with no error. No matter what three points we draw, there will be a parabola exactly fitting that. But with each three point sample from our underlying process, the parabola will change chaotically, sometimes it will point upwards, sometimes downwards, with no relationship to the underlying process, which was a noisy straight line. A strong regularization will be needed to properly estimate a straight line, in order to get an almost zero coefficient for the <span class="math-container">$x^2$</span> term.</p>
  <p>Let us go now to the other extreme. A huge lot of points from the noisy straight line are taken. What will happen to the <span class="math-container">$x^2$</span> coefficient? Well, most likely the effects of the different datapoints can be expected to cancel and there will be a very small concavity/convexity. The need for regularization is not that high.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-18T10:52:36.797Z
Answers.Text: Thanks, that's a good way to think about it and it makes sense. This still doesn't explain why other sources don't divide by m.
Answers.UserDisplayName: martinkunev
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-18T14:12:14.657Z
Answers.Text: Maybe the m is assumed to be a constant included in the lambda?
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-18T14:37:30.210Z
Answers.Text: But... in the video the same m divides the sum of losses to get the cost so that is a simpler explanation, to scale the sums accordingly.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-18T06:42:57.987Z
Answers.LastActivityDate: 2023-02-18T06:42:57.987Z
Answers.UserDisplayName: Jaume Oliver Lafont
Body: <p>I was watching a <a href="https://www.youtube.com/watch?v=6g0t3Phly2M&amp;t=300" rel="nofollow noreferrer">video by Andrew Ng</a> about regularization in logistic regression and neural network models.</p>
  <p>He uses the following term for regularization to (the sum is over the weights in the network). The regularization term is added to the loss function. Here <span class="math-container">$m$</span> is the number of training samples.</p>
  <p><span class="math-container">$\frac{\lambda}{2m} \sum_{i} \|w_i\|^2$</span></p>
  <p>Why is the term divided by <span class="math-container">$m$</span>? Looking at other sources I don't see <span class="math-container">$m$</span> in the denuminator. Why would we want smaller weights when the batch size is bigger?</p>
  <p>I wouldn't have been surprized to see some factor to account for the number of weights but I don't see how dividing by <span class="math-container">$m$</span> makes sense.</p>
  
Tags: <neural-networks><regularization>
Title: Should weight decay regularization be divided by the number of samples?
UserDisplayName: martinkunev
ThreadText: # Should weight decay regularization be divided by the number of samples?
  
  (Asked by: username_0 on 2023-02-12)
  
  I was watching a [video by Andrew Ng](https://www.youtube.com/watch?v=6g0t3Phly2M&t=300) about regularization in logistic regression and neural network models.
  
  He uses the following term for regularization to (the sum is over the weights in the network). The regularization term is added to the loss function. Here $m$ is the number of training samples.
  
  $\frac{\lambda}{2m} \sum\_{i} \|w\_i\|^2$
  
  Why is the term divided by $m$? Looking at other sources I don't see $m$ in the denuminator. Why would we want smaller weights when the batch size is bigger?
  
  I wouldn't have been surprized to see some factor to account for the number of weights but I don't see how dividing by $m$ makes sense.
  
  **Tags:** neural-networks, regularization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-18. Score: 1
  
  Let us think about a very simple exemple, polynomial regression. It is not logistic regression nor neural network, but it should help to get the feel of regularization and dataset size.
  
  Imagine we have a typical straight line relationship between $y$ and $x$ with some random noise added. A balanced model would get an intercept, a slope and very low coefficients for higher order powers of the input variable, right?
  
  In our model $y=b+w\_1 x+w\_2 x^2+...$, $w\_i$ should be small for $i>1$.
  
  Now consider extreme cases for the size of the training dataset. If we have a dataset of three points, a parabola will be fitted to those three points with no error. No matter what three points we draw, there will be a parabola exactly fitting that. But with each three point sample from our underlying process, the parabola will change chaotically, sometimes it will point upwards, sometimes downwards, with no relationship to the underlying process, which was a noisy straight line. A strong regularization will be needed to properly estimate a straight line, in order to get an almost zero coefficient for the $x^2$ term.
  
  Let us go now to the other extreme. A huge lot of points from the noisy straight line are taken. What will happen to the $x^2$ coefficient? Well, most likely the effects of the different datapoints can be expected to cancel and there will be a very small concavity/convexity. The need for regularization is not that high.
  
  > **Comments:**
  > **username_0**: Thanks, that's a good way to think about it and it makes sense. This still doesn't explain why other sources don't divide by m.  
  > **username_1**: Maybe the m is assumed to be a constant included in the lambda?  
  > **username_1**: But... in the video the same m divides the sum of losses to get the cost so that is a simpler explanation, to scale the sums accordingly.  
  
  ---
  
  