Answers.Body: <p>The answer is largely the same whether we consider <span class="math-container">$\ell_1$</span> or <span class="math-container">$\ell_2$</span> regularisation, so I will just speak generally about regularisation.</p>
  <h2>Mean square error for training data</h2>
  <p>Given some training data <span class="math-container">$\{(x_i, y_i)\}_{i = 1}^n$</span>, a linear regression line <span class="math-container">$Y = aX + b$</span> fit using the least squares method looks for coefficients that minimise the <em>sum of squares</em>, i.e. they are the minimisers given by</p>
  <p><span class="math-container">$$ \mathrm{arg\,min}_{a, b} \sum_{i = 1}^n \left(y_i - (ax_i + b)\right)^2.$$</span></p>
  <p>This gives the same coefficients as minimising the <em>mean square error</em></p>
  <p><span class="math-container">$$ \mathrm{MSE}\left((x_1, y_1), \dots, (x_n, y_n)\right) = \frac{1}{n} \sum_{i = 1}^n \left(y_i - (ax_i + b)\right)^2.$$</span></p>
  <p>So, by definition, the coefficients <span class="math-container">$(a, b)$</span> minimise the MSE on the training data. Any regularisation will only increase the MSE on the training data.</p>
  <h2>Generalisation performance</h2>
  <p>The main point of regularisation is to prevent overfitting on the data and improve the generalisation performance (i.e. on the test set).</p>
  <p>With an appropriate parameter for regularisation, you may obtain a smaller MSE on the test set. This depends on your dataset and the parameters you choose: strong regularisation may lead to underfitting, whereas weak regularisation might not make much difference to the coefficients that you fit.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-05T13:02:31.397Z
Answers.LastActivityDate: 2021-02-05T13:02:31.397Z
Answers.UserDisplayName: htl
Body: <p>Consider linear regression. The mean squared error (MSE) is 120.5 for the training dataset. We've reached the minimum for the training data.</p>
  <p>Is it possible that by applying Lasso (L1 regularization) we would get a lower MSE for the training data? Would it get lower for the test data? Would this also hold for ridge regression (L2 regularization)?</p>
  
Tags: <linear-regression><mean-squared-error><l2-regularization><l1-regularization>
Title: Would either $L_1$ or $L_2$ regularisation lower the MSE on the training and test data?
UserDisplayName: user6394019
ThreadText: # Would either $L_1$ or $L_2$ regularisation lower the MSE on the training and test data?
  
  (Asked by: username_0 on 2021-02-05)
  
  Consider linear regression. The mean squared error (MSE) is 120.5 for the training dataset. We've reached the minimum for the training data.
  
  Is it possible that by applying Lasso (L1 regularization) we would get a lower MSE for the training data? Would it get lower for the test data? Would this also hold for ridge regression (L2 regularization)?
  
  **Tags:** linear-regression, mean-squared-error, l2-regularization, l1-regularization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-05. Score: 1 | Accepted answer
  
  The answer is largely the same whether we consider $\ell\_1$ or $\ell\_2$ regularisation, so I will just speak generally about regularisation.
  
  ## Mean square error for training data
  
  Given some training data $\{(x\_i, y\_i)\}\_{i = 1}^n$, a linear regression line $Y = aX + b$ fit using the least squares method looks for coefficients that minimise the *sum of squares*, i.e. they are the minimisers given by
  
  $$ \mathrm{arg\,min}\_{a, b} \sum\_{i = 1}^n \left(y\_i - (ax\_i + b)\right)^2.$$
  
  This gives the same coefficients as minimising the *mean square error*
  
  $$ \mathrm{MSE}\left((x\_1, y\_1), \dots, (x\_n, y\_n)\right) = \frac{1}{n} \sum\_{i = 1}^n \left(y\_i - (ax\_i + b)\right)^2.$$
  
  So, by definition, the coefficients $(a, b)$ minimise the MSE on the training data. Any regularisation will only increase the MSE on the training data.
  
  ## Generalisation performance
  
  The main point of regularisation is to prevent overfitting on the data and improve the generalisation performance (i.e. on the test set).
  
  With an appropriate parameter for regularisation, you may obtain a smaller MSE on the test set. This depends on your dataset and the parameters you choose: strong regularisation may lead to underfitting, whereas weak regularisation might not make much difference to the coefficients that you fit.
  
  
  ---
  
  