Answers.Body: <p>I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE):</p>
  <p>MSE:</p>
  <ol>
  <li>L2 loss is continuously-differentiable across any domain, unlike L1 loss.  This makes training more stable and allows for gradient-based optimization, as opposed to combinatorial optimization.</li>
  <li>Using L2 loss (without any regularization) corresponds to the Ordinary Least Squares Estimator, which, if you're able to invoke Gauss-Markov assumptions, can lead to some beneficial theoretical guarantees about your estimator/model (e.g. that it is the &quot;Best Linear Unbiased Estimator&quot;).  Source: <a href="https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem" rel="noreferrer">https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem</a>.</li>
  </ol>
  <p>L2 Regularization:</p>
  <ol>
  <li><p>Using L2 regularization is equivalent to invoking a Gaussian prior (see <a href="https://stats.stackexchange.com/questions/163388/why-is-the-l2-regularization-equivalent-to-gaussian-prior">https://stats.stackexchange.com/questions/163388/why-is-the-l2-regularization-equivalent-to-gaussian-prior</a>) on your model/estimator.  If modeling your problem as a Maximum A Posteriori Inference (MAP) problem, if your likelihood model (p(y|x)) is Gaussian, then your posterior distribution over parameters (p(x|y)) will also be Gaussian.  From Wikipedia: &quot;If the likelihood function is Gaussian, choosing a Gaussian prior over the mean will ensure that the posterior distribution is also Gaussian&quot; (source: <a href="https://en.wikipedia.org/wiki/Conjugate_prior" rel="noreferrer">https://en.wikipedia.org/wiki/Conjugate_prior</a>).</p>
  </li>
  <li><p>As in the case above, L2 loss is continuously-differentiable across any domain, unlike L1 loss.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T04:46:06.277Z
Answers.Text: I think it's important to note that L2 loss definitely isn't used more commonly than other loss functions. In fact if anything I would say cross-entropy has the largest appearance as it ties in with softmax activations which is all around excellent for classification.
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T10:24:03.723Z
Answers.Text: @Recessive I agree
Answers.UserDisplayName: pedrum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T13:03:10.737Z
Answers.Text: I think this answer doesn't clarify an important point. An "l2 loss" would be any loss that uses the "l2 norm" as a regularisation term (and, in that case, you will get MAP). This loss can be the MSE  or it can e.g. the cross-entropy, i.e. l2 norm can be used for regression or classification.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T19:17:15.743Z
Answers.Text: @nbro MSE loss specifically is sometimes referred to as L2 loss, so that one was probably meant in the question (e.g., see https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#mse-l2)
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T20:04:56.207Z
Answers.Text: @DennisSoemers Right. I suspected that L2 loss couldn't just refer to any loss that uses L2 norm, but, well, I guess it's a little bit misleading to call MSE L2 loss when it doesn't use L2 regularisation, but I guess the 2 stands for the square in that case. Anyway, why not just calling it MSE? I don't get it.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T03:58:09.017Z
Answers.LastActivityDate: 2020-07-28T03:58:09.017Z
Answers.UserDisplayName: Ryan S
Body: <p>Why L2 loss is more commonly used in Neural Networks than other loss functions?
  What is the reason to L2 being a default choice in Neural Networks?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-27T20:39:03.843Z
Comments.Text: By "L2 loss" do you mean L2 penalty/regularisation?
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><objective-functions><regularization><loss>
Title: Why L2 loss is more commonly used in Neural Networks than other loss functions?
UserDisplayName: Ali Khalili
ThreadText: # Why L2 loss is more commonly used in Neural Networks than other loss functions?
  
  (Asked by: username_0 on 2020-07-27)
  
  Why L2 loss is more commonly used in Neural Networks than other loss functions?
  What is the reason to L2 being a default choice in Neural Networks?
  
  **Tags:** neural-networks, deep-learning, objective-functions, regularization, loss
  
  > **Comments:**
  > **username_1**: By "L2 loss" do you mean L2 penalty/regularisation?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-28. Score: 5 | Accepted answer
  
  I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE):
  
  MSE:
  
  1. L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows for gradient-based optimization, as opposed to combinatorial optimization.
  2. Using L2 loss (without any regularization) corresponds to the Ordinary Least Squares Estimator, which, if you're able to invoke Gauss-Markov assumptions, can lead to some beneficial theoretical guarantees about your estimator/model (e.g. that it is the "Best Linear Unbiased Estimator"). Source: <https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem>.
  
  L2 Regularization:
  
  1. Using L2 regularization is equivalent to invoking a Gaussian prior (see <https://stats.stackexchange.com/questions/163388/why-is-the-l2-regularization-equivalent-to-gaussian-prior>) on your model/estimator. If modeling your problem as a Maximum A Posteriori Inference (MAP) problem, if your likelihood model (p(y|x)) is Gaussian, then your posterior distribution over parameters (p(x|y)) will also be Gaussian. From Wikipedia: "If the likelihood function is Gaussian, choosing a Gaussian prior over the mean will ensure that the posterior distribution is also Gaussian" (source: <https://en.wikipedia.org/wiki/Conjugate_prior>).
  2. As in the case above, L2 loss is continuously-differentiable across any domain, unlike L1 loss.
  
  > **Comments:**
  > **username_3**: I think it's important to note that L2 loss definitely isn't used more commonly than other loss functions. In fact if anything I would say cross-entropy has the largest appearance as it ties in with softmax activations which is all around excellent for classification.  
  > **username_4**: @username_3 I agree  
  > **username_1**: I think this answer doesn't clarify an important point. An "l2 loss" would be any loss that uses the "l2 norm" as a regularisation term (and, in that case, you will get MAP). This loss can be the MSE  or it can e.g. the cross-entropy, i.e. l2 norm can be used for regression or classification.  
  > **username_5**: @username_1 MSE loss specifically is sometimes referred to as L2 loss, so that one was probably meant in the question (e.g., see https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#mse-l2)  
  > **username_1**: @username_5 Right. I suspected that L2 loss couldn't just refer to any loss that uses L2 norm, but, well, I guess it's a little bit misleading to call MSE L2 loss when it doesn't use L2 regularisation, but I guess the 2 stands for the square in that case. Anyway, why not just calling it MSE? I don't get it.  
  
  ---
  
  