Answers.Body: <p><strong>We use Mean Squared Error (MSE) and Mean Absolute Error (MAE) for regression, so why is it a problem for classification?</strong></p>
  <p>First, note that while this passage from the book does not explicitly specify whether it is talking about classification or regression, the mention of crossentropy loss necessitates that the discussion revolves around classification.</p>
  <p>In linear regression, the outputs range is (-inf, inf), and in this case, the <a href="https://math.stackexchange.com/questions/3669797/how-to-prove-a-mse-loss-function-is-convex-for-linear-regression">MSE</a> and <a href="https://math.stackexchange.com/questions/2579595/proof-convexity-of-mean-absolute-error">MAE</a> functions are both convex and can be minimized.  In classification, the output (assuming softmax activation of the final dense layer) is a vector of probabilities (with a probability for each class), each of which ranges from [0, 1] and together add to 1.  In this situation, <a href="https://towardsdatascience.com/why-not-mse-as-a-loss-function-for-logistic-regression-589816b5e03c" rel="nofollow noreferrer">MSE</a> and MAE are non-convex and may have a broad plateau.  When you have &quot;output units that saturate&quot;, you are in the plateau portion of the curve.  In the plateau regions, gradients (slopes/derivatives) are very small and, therefore, training is very slow (see Fig 4 of <a href="https://towardsdatascience.com/why-using-mean-squared-error-mse-cost-function-for-binary-classification-is-a-bad-idea-933089e90df7#:%7E:text=There%20are%20two%20reasons%20why,we%20assume%20a%20Gaussian%20prior." rel="nofollow noreferrer">article</a>).  Therefore, MSE and MAE &quot;often lead to poor results when used with gradient-based optimization.&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-04T20:17:27.663Z
Answers.LastActivityDate: 2022-11-06T01:37:37.950Z
Answers.LastEditDate: 2022-11-06T01:37:37.950Z
Answers.UserDisplayName: Snehal Patel
Body: <p><a href="https://www.deeplearningbook.org/contents/mlp.html" rel="nofollow noreferrer">Deep learning book chapter 6</a>: In 6.2.1.2 last paragraph:</p>
  <p>Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output units that saturate produce very small gradients when combined with these cost functions. This is one reason that the cross-entropy cost function is more popular than mean squared error or mean absolute error, even when it is not necessary to estimate an entire distribution p(y | x).</p>
  <p>explain the above sentence</p>
  <p>Doubt: But we use mean squared error (MSE) and mean absolute error (MAE) for regression.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-04T14:49:40.880Z
Comments.Text: is there more context? i.e. are they talking about classification? I agree that in general the statement isn't true, MSE is used in regression as you point out, but for e.g. a classification task the statement would be correct.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-04T17:02:09.810Z
Comments.Text: Yes, I went back through the sections and checked, it refers to classification, though not mentioned explicitly.
Comments.UserDisplayName: vivian.ai
Tags: <machine-learning><deep-learning><objective-functions><deep-neural-networks>
Title: Why MSE and MAE yield poor results when used with gradient-based optimization for classification?
UserDisplayName: vivian.ai
ThreadText: # Why MSE and MAE yield poor results when used with gradient-based optimization for classification?
  
  (Asked by: username_0 on 2022-11-04)
  
  [Deep learning book chapter 6](https://www.deeplearningbook.org/contents/mlp.html): In 6.2.1.2 last paragraph:
  
  Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output units that saturate produce very small gradients when combined with these cost functions. This is one reason that the cross-entropy cost function is more popular than mean squared error or mean absolute error, even when it is not necessary to estimate an entire distribution p(y | x).
  
  explain the above sentence
  
  Doubt: But we use mean squared error (MSE) and mean absolute error (MAE) for regression.
  
  **Tags:** machine-learning, deep-learning, objective-functions, deep-neural-networks
  
  > **Comments:**
  > **username_1**: is there more context? i.e. are they talking about classification? I agree that in general the statement isn't true, MSE is used in regression as you point out, but for e.g. a classification task the statement would be correct.  
  > **username_0**: Yes, I went back through the sections and checked, it refers to classification, though not mentioned explicitly.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-11-04. Score: 1 | Accepted answer
  
  **We use Mean Squared Error (MSE) and Mean Absolute Error (MAE) for regression, so why is it a problem for classification?**
  
  First, note that while this passage from the book does not explicitly specify whether it is talking about classification or regression, the mention of crossentropy loss necessitates that the discussion revolves around classification.
  
  In linear regression, the outputs range is (-inf, inf), and in this case, the [MSE](https://math.stackexchange.com/questions/3669797/how-to-prove-a-mse-loss-function-is-convex-for-linear-regression) and [MAE](https://math.stackexchange.com/questions/2579595/proof-convexity-of-mean-absolute-error) functions are both convex and can be minimized. In classification, the output (assuming softmax activation of the final dense layer) is a vector of probabilities (with a probability for each class), each of which ranges from [0, 1] and together add to 1. In this situation, [MSE](https://towardsdatascience.com/why-not-mse-as-a-loss-function-for-logistic-regression-589816b5e03c) and MAE are non-convex and may have a broad plateau. When you have "output units that saturate", you are in the plateau portion of the curve. In the plateau regions, gradients (slopes/derivatives) are very small and, therefore, training is very slow (see Fig 4 of [article](https://towardsdatascience.com/why-using-mean-squared-error-mse-cost-function-for-binary-classification-is-a-bad-idea-933089e90df7#:%7E:text=There%20are%20two%20reasons%20why,we%20assume%20a%20Gaussian%20prior.)). Therefore, MSE and MAE "often lead to poor results when used with gradient-based optimization."
  
  
  ---
  
  