Answers.Body: <p>There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems.</p>
  <p><strong>A theoretical reason</strong></p>
  <p>Both aforementioned losses are negative logarithmic likelihoods for following parametric statistical models.</p>
  <p>MSE corresponds to the parametric regression model <span class="math-container">$Y = f(\theta, X) + \epsilon$</span>, where <span class="math-container">$\epsilon$</span> is a normally distributed zero-mean random error, independent of <span class="math-container">$X$</span> (and if you make <span class="math-container">$\epsilon$</span> not normally distributed, but Laplace-distributed, you will get another well known loss — mean average error).</p>
  <p>Cross-entropy loss corresponds to a classification model with two classes <span class="math-container">$C_0$</span> and <span class="math-container">$C_1$</span> where <span class="math-container">$P(C_1|x) = f(\theta, x)$</span>.</p>
  <p>Minimizing those losses means getting a maximum likelihood estimator for the regression/classification parameter <span class="math-container">$\theta$</span>.</p>
  <p><strong>A practical reason</strong></p>
  <p>Cross-entropy loss is just plain better in distinguishing “good” classification from a “bad” one.</p>
  <p>Consider a following example:</p>
  <p>Suppose we have <span class="math-container">$101$</span> object from class 1 and two experts deciding whether they are from class 0 or from class 1. The first expert made one blatant error, assigning score (probability of belonging to class 1) 0.00001 to one of the objects and 0.99 to the others. The second expert is always correct but a bit less confident — they assigned score 0.9 to all objects. Based on common sense, predictions of the second expert are obviously better than those of the first one. Now let’s look at the losses.</p>
  <p>First expert has MSE <span class="math-container">$\sim 0.0099998$</span> and Cross-Entropy loss <span class="math-container">$\sim 12.52$</span>.</p>
  <p>Second expert has MSE <span class="math-container">$0.01$</span> and Cross-Entropy loss <span class="math-container">$\sim 10.64$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-10T18:08:32.590Z
Answers.Text: +1 The same idea applies to Frank Harrell’s comments [here](https://stats.stackexchange.com/q/540483/247274).
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-10T14:05:03.600Z
Answers.LastActivityDate: 2022-09-10T14:19:20.730Z
Answers.LastEditDate: 2022-09-10T14:19:20.730Z
Answers.UserDisplayName: Vladimir Yankovskiy
Body: <p>I currently have a neural network that can manage to perform polynomial (single output) regression problems. I now want to upscale to classification problems (eg: image recognition). <strong>Can I do this with MSE loss</strong>, or do I need to switch to a more complicated categorical cross-entropy loss function...?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-09T22:20:40.537Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-10T00:02:57.547Z
Comments.Text: You sound confused; SGD is not a *loss*, it is an optimization method that can be used for minimizing various functions usually used as loss ones in NN (both for classification and regression).
Comments.UserDisplayName: desertnaut
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-10T08:52:53.833Z
Comments.Text: Yeah sorry I meant to say MSE.
Comments.UserDisplayName: Gamaray
Tags: <neural-networks><machine-learning><classification><loss><stochastic-gradient-descent>
Title: Can MSE be used for NN categorical classification problems
UserDisplayName: Gamaray
ThreadText: # Can MSE be used for NN categorical classification problems
  
  (Asked by: username_0 on 2022-09-09)
  
  I currently have a neural network that can manage to perform polynomial (single output) regression problems. I now want to upscale to classification problems (eg: image recognition). **Can I do this with MSE loss**, or do I need to switch to a more complicated categorical cross-entropy loss function...?
  
  **Tags:** neural-networks, machine-learning, classification, loss, stochastic-gradient-descent
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  > **username_2**: You sound confused; SGD is not a *loss*, it is an optimization method that can be used for minimizing various functions usually used as loss ones in NN (both for classification and regression).  
  > **username_0**: Yeah sorry I meant to say MSE.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-09-10. Score: 1 | Accepted answer
  
  There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems.
  
  **A theoretical reason**
  
  Both aforementioned losses are negative logarithmic likelihoods for following parametric statistical models.
  
  MSE corresponds to the parametric regression model $Y = f(\theta, X) + \epsilon$, where $\epsilon$ is a normally distributed zero-mean random error, independent of $X$ (and if you make $\epsilon$ not normally distributed, but Laplace-distributed, you will get another well known loss — mean average error).
  
  Cross-entropy loss corresponds to a classification model with two classes $C\_0$ and $C\_1$ where $P(C\_1|x) = f(\theta, x)$.
  
  Minimizing those losses means getting a maximum likelihood estimator for the regression/classification parameter $\theta$.
  
  **A practical reason**
  
  Cross-entropy loss is just plain better in distinguishing “good” classification from a “bad” one.
  
  Consider a following example:
  
  Suppose we have $101$ object from class 1 and two experts deciding whether they are from class 0 or from class 1. The first expert made one blatant error, assigning score (probability of belonging to class 1) 0.00001 to one of the objects and 0.99 to the others. The second expert is always correct but a bit less confident — they assigned score 0.9 to all objects. Based on common sense, predictions of the second expert are obviously better than those of the first one. Now let’s look at the losses.
  
  First expert has MSE $\sim 0.0099998$ and Cross-Entropy loss $\sim 12.52$.
  
  Second expert has MSE $0.01$ and Cross-Entropy loss $\sim 10.64$.
  
  > **Comments:**
  > **username_4**: +1 The same idea applies to Frank Harrell’s comments [here](https://stats.stackexchange.com/q/540483/247274).  
  
  ---
  
  