Answers.Body: <p>See the blog post <a href="https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/" rel="nofollow noreferrer">Why You Should Use Cross-Entropy Error Instead Of Classification Error Or Mean Squared Error For Neural Network Classifier Training</a> (2013) by James D. McCaffrey. </p>
  
  <p>It should give you an intuition of why the average cross-entropy (ACE) is more appropriate than MSE (but MSE is also applicable).</p>
  
  <p>In a few words, <span class="math-container">$\tanh$</span> + MSE is like sigmoid + MSE, but with labels for classes <span class="math-container">$-1$</span> and <span class="math-container">$1$</span> instead of <span class="math-container">$0$</span> and <span class="math-container">$1$</span>. If you look at the shape of <span class="math-container">$\tanh$</span> function, it has the same flat tails where changes of the argument don't change the result.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T06:55:27.730Z
Answers.Text: So, he's saying the problem with MSE is that the gradients get smaller when the predictions are close to 1 and 0, and so the network might get stuck on these outputs even when they're wrong? I don't see why cross entropy is different in this regard... He said that it applies to softmax and that the math is very cool, but didn't elaborate. Can you expand on his explanation, and also apply it to tanh?
Answers.UserDisplayName: feature_engineer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-02T14:53:37.260Z
Answers.LastActivityDate: 2019-11-16T19:05:52.027Z
Answers.LastEditDate: 2019-11-16T19:05:52.027Z
Answers.UserDisplayName: user2674414
Body: <p>Usually, in binary classification problems, we use <strong>sigmoid as the activation function of the last layer plus the binary cross-entropy as cost function</strong>.</p>
  
  <p>However, I have already experienced (more than once) that <strong><span class="math-container">$\tanh$</span> as activation function of last layer + MSE as cost function</strong> worked slightly better for binary classification problems.</p>
  
  <p>Using a binary image segmentation problem as an example, we have two scenarios:</p>
  
  <ol>
  <li>sigmoid (in the last layer) + cross-entropy: the output of the network will be a probability for each pixel and we want to maximize it according to the correct class. </li>
  <li><span class="math-container">$\tanh$</span> (in the last layer) + MSE: the output of the network will be a normalized pixel value [-1, 1] and we want to make it as close as possible the original value (normalized too).</li>
  </ol>
  
  <p>We all know the problems associated with a sigmoid (vanishing of gradients) and the benefits of the cross-entropy cost function. We also know <span class="math-container">$\tanh$</span> is slightly better than sigmoid (zero-centered and little less prone to gradient vanishing), but when we use MSE as the cost function, we are trying to minimize a completely different problem - regression instead of classification.</p>
  
  <p>Why is the hyperbolic tangent (<span class="math-container">$\tanh$</span>) combined with MSE more appropriate than the sigmoid combined with cross-entropy for binary classification problems? What's the intuition behind it?</p>
  
Tags: <neural-networks><classification><comparison><objective-functions><regression>
Title: Why is the hyperbolic tangent with MSE better than the sigmoid with cross-entropy?
UserDisplayName: Arnaldo Gualberto
ThreadText: # Why is the hyperbolic tangent with MSE better than the sigmoid with cross-entropy?
  
  (Asked by: username_0 on 2018-02-28)
  
  Usually, in binary classification problems, we use **sigmoid as the activation function of the last layer plus the binary cross-entropy as cost function**.
  
  However, I have already experienced (more than once) that **$\tanh$ as activation function of last layer + MSE as cost function** worked slightly better for binary classification problems.
  
  Using a binary image segmentation problem as an example, we have two scenarios:
  
  1. sigmoid (in the last layer) + cross-entropy: the output of the network will be a probability for each pixel and we want to maximize it according to the correct class.
  2. $\tanh$ (in the last layer) + MSE: the output of the network will be a normalized pixel value [-1, 1] and we want to make it as close as possible the original value (normalized too).
  
  We all know the problems associated with a sigmoid (vanishing of gradients) and the benefits of the cross-entropy cost function. We also know $\tanh$ is slightly better than sigmoid (zero-centered and little less prone to gradient vanishing), but when we use MSE as the cost function, we are trying to minimize a completely different problem - regression instead of classification.
  
  Why is the hyperbolic tangent ($\tanh$) combined with MSE more appropriate than the sigmoid combined with cross-entropy for binary classification problems? What's the intuition behind it?
  
  **Tags:** neural-networks, classification, comparison, objective-functions, regression
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-03-02. Score: 2 | Accepted answer
  
  See the blog post [Why You Should Use Cross-Entropy Error Instead Of Classification Error Or Mean Squared Error For Neural Network Classifier Training](https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/) (2013) by James D. McCaffrey.
  
  It should give you an intuition of why the average cross-entropy (ACE) is more appropriate than MSE (but MSE is also applicable).
  
  In a few words, $\tanh$ + MSE is like sigmoid + MSE, but with labels for classes $-1$ and $1$ instead of $0$ and $1$. If you look at the shape of $\tanh$ function, it has the same flat tails where changes of the argument don't change the result.
  
  > **Comments:**
  > **username_2**: So, he's saying the problem with MSE is that the gradients get smaller when the predictions are close to 1 and 0, and so the network might get stuck on these outputs even when they're wrong? I don't see why cross entropy is different in this regard... He said that it applies to softmax and that the math is very cool, but didn't elaborate. Can you expand on his explanation, and also apply it to tanh?  
  
  ---
  
  