Answers.Body: <p>We sometimes see that binary cross-entropy (BCE) loss is used for regression problems.  This post is my opinion on using BCE for regression problems.</p>
  <p>The figure below is the plots of BCE, <span class="math-container">$-t*\log(x) - (1-t)*\log(1-x)$</span>, for several target values <span class="math-container">$t = 0.0, 0.1, ..., 0.5$</span>. (The plots for <span class="math-container">$t&gt;0.5$</span> are mirror images of those for <span class="math-container">$t&lt;0.5$</span>, so I omitted them.)</p>
  <p>As you can see, when the target value <span class="math-container">$t$</span> is closer to the medium (<span class="math-container">$t=0.5$</span>), BCE is flatter around its minimum (<span class="math-container">$x \sim t$</span>). This means that BCE is less 'focal' when the target value is intermediate value.</p>
  <p>So, BCE suits your purpose when the edge values (<span class="math-container">$t=0$</span> and <span class="math-container">$t=1$</span>) are of special importance for you but the difference between intermediate values (<span class="math-container">$t=0.4$</span> and <span class="math-container">$t=0.5$</span>, for example) is not very important for you.</p>
  <p>On the other hand, when any value of target is equally important for you, then BCE will not be a good choice.  Another loss function, MSE for example, is better for you. <a href="https://i.sstatic.net/xIx00.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xIx00.png" alt="The binary cross entropy for several target values" /></a></p>
  <p>Note added:  If you use BCE for regression problems, it will be better to subtract <span class="math-container">$-x*\log(x) - (1-x)*\log(1-x)$</span> from the original BCE expression, so that the loss becomes zero when the prediction value coincides with the target one, <span class="math-container">$x=t$</span>.  This will not matter for backpropagation, but will be convenient to monitor the value.</p>
  <p>Note added:  After I submit this post, I came to think that we can tune how 'focal' the loss function is around its minimum, by simply multiplying an arbitrary factor of target values.  For example, we can tune BCE loss <span class="math-container">$L_{\rm BCE}(x,t)$</span> by making it <span class="math-container">$f(t)*L_{\rm BCE}(x,t)$</span> where <span class="math-container">$f(t)$</span> is whatever factor you want.  This factor tunes how focal the loss is around its minimum for each target value <span class="math-container">$t$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T06:25:29.753Z
Answers.LastActivityDate: 2021-03-11T08:40:38.120Z
Answers.LastEditDate: 2021-03-11T08:40:38.120Z
Answers.UserDisplayName: Toru Kikuchi
Answers.Body: <p>In a classification problem it's better to get higher error and higher error slope when we predict the label wrong.</p>
  <p>As you see in the graph by using cross-entropy you get high error when the algorithm predict a label wrong and small error when the prediacted label is close enough, so it helps us to separate the predicted classes better.</p>
  <p><a href="https://i.sstatic.net/CwHzl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CwHzl.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-08T15:19:27.263Z
Answers.Text: Zero is one and one is zero
Answers.UserDisplayName: Mr Tarsa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-16T18:22:44.650Z
Answers.LastActivityDate: 2020-07-23T12:56:51.987Z
Answers.LastEditDate: 2020-07-23T12:56:51.987Z
Answers.UserDisplayName: amin msh
Answers.Body: <p>As a rule of thumb, <a href="https://en.wikipedia.org/wiki/Mean_squared_error" rel="noreferrer">mean squared error (MSE)</a> is more appropriate for <a href="https://en.wikipedia.org/wiki/Regression_analysis" rel="noreferrer">regression</a> problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real number).  However, in principle, <a href="https://stats.stackexchange.com/q/46413/82135">you can use the MSE for classification problems</a> too (even though that may not be a good idea). MSE can be preceded by the <a href="https://en.wikipedia.org/wiki/Sigmoid_function" rel="noreferrer">sigmoid function</a>, which outputs a number <span class="math-container">$p \in [0, 1]$</span>, which can be interpreted as the probability of the input belonging to one of the classes, so the probability of the input belonging to the other class is <span class="math-container">$1 - p$</span>.</p>
  
  <p>Similarly, <a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="noreferrer">cross-entropy (CE)</a> is mainly used for <a href="https://en.wikipedia.org/wiki/Statistical_classification" rel="noreferrer">classification</a> problems, that is, problems where the output can belong to one of a discrete set of classes. The CE loss function is usually separately implemented for <a href="https://en.wikipedia.org/wiki/Binary_classification" rel="noreferrer">binary</a> and <a href="https://en.wikipedia.org/wiki/Multiclass_classification" rel="noreferrer">multi-class</a> classification problems. In the first case, it is called the binary cross-entropy (BCE), and, in the second case, it is called categorical cross-entropy (CCE). The CE requires its inputs to be distributions, so the CCE is usually preceded by a <a href="https://en.wikipedia.org/wiki/Softmax_function" rel="noreferrer">softmax function</a> (so that the resulting vector represents a probability distribution), while the BCE is usually preceded by a sigmoid. </p>
  
  <p>See also <a href="https://stats.stackexchange.com/q/288451/82135">Why is mean squared error the cross-entropy between the empirical distribution and a Gaussian model?</a> for more details about the relationship between the MSE and the cross-entropy. In case you use TensorFlow (TF) or Keras, see also <a href="https://stackoverflow.com/q/47034888/3924118">How to choose cross-entropy loss in TensorFlow?</a>, which gives you some guidelines for how to choose the appropriate TF implementation of the cross-entropy function for your (classification) problem. See also <a href="https://stats.stackexchange.com/q/260505/82135">Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions?</a> and <a href="https://stats.stackexchange.com/q/223256/82135">Does the cross-entropy cost make sense in the context of regression?</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-20T03:39:04.973Z
Answers.Text: tks, i have my thumb rule now, commonly, MSE for regression, CCE for classification
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-19T01:45:39.747Z
Answers.LastActivityDate: 2020-01-19T03:00:56.557Z
Answers.LastEditDate: 2020-01-19T03:00:56.557Z
Answers.UserDisplayName: nbro
Body: <p>In my code, I usually use the mean squared error (MSE), but the TensorFlow tutorials always use the categorical cross-entropy (CCE). Is the CCE loss function better than MSE? Or is it better only in certain cases?</p>
  
Tags: <machine-learning><comparison><objective-functions><mean-squared-error><categorical-crossentropy>
Title: In which cases is the categorical cross-entropy better than the mean squared error?
UserDisplayName: Dan D.
ThreadText: # In which cases is the categorical cross-entropy better than the mean squared error?
  
  (Asked by: username_0 on 2020-01-19)
  
  In my code, I usually use the mean squared error (MSE), but the TensorFlow tutorials always use the categorical cross-entropy (CCE). Is the CCE loss function better than MSE? Or is it better only in certain cases?
  
  **Tags:** machine-learning, comparison, objective-functions, mean-squared-error, categorical-crossentropy
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2020-01-19. Score: 8 | Accepted answer
  
  As a rule of thumb, [mean squared error (MSE)](https://en.wikipedia.org/wiki/Mean_squared_error) is more appropriate for [regression](https://en.wikipedia.org/wiki/Regression_analysis) problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real number). However, in principle, [you can use the MSE for classification problems](https://stats.stackexchange.com/q/46413/82135) too (even though that may not be a good idea). MSE can be preceded by the [sigmoid function](https://en.wikipedia.org/wiki/Sigmoid_function), which outputs a number $p \in [0, 1]$, which can be interpreted as the probability of the input belonging to one of the classes, so the probability of the input belonging to the other class is $1 - p$.
  
  Similarly, [cross-entropy (CE)](https://en.wikipedia.org/wiki/Cross_entropy) is mainly used for [classification](https://en.wikipedia.org/wiki/Statistical_classification) problems, that is, problems where the output can belong to one of a discrete set of classes. The CE loss function is usually separately implemented for [binary](https://en.wikipedia.org/wiki/Binary_classification) and [multi-class](https://en.wikipedia.org/wiki/Multiclass_classification) classification problems. In the first case, it is called the binary cross-entropy (BCE), and, in the second case, it is called categorical cross-entropy (CCE). The CE requires its inputs to be distributions, so the CCE is usually preceded by a [softmax function](https://en.wikipedia.org/wiki/Softmax_function) (so that the resulting vector represents a probability distribution), while the BCE is usually preceded by a sigmoid.
  
  See also [Why is mean squared error the cross-entropy between the empirical distribution and a Gaussian model?](https://stats.stackexchange.com/q/288451/82135) for more details about the relationship between the MSE and the cross-entropy. In case you use TensorFlow (TF) or Keras, see also [How to choose cross-entropy loss in TensorFlow?](https://stackoverflow.com/q/47034888/3924118), which gives you some guidelines for how to choose the appropriate TF implementation of the cross-entropy function for your (classification) problem. See also [Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions?](https://stats.stackexchange.com/q/260505/82135) and [Does the cross-entropy cost make sense in the context of regression?](https://stats.stackexchange.com/q/223256/82135).
  
  > **Comments:**
  > **username_0**: tks, i have my thumb rule now, commonly, MSE for regression, CCE for classification  
  
  ---
  
  ### Answer by username_1 on 2021-03-04. Score: 3
  
  We sometimes see that binary cross-entropy (BCE) loss is used for regression problems. This post is my opinion on using BCE for regression problems.
  
  The figure below is the plots of BCE, $-t\*\log(x) - (1-t)\*\log(1-x)$, for several target values $t = 0.0, 0.1, ..., 0.5$. (The plots for $t>0.5$ are mirror images of those for $t<0.5$, so I omitted them.)
  
  As you can see, when the target value $t$ is closer to the medium ($t=0.5$), BCE is flatter around its minimum ($x \sim t$). This means that BCE is less 'focal' when the target value is intermediate value.
  
  So, BCE suits your purpose when the edge values ($t=0$ and $t=1$) are of special importance for you but the difference between intermediate values ($t=0.4$ and $t=0.5$, for example) is not very important for you.
  
  On the other hand, when any value of target is equally important for you, then BCE will not be a good choice. Another loss function, MSE for example, is better for you. [![The binary cross entropy for several target values](https://i.sstatic.net/xIx00.png)](https://i.sstatic.net/xIx00.png)
  
  Note added: If you use BCE for regression problems, it will be better to subtract $-x\*\log(x) - (1-x)\*\log(1-x)$ from the original BCE expression, so that the loss becomes zero when the prediction value coincides with the target one, $x=t$. This will not matter for backpropagation, but will be convenient to monitor the value.
  
  Note added: After I submit this post, I came to think that we can tune how 'focal' the loss function is around its minimum, by simply multiplying an arbitrary factor of target values. For example, we can tune BCE loss $L\_{\rm BCE}(x,t)$ by making it $f(t)\*L\_{\rm BCE}(x,t)$ where $f(t)$ is whatever factor you want. This factor tunes how focal the loss is around its minimum for each target value $t$.
  
  
  ---
  
  ### Answer by username_2 on 2020-05-16. Score: 3
  
  In a classification problem it's better to get higher error and higher error slope when we predict the label wrong.
  
  As you see in the graph by using cross-entropy you get high error when the algorithm predict a label wrong and small error when the prediacted label is close enough, so it helps us to separate the predicted classes better.
  
  [![enter image description here](https://i.sstatic.net/CwHzl.png)](https://i.sstatic.net/CwHzl.png)
  
  > **Comments:**
  > **username_3**: Zero is one and one is zero  
  
  ---
  
  