Answers.Body: <p><strong>Optimizing the cross-entropy is equivalent to optimizing the log-likelihood of the parameters given the data, <span class="math-container">$\ell(\theta)$</span>, which is what we want, i.e. find the parameters that most likely generated the data</strong>.</p>
  <p>So, the <strong>likelihood</strong> is defined as <span class="math-container">$$\mathcal{L}(\theta) = P(y \mid x; \theta),$$</span>
  i.e. a function of the parameters <span class="math-container">$\theta$</span>.</p>
  <p>The log-likelihood is just the logarithm of the likelihood</p>
  <p><span class="math-container">$$\ell(\theta) = \log \mathcal{L}(\theta)$$</span></p>
  <p>To understand why we take the logarithm, read <a href="https://ai.stackexchange.com/a/32787/2444">this answer</a>.</p>
  <p>Now, for a fixed <span class="math-container">$\hat{\theta}$</span>, <span class="math-container">$\mathcal{L}(\hat{\theta}) = P(y \mid x; \hat{\theta})$</span> is a conditional <a href="https://ai.stackexchange.com/q/16826/2444">probability distribution</a>.</p>
  <p>For simplicity, let's assume that <span class="math-container">$\mathcal{L}(\hat{\theta}) = P(y \mid x; \hat{\theta})$</span> is a (conditional) <a href="https://en.wikipedia.org/wiki/Bernoulli_distribution" rel="nofollow noreferrer">Bernoulli distribution</a>, which is defined by a single parameter <span class="math-container">$\hat{p}$</span>, whose probability mass function (pmf) is defined as</p>
  <p><span class="math-container">$$\hat{p}^y (1-\hat{p})^{1-y},$$</span></p>
  <p>where <span class="math-container">$y = \{0, 1\}$</span>.</p>
  <p>Now, in the context of neural networks, we use a neural network to output <span class="math-container">$\hat{p}$</span>, i.e. an estimate of the parameter of the Bernoulli, which we plug into the cross-entropy loss function. So, if <span class="math-container">$f_\theta: \mathcal{X} \rightarrow [0, 1]$</span> is your neural network, then <span class="math-container">$f_{\hat{\theta}}(x) = \hat{p}$</span>, given the input-output pair <span class="math-container">$(x, y)$</span>.</p>
  <p>So, the Bernoulli pmf for a given pair <span class="math-container">$(x, y) \in \mathcal{D}$</span> can be written as</p>
  <p><span class="math-container">\begin{align}
  P(y \mid x; \hat{\theta})
  &amp;= \hat{p}^y (1-\hat{p})^{1-y} \\
  &amp;= f_{\hat{\theta}}(x)^y (1-f_{\hat{\theta}}(x))^{1-y} 
  \end{align}</span>
  So, without fixing the parameters, we can write the likelihood (which is not a probability distribution wrt <span class="math-container">$\theta$</span>) as follows</p>
  <p><span class="math-container">\begin{align}
  \mathcal{L}(\theta) 
  &amp;= f_{\theta}(x)^y (1-f_{\theta}(x))^{1-y}  \\
  \iff \ell(\theta) &amp;= \log \left( f_{\theta}(x)^y (1-f_{\theta}(x))^{1-y}\right)  \\
  &amp;= \log f_{\theta}(x)^y + \log \left(1-f_{\theta}(x)\right)^{1-y} \\
  &amp;= y \log f_{\theta}(x) + (1-y)\log \left(1-f_{\theta}(x)\right) 
  \end{align}</span></p>
  <p>which is our well-known <strong>binary cross-entropy</strong> for a single pair <span class="math-container">$(x, y)$</span> (note that only one of the addends above is not zero). For multiple pairs, you can apply the same reasoning, and you will see why we also use the log, i.e. it will allow you to simplify a few expressions in the derivation (specifically, multiplications will turn into sums, <a href="https://ai.stackexchange.com/a/32787/2444">which is nice numerically</a>).</p>
  <p>So, if we maximize <span class="math-container">$\mathcal{L}$</span>, with respect to <span class="math-container">$\theta$</span> (e.g. the parameters of your neural network), then we will find an estimate <span class="math-container">$\hat{\theta}$</span>, that produces the parameter <span class="math-container">$\hat{p}$</span>, such that this parameter can be used to construct/define the Bernoulli distribution that most <strong>likely</strong> (hence the name likelihood) produced our given dataset <span class="math-container">$\mathcal{D}$</span>.</p>
  <p>This reasoning also applies to the case where <span class="math-container">$P$</span> is categorical or even Gaussian distribution. So, actually, <strong>when you're minimizing the MSE, you're actually also optimizing the log-likelihood.</strong> It just turns out that the expression is different because Gaussians are not Bernoullis.</p>
  <p>I will not talk about the back-propagation, but it's just the chain rule. If the log-likelihood is differentiable, then you can use it in the context of back-propagation and GD. You can ask another question, if you're interested, but note that, in the general case, computing the partial derivatives can be a pain in the neck, so your question may be too broad. Find a very simple case.</p>
  <p>To conclude, don't dwell on the word <em>error</em>. Think in terms of probabilities, probability distributions, likelihoods, and why we can optimize likelihoods. Read my answer multiple times, if you didn't understand something.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-02T09:19:06.877Z
Answers.Text: Overall, I think I understand your point. However, It seems little complicated that "when you're minimizing the MSE, you're actually also optimizing the log-likelihood." Can you be more specific, please?
Answers.UserDisplayName: JAEMTO
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-02T09:20:47.167Z
Answers.Text: @JAEMTO I think I can show you mathematically why this is the case, if you ask another separate question. Maybe in your new question you can refer to this answer, so that people have some context.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-29T09:49:57.033Z
Answers.LastActivityDate: 2021-12-30T08:57:24.823Z
Answers.LastEditDate: 2021-12-30T08:57:24.823Z
Answers.UserDisplayName: nbro
Body: <p>The question looks foolish, but I think cross-entropy is somewhat weird as a cost function.</p>
  <p>As a cost function for linear regression, the mean square error <span class="math-container">$ \sum_{i=1}^{n} (y_i - (ax_i+b)) ^2$</span> seems quite reasonable, because it literally/directly measures the error between real value and predicted value.</p>
  <p>However, in the case of the cross-entropy, I do not understand what it is.</p>
  <p>For multi-class classification, for example, with 3 classes, the true target is <span class="math-container">$[ 0, 0, 1 ]$</span>, while the output of the model is <span class="math-container">$[ 0.2, 0.3, 0.5 ]$</span> (maybe with a softmax activation at the last layer). So, the error of it is: <span class="math-container">$C(x) = -(0*log(0.2) + 0*log(0.3) + 1*log(0.5))$</span>.</p>
  <p>It looks... I don't know, why is it an &quot;error?&quot; How can it be updated with backpropagation?</p>
  <p>Also, what is the objective of it? Maybe optimization, so maybe minimizing error? Then what happens?</p>
  
Tags: <neural-networks><objective-functions><backpropagation><linear-regression><cross-entropy>
Title: Why is the cross-entropy a cost function?
UserDisplayName: JAEMTO
ThreadText: # Why is the cross-entropy a cost function?
  
  (Asked by: username_0 on 2021-12-29)
  
  The question looks foolish, but I think cross-entropy is somewhat weird as a cost function.
  
  As a cost function for linear regression, the mean square error $ \sum\_{i=1}^{n} (y\_i - (ax\_i+b)) ^2$ seems quite reasonable, because it literally/directly measures the error between real value and predicted value.
  
  However, in the case of the cross-entropy, I do not understand what it is.
  
  For multi-class classification, for example, with 3 classes, the true target is $[ 0, 0, 1 ]$, while the output of the model is $[ 0.2, 0.3, 0.5 ]$ (maybe with a softmax activation at the last layer). So, the error of it is: $C(x) = -(0\*log(0.2) + 0\*log(0.3) + 1\*log(0.5))$.
  
  It looks... I don't know, why is it an "error?" How can it be updated with backpropagation?
  
  Also, what is the objective of it? Maybe optimization, so maybe minimizing error? Then what happens?
  
  **Tags:** neural-networks, objective-functions, backpropagation, linear-regression, cross-entropy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-29. Score: 2 | Accepted answer
  
  **Optimizing the cross-entropy is equivalent to optimizing the log-likelihood of the parameters given the data, $\ell(\theta)$, which is what we want, i.e. find the parameters that most likely generated the data**.
  
  So, the **likelihood** is defined as $$\mathcal{L}(\theta) = P(y \mid x; \theta),$$
  i.e. a function of the parameters $\theta$.
  
  The log-likelihood is just the logarithm of the likelihood
  
  $$\ell(\theta) = \log \mathcal{L}(\theta)$$
  
  To understand why we take the logarithm, read [this answer](https://ai.stackexchange.com/a/32787/2444).
  
  Now, for a fixed $\hat{\theta}$, $\mathcal{L}(\hat{\theta}) = P(y \mid x; \hat{\theta})$ is a conditional [probability distribution](https://ai.stackexchange.com/q/16826/2444).
  
  For simplicity, let's assume that $\mathcal{L}(\hat{\theta}) = P(y \mid x; \hat{\theta})$ is a (conditional) [Bernoulli distribution](https://en.wikipedia.org/wiki/Bernoulli_distribution), which is defined by a single parameter $\hat{p}$, whose probability mass function (pmf) is defined as
  
  $$\hat{p}^y (1-\hat{p})^{1-y},$$
  
  where $y = \{0, 1\}$.
  
  Now, in the context of neural networks, we use a neural network to output $\hat{p}$, i.e. an estimate of the parameter of the Bernoulli, which we plug into the cross-entropy loss function. So, if $f\_\theta: \mathcal{X} \rightarrow [0, 1]$ is your neural network, then $f\_{\hat{\theta}}(x) = \hat{p}$, given the input-output pair $(x, y)$.
  
  So, the Bernoulli pmf for a given pair $(x, y) \in \mathcal{D}$ can be written as
  
  \begin{align}
  P(y \mid x; \hat{\theta})
  &= \hat{p}^y (1-\hat{p})^{1-y} \\
  &= f\_{\hat{\theta}}(x)^y (1-f\_{\hat{\theta}}(x))^{1-y}
  \end{align}
  So, without fixing the parameters, we can write the likelihood (which is not a probability distribution wrt $\theta$) as follows
  
  \begin{align}
  \mathcal{L}(\theta)
  &= f\_{\theta}(x)^y (1-f\_{\theta}(x))^{1-y} \\
  \iff \ell(\theta) &= \log \left( f\_{\theta}(x)^y (1-f\_{\theta}(x))^{1-y}\right) \\
  &= \log f\_{\theta}(x)^y + \log \left(1-f\_{\theta}(x)\right)^{1-y} \\
  &= y \log f\_{\theta}(x) + (1-y)\log \left(1-f\_{\theta}(x)\right)
  \end{align}
  
  which is our well-known **binary cross-entropy** for a single pair $(x, y)$ (note that only one of the addends above is not zero). For multiple pairs, you can apply the same reasoning, and you will see why we also use the log, i.e. it will allow you to simplify a few expressions in the derivation (specifically, multiplications will turn into sums, [which is nice numerically](https://ai.stackexchange.com/a/32787/2444)).
  
  So, if we maximize $\mathcal{L}$, with respect to $\theta$ (e.g. the parameters of your neural network), then we will find an estimate $\hat{\theta}$, that produces the parameter $\hat{p}$, such that this parameter can be used to construct/define the Bernoulli distribution that most **likely** (hence the name likelihood) produced our given dataset $\mathcal{D}$.
  
  This reasoning also applies to the case where $P$ is categorical or even Gaussian distribution. So, actually, **when you're minimizing the MSE, you're actually also optimizing the log-likelihood.** It just turns out that the expression is different because Gaussians are not Bernoullis.
  
  I will not talk about the back-propagation, but it's just the chain rule. If the log-likelihood is differentiable, then you can use it in the context of back-propagation and GD. You can ask another question, if you're interested, but note that, in the general case, computing the partial derivatives can be a pain in the neck, so your question may be too broad. Find a very simple case.
  
  To conclude, don't dwell on the word *error*. Think in terms of probabilities, probability distributions, likelihoods, and why we can optimize likelihoods. Read my answer multiple times, if you didn't understand something.
  
  > **Comments:**
  > **username_0**: Overall, I think I understand your point. However, It seems little complicated that "when you're minimizing the MSE, you're actually also optimizing the log-likelihood." Can you be more specific, please?  
  > **username_1**: @JAEMTO I think I can show you mathematically why this is the case, if you ask another separate question. Maybe in your new question you can refer to this answer, so that people have some context.  
  
  ---
  
  