Answers.Body: <p>There's no essential difference, information theoretically speaking any loss consisting of a negative log-likelihood (NLL) is a <a href="https://en.wikipedia.org/wiki/Cross-entropy" rel="nofollow noreferrer">cross entropy</a> between the empirical distribution defined by the training set and the probability distribution defined by a specific model. Even the usual mean squared error (MSE) can be shown as the cross entropy between the empirical distribution and a Gaussian distribution.</p>
  <blockquote>
  <p>For discrete probability distributions <span class="math-container">$p$</span> and <span class="math-container">$q$</span> with the same support <span class="math-container">$\mathcal {X}$</span>, this means <span class="math-container">${H(p,q)=-\sum _{x\in {\mathcal {X}}}p(x)\,\log q(x)}$</span>. (Eq.1)</p>
  </blockquote>
  <p>Therefore clearly binary cross entropy is just a special case of above equation for  discrete probability distributions where your <span class="math-container">$k_i$</span> is the ground truth label for the <span class="math-container">$i$</span>-th example, say <span class="math-container">$0$</span> for negative case and <span class="math-container">$1$</span> for positive case here, acting as the empirical distribution of the training set, and your <span class="math-container">$p_i$</span> is the predicted probability assigned by the model to the positive class for the <span class="math-container">$i$</span>-th example. And now we can derive the binary cross-entropy loss using the maximum likelihood principle and the Binomial distribution to show their relation.</p>
  <p>Since each example of Binomial data-generating process is independent, the likelihood function can be expressed as the product of the probabilities of each individual example <span class="math-container">$L=∏_{i=1}^Np_i^{k_i}(1−p_i)^{1−k_i}$</span>. To simplify computations and prevent numerical underflow, it's common to work with the logarithm of the likelihood function, known as the log-likelihood <span class="math-container">$\log L=\sum_{i=1}^N[k_i\log(p_i)+(1−k_i)\log(1−p_i)]$</span>. Finally maximizing the log-likelihood for a fixed <span class="math-container">$N$</span> is equivalent to minimizing the negative log-likelihood (NLL) which leads to your binary cross-entropy loss function <span class="math-container">$-\sum [k_i \log(p_i)+(1-k_i) \log(1-p_i)]$</span> which measures the discrepancy between the predicted probabilities and the true labels.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T12:48:24.320Z
Answers.Text: Thank you for answering. your perfect reply help me! and i have additional questions. First, how can i proove that each data assumes an iid condition.
  Second, can you explain how MSE can be shown as C.E? have a good day.
Answers.UserDisplayName: Shin Joong Hyun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T22:40:50.533Z
Answers.Text: iid example condition is assumed in most supervised/unsupervised learning, and my answer is unsupervised to learn a Bernoulli distribution for a single trial. For any iid simple random sampling hypothesis claimed in a real experiment, you can turn to statistics and employ [Pearson's chi-squared test](https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test) or Fisher's exact test especially when samples are small. Of course if you training data is dependent your trained binomial model won't converge or with large test error. BCE can also relate to logistic regression supervised learning.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T23:21:45.377Z
Answers.Text: For MSE it's a different question but it's very similar, assuming iid examples sampled from a single Gaussian distribution, you do the same steps as shown in my answer except replacing the single trial Bernoulli distribution with Gaussian whose pdf has an obvious MSE like term $(x-\mu)^2$. Finally the derived NLL will contain MSE with some scaling and offset term of logarithm of variance which is assumed to be a constant since we usually only try to learn its mean (same as above case's goal to learn Bernoulli distribution's mean as the positive probability $p$).
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-22T22:41:38.263Z
Answers.LastActivityDate: 2024-02-22T22:41:38.263Z
Answers.UserDisplayName: cinch
Answers.Body: <p>The cross entropy loss with 2 classes is equivalent to the binary cross entropy loss:</p>
  <p><span class="math-container">$$
  \begin{equation}
  H_p(q) = - \sum_{i=1}^2 [q_i \log p_i] = q_1 \log p_1 + q_2 \log p_2 = q_1 \log p_1 + (1 - q_1) \log (1 - p_1)
  \end{equation}
  $$</span></p>
  <p>The last equality follows from the fact that <span class="math-container">$q$</span> and <span class="math-container">$p$</span> are discrete probability distributions with support <span class="math-container">$\{1, 2\}$</span>, so <span class="math-container">$q_1 = 1 - q_2$</span> and <span class="math-container">$p_1 = 1 - p_2$</span>.</p>
  <p>The summation in your binary cross entropy loss is there because you are summing over the entire batch of data points, while the cross entropy formula is the loss on a single data point with prediction <span class="math-container">$p \in [0,1]^2$</span> and ground truth <span class="math-container">$q \in \{0,1\}^2$</span>. If you have soft ground truth, for example if you have <a href="https://paperswithcode.com/method/label-smoothing" rel="nofollow noreferrer">label smoothing</a> then in general <span class="math-container">$q \in [0,1]^2$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T12:53:07.833Z
Answers.Text: Thank you for answering. After all, binary cross entorpy is equivalent to cross entorpy. have a nice day!!
Answers.UserDisplayName: Shin Joong Hyun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-22T12:17:00.363Z
Answers.LastActivityDate: 2024-02-22T12:17:00.363Z
Answers.UserDisplayName: c p
Body: <p>why cross entropy formulation is different with binary cross entropy?</p>
  <p>cross entorpy loss</p>
  <p><span class="math-container">$$
  H_p(q) = -\sum_{q_i}^C [q_i \log(p_i)]
  $$</span></p>
  <p>binary cross entorpy</p>
  <p><span class="math-container">$$
   -\sum [k_i \log(p_i)+(1-k_i) \log(1-p_i)]
  $$</span>
  <span class="math-container">$k_i \in \{0,1\}$</span></p>
  <p>&quot;Where did the Binary Cross-Entropy Loss Function come from?&quot;
  Relation between Binary Cross Entropy and Binomial Distribution</p>
  
Tags: <cross-entropy>
Title: Why is the cross entropy formula different from the binary cross entropy?
UserDisplayName: Shin Joong Hyun
ThreadText: # Why is the cross entropy formula different from the binary cross entropy?
  
  (Asked by: username_0 on 2024-02-22)
  
  why cross entropy formulation is different with binary cross entropy?
  
  cross entorpy loss
  
  $$
  H\_p(q) = -\sum\_{q\_i}^C [q\_i \log(p\_i)]
  $$
  
  binary cross entorpy
  
  $$
  -\sum [k\_i \log(p\_i)+(1-k\_i) \log(1-p\_i)]
  $$
  $k\_i \in \{0,1\}$
  
  "Where did the Binary Cross-Entropy Loss Function come from?"
  Relation between Binary Cross Entropy and Binomial Distribution
  
  **Tags:** cross-entropy
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-02-22. Score: 1
  
  There's no essential difference, information theoretically speaking any loss consisting of a negative log-likelihood (NLL) is a [cross entropy](https://en.wikipedia.org/wiki/Cross-entropy) between the empirical distribution defined by the training set and the probability distribution defined by a specific model. Even the usual mean squared error (MSE) can be shown as the cross entropy between the empirical distribution and a Gaussian distribution.
  
  > For discrete probability distributions $p$ and $q$ with the same support $\mathcal {X}$, this means ${H(p,q)=-\sum \_{x\in {\mathcal {X}}}p(x)\,\log q(x)}$. (Eq.1)
  
  Therefore clearly binary cross entropy is just a special case of above equation for discrete probability distributions where your $k\_i$ is the ground truth label for the $i$-th example, say $0$ for negative case and $1$ for positive case here, acting as the empirical distribution of the training set, and your $p\_i$ is the predicted probability assigned by the model to the positive class for the $i$-th example. And now we can derive the binary cross-entropy loss using the maximum likelihood principle and the Binomial distribution to show their relation.
  
  Since each example of Binomial data-generating process is independent, the likelihood function can be expressed as the product of the probabilities of each individual example $L=∏\_{i=1}^Np\_i^{k\_i}(1−p\_i)^{1−k\_i}$. To simplify computations and prevent numerical underflow, it's common to work with the logarithm of the likelihood function, known as the log-likelihood $\log L=\sum\_{i=1}^N[k\_i\log(p\_i)+(1−k\_i)\log(1−p\_i)]$. Finally maximizing the log-likelihood for a fixed $N$ is equivalent to minimizing the negative log-likelihood (NLL) which leads to your binary cross-entropy loss function $-\sum [k\_i \log(p\_i)+(1-k\_i) \log(1-p\_i)]$ which measures the discrepancy between the predicted probabilities and the true labels.
  
  > **Comments:**
  > **username_0**: Thank you for answering. your perfect reply help me! and i have additional questions. First, how can i proove that each data assumes an iid condition. Second, can you explain how MSE can be shown as C.E? have a good day.  
  > **username_1**: iid example condition is assumed in most supervised/unsupervised learning, and my answer is unsupervised to learn a Bernoulli distribution for a single trial. For any iid simple random sampling hypothesis claimed in a real experiment, you can turn to statistics and employ [Pearson's chi-squared test](https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test) or Fisher's exact test especially when samples are small. Of course if you training data is dependent your trained binomial model won't converge or with large test error. BCE can also relate to logistic regression supervised learning.  
  > **username_1**: For MSE it's a different question but it's very similar, assuming iid examples sampled from a single Gaussian distribution, you do the same steps as shown in my answer except replacing the single trial Bernoulli distribution with Gaussian whose pdf has an obvious MSE like term $(x-\mu)^2$. Finally the derived NLL will contain MSE with some scaling and offset term of logarithm of variance which is assumed to be a constant since we usually only try to learn its mean (same as above case's goal to learn Bernoulli distribution's mean as the positive probability $p$).  
  
  ---
  
  ### Answer by username_2 on 2024-02-22. Score: 0
  
  The cross entropy loss with 2 classes is equivalent to the binary cross entropy loss:
  
  $$
  \begin{equation}
  H\_p(q) = - \sum\_{i=1}^2 [q\_i \log p\_i] = q\_1 \log p\_1 + q\_2 \log p\_2 = q\_1 \log p\_1 + (1 - q\_1) \log (1 - p\_1)
  \end{equation}
  $$
  
  The last equality follows from the fact that $q$ and $p$ are discrete probability distributions with support $\{1, 2\}$, so $q\_1 = 1 - q\_2$ and $p\_1 = 1 - p\_2$.
  
  The summation in your binary cross entropy loss is there because you are summing over the entire batch of data points, while the cross entropy formula is the loss on a single data point with prediction $p \in [0,1]^2$ and ground truth $q \in \{0,1\}^2$. If you have soft ground truth, for example if you have [label smoothing](https://paperswithcode.com/method/label-smoothing) then in general $q \in [0,1]^2$.
  
  > **Comments:**
  > **username_0**: Thank you for answering. After all, binary cross entorpy is equivalent to cross entorpy. have a nice day!!  
  
  ---
  
  