Answers.Body: <p>It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of samples. In the second case you have multiple classes and in the current form it's only for a single sample.</p>
  <p>In the first case there is only one output, if you had two outputs it would have been
  <span class="math-container">\begin{equation}
  -\frac{1}{N} \sum_{n=1}^N \sum_{j=1}^2 y_{n,j} \log(\hat y_{n,j})
  \end{equation}</span>
  where <span class="math-container">$n$</span> iterates over batch samples, and <span class="math-container">$j$</span> over two classes. The reason why it was written like that is that if you have two classes you can have only one output because you can immediately conclude about probability of the second class if you have the probability of the first class, it would simply be <span class="math-container">$p_1 = 1-p_0$</span>.</p>
  <p>In the second case, with batch samples included, it would be
  <span class="math-container">\begin{equation}
   -\frac{1}{N} \sum_{n=1}^N \sum_{j=1}^c y_{n,j} \log(\hat y_{n,j})
  \end{equation}</span>
  wheren <span class="math-container">$n$</span> iterates over batch samples, and <span class="math-container">$j$</span> over <span class="math-container">$c$</span> output classes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-24T09:07:50.660Z
Answers.LastActivityDate: 2020-10-24T09:07:50.660Z
Answers.UserDisplayName: Brale
Body: <p>I have seen 2 forms of softmax cross-entropy loss and are confused by the two. Which one is the right one?
  For example in this <a href="https://www.quora.com/Is-the-softmax-loss-the-same-as-the-cross-entropy-loss" rel="nofollow noreferrer">Quora answer</a>, there are 2 answers:</p>
  <ol>
  <li><p><span class="math-container">$L(\mathbf{w})=\frac{1}{N} \sum_{n=1}^{N} H\left(p_{n}, q_{n}\right)=-\frac{1}{N} \sum_{n=1}^{N}\left[y_{n} \log \hat{y}_{n}+\left(1-y_{n}\right) \log \left(1-\hat{y}_{n}\right)\right]$</span></p>
  </li>
  <li><p><span class="math-container">$\mathrm{L}(y, \hat{y})=-\Sigma y(i) \log \hat{y}(i)$</span>, which is only the first part of the version one.</p>
  </li>
  </ol>
  
Tags: <machine-learning><comparison><cross-entropy><softmax><categorical-crossentropy>
Title: Why are there two versions of softmax cross entropy? Which one to use in what situation?
UserDisplayName: Herbert
ThreadText: # Why are there two versions of softmax cross entropy? Which one to use in what situation?
  
  (Asked by: username_0 on 2020-10-24)
  
  I have seen 2 forms of softmax cross-entropy loss and are confused by the two. Which one is the right one?
  For example in this [Quora answer](https://www.quora.com/Is-the-softmax-loss-the-same-as-the-cross-entropy-loss), there are 2 answers:
  
  1. $L(\mathbf{w})=\frac{1}{N} \sum\_{n=1}^{N} H\left(p\_{n}, q\_{n}\right)=-\frac{1}{N} \sum\_{n=1}^{N}\left[y\_{n} \log \hat{y}\_{n}+\left(1-y\_{n}\right) \log \left(1-\hat{y}\_{n}\right)\right]$
  2. $\mathrm{L}(y, \hat{y})=-\Sigma y(i) \log \hat{y}(i)$, which is only the first part of the version one.
  
  **Tags:** machine-learning, comparison, cross-entropy, softmax, categorical-crossentropy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-24. Score: 5 | Accepted answer
  
  It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of samples. In the second case you have multiple classes and in the current form it's only for a single sample.
  
  In the first case there is only one output, if you had two outputs it would have been
  \begin{equation}
  -\frac{1}{N} \sum\_{n=1}^N \sum\_{j=1}^2 y\_{n,j} \log(\hat y\_{n,j})
  \end{equation}
  where $n$ iterates over batch samples, and $j$ over two classes. The reason why it was written like that is that if you have two classes you can have only one output because you can immediately conclude about probability of the second class if you have the probability of the first class, it would simply be $p\_1 = 1-p\_0$.
  
  In the second case, with batch samples included, it would be
  \begin{equation}
  -\frac{1}{N} \sum\_{n=1}^N \sum\_{j=1}^c y\_{n,j} \log(\hat y\_{n,j})
  \end{equation}
  wheren $n$ iterates over batch samples, and $j$ over $c$ output classes.
  
  
  ---
  
  