Answers.Body: <p>Let's first recap the definition of the <strong>binary cross-entropy (BCE)</strong> and the <strong>categorical cross-entropy (CCE)</strong>.</p>
  <p>Here's the <strong>BCE</strong> (<a href="http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf#page=226" rel="nofollow noreferrer">equation 4.90 from this book</a>)</p>
  <p><span class="math-container">$$-\sum_{n=1}^{N}\left( t_{n} \ln y_{n}+\left(1-t_{n}\right) \ln \left(1-y_{n}\right)\right) \label{1}\tag{1},$$</span></p>
  <p>where</p>
  <ul>
  <li><span class="math-container">$t_{n} \in\{0,1\}$</span> is the target</li>
  <li><span class="math-container">$y_n \in [0, 1]$</span> is the prediction (as produced by the <em>sigmoid</em>), so <span class="math-container">$1 - y_n$</span> is the probability that <span class="math-container">$n$</span> belongs to the other class</li>
  </ul>
  <p>Here's the <strong>CCE</strong> (<a href="http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf#page=229" rel="nofollow noreferrer">equation 4.108</a>)</p>
  <p><span class="math-container">$$
  -\sum_{n=1}^{N} \sum_{k=1}^{K} t_{n k} \ln y_{n k}\label{2}\tag{2},
  $$</span>
  where</p>
  <ul>
  <li><span class="math-container">$t_{n k} = \{0, 1\}$</span> is the target of input <span class="math-container">$n$</span> for class <span class="math-container">$k$</span>, i.e. it's <span class="math-container">$1$</span> when <span class="math-container">$n$</span> is labelled as <span class="math-container">$k$</span> and <span class="math-container">$0$</span> otherwise (so it's <span class="math-container">$0$</span> for all <span class="math-container">$K$</span> except for one of them)</li>
  <li><span class="math-container">$y_{n k}$</span> is the probability that <span class="math-container">$n$</span> belongs to the class <span class="math-container">$k$</span>, as produced by the <em>softmax</em> function</li>
  </ul>
  <p>Let <span class="math-container">$K=2$</span>. Then equation \ref{2} becomes</p>
  <p><span class="math-container">$$
  -\sum_{n=1}^{N} \sum_{k=1}^{2} t_{n k} \ln y_{n k} =
  -\sum_{n=1}^{N} \left( t_{n 1} \ln y_{n 1} + t_{n 2} \ln y_{n 2} \right)
  \label{3}\tag{3} 
  $$</span></p>
  <p>So, if <span class="math-container">$[y_{n 1},  y_{n 2}]$</span> is a <a href="https://en.wikipedia.org/wiki/Probability_vector" rel="nofollow noreferrer">probability vector</a> (which is the case if you use the softmax as the activation function of the last layer), then, <em>in theory</em>, the BCE and CCE are equivalent in the case of <strong>binary classification</strong>. <em>In practice</em>, if you are using TensorFlow, to choose the most suitable loss function for your problem, you could take a look at <a href="https://stackoverflow.com/a/47034889/3924118">this answer</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-17T00:01:08.173Z
Answers.LastActivityDate: 2021-03-17T08:59:39.590Z
Answers.LastEditDate: 2021-03-17T08:59:39.590Z
Answers.UserDisplayName: nbro
Body: <p>I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other.</p>
  <p>For this reason, in my neural network, I have specified a softmax activation in the last layer with 2 outputs and a categorical crossentropy for the loss. Using tensorflow:</p>
  <pre><code>model=tf.keras.models.Sequential()
  model.add(tf.keras.layers.Dense(units=64, input_shape=(100,), activation='relu'))
  model.add(tf.keras.layers.Dropout(0.4))
  model.add(tf.keras.layers.Dense(units=32, activation='relu'))
  model.add(tf.keras.layers.Dropout(0.4))
  model.add(tf.keras.layers.Dense(units=2, activation='softmax'))
  
  model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
  </code></pre>
  <p>Here are my questions.</p>
  <ol>
  <li><p>If the sigmoid is equivalent to the softmax, firstly is it valid to specify 2 units with a softmax and categorical_crossentropy?</p>
  </li>
  <li><p>Is it the same as using binary_crossentropy (<em>in this particular use case</em>) with 2 classes and a sigmoid activation, and if so why?</p>
  </li>
  </ol>
  <p>I know that for non-exclusive multi-label problems with more than 2 classes, a binary_crossentropy with a sigmoid activation is used, why is the non-exclusivity about the multi-label case uniquely different from a binary classification with 2 classes only, with 1 (class 0 or class 1) output and a sigmoid with binary_crossentropy loss.</p>
  
Tags: <binary-classification><sigmoid><softmax><categorical-crossentropy><binary-crossentropy>
Title: Is it appropriate to use a softmax activation with a categorical crossentropy loss?
UserDisplayName: user9317212
ThreadText: # Is it appropriate to use a softmax activation with a categorical crossentropy loss?
  
  (Asked by: username_0 on 2021-03-16)
  
  I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other.
  For this reason, in my neural network, I have specified a softmax activation in the last layer with 2 outputs and a categorical crossentropy for the loss. Using tensorflow:
  model=tf.keras.models.Sequential()
  model.add(tf.keras.layers.Dense(units=64, input_shape=(100,), activation='relu'))
  model.add(tf.keras.layers.Dropout(0.4))
  model.add(tf.keras.layers.Dense(units=32, activation='relu'))
  model.add(tf.keras.layers.Dropout(0.4))
  model.add(tf.keras.layers.Dense(units=2, activation='softmax'))
  
  model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
  
  Here are my questions.
  
  If the sigmoid is equivalent to the softmax, firstly is it valid to specify 2 units with a softmax and categorical_crossentropy?
  
  Is it the same as using binary_crossentropy (in this particular use case) with 2 classes and a sigmoid activation, and if so why?
  
  
  I know that for non-exclusive multi-label problems with more than 2 classes, a binary_crossentropy with a sigmoid activation is used, why is the non-exclusivity about the multi-label case uniquely different from a binary classification with 2 classes only, with 1 (class 0 or class 1) output and a sigmoid with binary_crossentropy loss.
  
  **Tags:** binary-classification, sigmoid, softmax, categorical-crossentropy, binary-crossentropy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-17. Score: 2 | Accepted answer
  
  Let's first recap the definition of the **binary cross-entropy (BCE)** and the **categorical cross-entropy (CCE)**.
  
  Here's the **BCE** ([equation 4.90 from this book](http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf#page=226))
  
  $$-\sum\_{n=1}^{N}\left( t\_{n} \ln y\_{n}+\left(1-t\_{n}\right) \ln \left(1-y\_{n}\right)\right) \label{1}\tag{1},$$
  
  where
  
  * $t\_{n} \in\{0,1\}$ is the target
  * $y\_n \in [0, 1]$ is the prediction (as produced by the *sigmoid*), so $1 - y\_n$ is the probability that $n$ belongs to the other class
  
  Here's the **CCE** ([equation 4.108](http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf#page=229))
  
  $$
  -\sum\_{n=1}^{N} \sum\_{k=1}^{K} t\_{n k} \ln y\_{n k}\label{2}\tag{2},
  $$
  where
  
  * $t\_{n k} = \{0, 1\}$ is the target of input $n$ for class $k$, i.e. it's $1$ when $n$ is labelled as $k$ and $0$ otherwise (so it's $0$ for all $K$ except for one of them)
  * $y\_{n k}$ is the probability that $n$ belongs to the class $k$, as produced by the *softmax* function
  
  Let $K=2$. Then equation \ref{2} becomes
  
  $$
  -\sum\_{n=1}^{N} \sum\_{k=1}^{2} t\_{n k} \ln y\_{n k} =
  -\sum\_{n=1}^{N} \left( t\_{n 1} \ln y\_{n 1} + t\_{n 2} \ln y\_{n 2} \right)
  \label{3}\tag{3}
  $$
  
  So, if $[y\_{n 1}, y\_{n 2}]$ is a [probability vector](https://en.wikipedia.org/wiki/Probability_vector) (which is the case if you use the softmax as the activation function of the last layer), then, *in theory*, the BCE and CCE are equivalent in the case of **binary classification**. *In practice*, if you are using TensorFlow, to choose the most suitable loss function for your problem, you could take a look at [this answer](https://stackoverflow.com/a/47034889/3924118).
  
  
  ---
  
  