Answers.Body: <p>Both the <strong>sparse categorical cross-entropy (SCE)</strong> and the <strong>categorical cross-entropy (CCE)</strong> can be greater than <span class="math-container">$1$</span>. By the way, they are the same exact loss function: the only difference is really the implementation, where the SCE assumes that the labels (or classes) are given as integers, while the CCE assumes that the labels are given as one-hot vectors.</p>
  <p>Here is the explanation with examples.</p>
  <p>Let <span class="math-container">$(x, y) \in D$</span> be an input-output pair from the labelled dataset <span class="math-container">$D$</span>, where <span class="math-container">$x$</span> is the input and <span class="math-container">$y$</span> is the ground-truth class/label for <span class="math-container">$x$</span>, which is an integer between <span class="math-container">$0$</span> and <span class="math-container">$C-1$</span>. Let's suppose that your neural network <span class="math-container">$f$</span> produces a probability vector <span class="math-container">$f(x) = \hat{y} \in [0, 1]^C$</span> (e.g. with a softmax), where <span class="math-container">$\hat{y}_i \in [0, 1]$</span> is the <span class="math-container">$i$</span>th element of <span class="math-container">$\hat{y}$</span>.</p>
  <p>The formula for SCE is (which is consistent with <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses/sparse_categorical_crossentropy" rel="nofollow noreferrer">the TensorFlow implementation of this loss function</a>)</p>
  <p><span class="math-container">$$
  \text{SCE}(y, \hat{y}) = - \ln (\hat{y}_{y}) \label{1}\tag{1},
  $$</span></p>
  <p>where <span class="math-container">$\hat{y}_{y}$</span> is the <span class="math-container">$y$</span>th element of the output probability vector <span class="math-container">$\hat{y}$</span> that corresponds to the probability that <span class="math-container">$x$</span> belongs to class <span class="math-container">$y$</span>, according to <span class="math-container">$f$</span>.</p>
  <p>Actually, the equation \ref{1} is also the definition of the CCE with one-hot vectors as targets (which behave as <a href="https://www.statlect.com/fundamentals-of-probability/indicator-functions" rel="nofollow noreferrer">indicator functions</a>). The only difference between CCE and SSE is really just the representation of the targets, which can slightly change the implementation under the hood. Moreover, note that this is the definition of the CE for only <span class="math-container">$1$</span> training pair. If you have multiple pairs, you have to compute the CE for all pairs, then average these CEs (for a reference, see <a href="http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf#page=229" rel="nofollow noreferrer">equation 4.108, section 4.3.4 <em>Multiclass logistic regression</em> of Bishop's book <em>PRML</em></a>).</p>
  <p>Let's have a look at a concrete example with concrete numbers. Let <span class="math-container">$C=5$</span>, <span class="math-container">$y = 3$</span>, <span class="math-container">$\hat{y} = [0.2, 0.2, 0.1, 0.4, 0.1]$</span>, then the SCE is</p>
  <p><span class="math-container">\begin{align}
  \text{SCE}(y, \hat{y}) 
  &amp;= 
  - \ln (0.4) \approx 0.92,
  \end{align}</span></p>
  <p>If <span class="math-container">$\hat{y} = [0.2, 0.2, 0.2, 0.1, 0.3]$</span>, so <span class="math-container">$\hat{y}_{y} = 0.1$</span>, and we still have <span class="math-container">$y = 3$</span>, then the CCE is <span class="math-container">$2.3 &gt; 1$</span>.</p>
  <p>You can execute this Python code to check yourself.</p>
  <pre><code>import numpy as np
  import tensorflow as tf # Install TensorFlow 2.3!
  
  y = 1
  y_true = [3] # sparse label (integer)
  y_true2 = [0, 0, 0, 1, 0] # one-hot vector
  
  for y_y in [0.4, 0.1]:
      sce_np = -(y * np.log(y_y))
      print(&quot;SCE (NumPy) =&quot;, sce_np)
  
  y_preds = [[0.2, 0.2, 0.1, 0.4, 0.1],
             [0.2, 0.2, 0.2, 0.1, 0.3]]
  for y_pred in y_preds:
      sce_tf = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)
      cce_tf = tf.keras.losses.categorical_crossentropy(y_true2, y_pred)
  
      print(&quot;SCE (TensorFlow) =&quot;, sce_tf)
      print(&quot;CCE (TensorFlow) =&quot;, cce_tf)
  </code></pre>
  <p>To answer the following question more directly.</p>
  <blockquote>
  <p>However, while training, my loss value is greater than one (almost equal to 1.2) in the first epoch, but until epoch 5 it comes near 0.8. Is it normal? If not, how can I solve this?</p>
  </blockquote>
  <p>Yes. It can happen, as explained above. (However, this does not mean that you do not have mistakes in your code.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-18T18:34:07.850Z
Answers.LastActivityDate: 2020-11-19T12:00:53.390Z
Answers.LastEditDate: 2020-11-19T12:00:53.390Z
Answers.UserDisplayName: nbro
Body: <p>I am using AlexNet CNN to classify my dataset which contains 10 classes and 1000 data for each class, with 60-30-10, splits for train, validation, and test. I used different batch sizes, learning rates, activation functions, and initializers. I'm using the sparse categorical cross-entropy loss function.</p>
  <p>However, while training, my loss value is greater than one (almost equal to 1.2) in the first epoch, but until epoch 5 it comes near 0.8.
  Is it normal? If not, how can I solve this?</p>
  
Tags: <convolutional-neural-networks><training><object-recognition><categorical-crossentropy><alexnet>
Title: Can the (sparse) categorical cross-entropy be greater than one?
UserDisplayName: SahaTib
ThreadText: # Can the (sparse) categorical cross-entropy be greater than one?
  
  (Asked by: username_0 on 2020-11-17)
  
  I am using AlexNet CNN to classify my dataset which contains 10 classes and 1000 data for each class, with 60-30-10, splits for train, validation, and test. I used different batch sizes, learning rates, activation functions, and initializers. I'm using the sparse categorical cross-entropy loss function.
  
  However, while training, my loss value is greater than one (almost equal to 1.2) in the first epoch, but until epoch 5 it comes near 0.8.
  Is it normal? If not, how can I solve this?
  
  **Tags:** convolutional-neural-networks, training, object-recognition, categorical-crossentropy, alexnet
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-18. Score: 3 | Accepted answer
  
  Both the sparse categorical cross-entropy (SCE) and the categorical cross-entropy (CCE) can be greater than $1$. By the way, they are the same exact loss function: the only difference is really the implementation, where the SCE assumes that the labels (or classes) are given as integers, while the CCE assumes that the labels are given as one-hot vectors.
  Here is the explanation with examples.
  Let $(x, y) \in D$ be an input-output pair from the labelled dataset $D$, where $x$ is the input and $y$ is the ground-truth class/label for $x$, which is an integer between $0$ and $C-1$. Let's suppose that your neural network $f$ produces a probability vector $f(x) = \hat{y} \in [0, 1]^C$ (e.g. with a softmax), where $\hat{y}_i \in [0, 1]$ is the $i$th element of $\hat{y}$.
  The formula for SCE is (which is consistent with the TensorFlow implementation of this loss function)
  $$
  \text{SCE}(y, \hat{y}) = - \ln (\hat{y}_{y}) \label{1}\tag{1},
  $$
  where $\hat{y}_{y}$ is the $y$th element of the output probability vector $\hat{y}$ that corresponds to the probability that $x$ belongs to class $y$, according to $f$.
  Actually, the equation \ref{1} is also the definition of the CCE with one-hot vectors as targets (which behave as indicator functions). The only difference between CCE and SSE is really just the representation of the targets, which can slightly change the implementation under the hood. Moreover, note that this is the definition of the CE for only $1$ training pair. If you have multiple pairs, you have to compute the CE for all pairs, then average these CEs (for a reference, see equation 4.108, section 4.3.4 Multiclass logistic regression of Bishop's book PRML).
  Let's have a look at a concrete example with concrete numbers. Let $C=5$, $y = 3$, $\hat{y} = [0.2, 0.2, 0.1, 0.4, 0.1]$, then the SCE is
  \begin{align}
  \text{SCE}(y, \hat{y}) 
  &= 
  - \ln (0.4) \approx 0.92,
  \end{align}
  If $\hat{y} = [0.2, 0.2, 0.2, 0.1, 0.3]$, so $\hat{y}_{y} = 0.1$, and we still have $y = 3$, then the CCE is $2.3 > 1$.
  You can execute this Python code to check yourself.
  import numpy as np
  import tensorflow as tf # Install TensorFlow 2.3!
  
  y = 1
  y_true = [3] # sparse label (integer)
  y_true2 = [0, 0, 0, 1, 0] # one-hot vector
  
  for y_y in [0.4, 0.1]:
      sce_np = -(y * np.log(y_y))
      print("SCE (NumPy) =", sce_np)
  
  y_preds = [[0.2, 0.2, 0.1, 0.4, 0.1],
             [0.2, 0.2, 0.2, 0.1, 0.3]]
  for y_pred in y_preds:
      sce_tf = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)
      cce_tf = tf.keras.losses.categorical_crossentropy(y_true2, y_pred)
  
      print("SCE (TensorFlow) =", sce_tf)
      print("CCE (TensorFlow) =", cce_tf)
  
  To answer the following question more directly.
  
  However, while training, my loss value is greater than one (almost equal to 1.2) in the first epoch, but until epoch 5 it comes near 0.8. Is it normal? If not, how can I solve this?
  
  Yes. It can happen, as explained above. (However, this does not mean that you do not have mistakes in your code.)
  
  
  ---
  
  