Answers.Body: <p>So class 0 has loss <span class="math-container">$− 0 \log \hat y=0$</span>
  and class 9 has loss <span class="math-container">$−9 \log \hat y$</span>
  ? This is clearly not correct.</p>
  <p>The correct implementation takes as input a vector <span class="math-container">$\hat y$</span>  with 10 elements and you want to select the <span class="math-container">$k$</span>th element from it, where <span class="math-container">$k$</span> is the correct class: <span class="math-container">$-\log \hat {y}_k$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-16T12:34:50.913Z
Answers.LastActivityDate: 2024-05-16T12:34:50.913Z
Answers.UserDisplayName: Sycorax
Body: <p>I've been attempting to mess around with Sparse Categorical Cross Entropy Loss for the MNIST dataset. I can't seem to figure out what might be wrong with my implementation, the loss seems to erroneously shoot up.</p>
  <p>I figure this to be the correct formula?</p>
  <p><span class="math-container">$CE_{sparse} = - \frac{1}{m} \sum y_{integerencoded} * \log{\hat{Y}}$</span> , where <span class="math-container">$m$</span> is the number of samples</p>
  <p>The model was working normally with Categorical Cross Entropy (with one_hot_encodings), but when I introduce integer encodings instead, things seem to blow up.</p>
  <p>I'm curious to know, what might be wrong with my implementation?</p>
  <pre><code># Y IS THE ORIGINAL MNIST LABELS, A2 IS THE RAW SOFTMAX OUTPUT.
  
  def sparse_cc_loss(y, a2):
      eps = 1e-10
      l = - np.sum(y * np.log(a2 + eps)) / 60000
      return l
  </code></pre>
  <p>If you might need more reference, check out the github repo <a href="https://github.com/vxnuaj/LABS/blob/main/DEEPLEARNING/NeuralNetworks/LossFuncs/sparse.py" rel="nofollow noreferrer">here</a></p>
  <p>Not sure if it's the calculation of <span class="math-container">$\frac{∂CE_{sparse}}{∂z_2}$</span> which might be playing a factor.</p>
  <p>I did so as:</p>
  <p><code>dz2 = a2 - y</code>, where <code>y</code> are the MNIST labels (0-9), and <code>a2</code>is the raw softmax output.</p>
  
Tags: <neural-networks><objective-functions>
Title: Sparse Cross Entropy
UserDisplayName: vxnuaj
ThreadText: # Sparse Cross Entropy
  
  (Asked by: username_0 on 2024-05-16)
  
  I've been attempting to mess around with Sparse Categorical Cross Entropy Loss for the MNIST dataset. I can't seem to figure out what might be wrong with my implementation, the loss seems to erroneously shoot up.
  I figure this to be the correct formula?
  $CE_{sparse} = - \frac{1}{m} \sum y_{integerencoded} * \log{\hat{Y}}$ , where $m$ is the number of samples
  The model was working normally with Categorical Cross Entropy (with one_hot_encodings), but when I introduce integer encodings instead, things seem to blow up.
  I'm curious to know, what might be wrong with my implementation?
  # Y IS THE ORIGINAL MNIST LABELS, A2 IS THE RAW SOFTMAX OUTPUT.
  
  def sparse_cc_loss(y, a2):
      eps = 1e-10
      l = - np.sum(y * np.log(a2 + eps)) / 60000
      return l
  
  If you might need more reference, check out the github repo here
  Not sure if it's the calculation of $\frac{∂CE_{sparse}}{∂z_2}$ which might be playing a factor.
  I did so as:
  dz2 = a2 - y, where y are the MNIST labels (0-9), and a2is the raw softmax output.
  
  **Tags:** neural-networks, objective-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-16. Score: 2
  
  So class 0 has loss $− 0 \log \hat y=0$
  and class 9 has loss $−9 \log \hat y$
  ? This is clearly not correct.
  
  The correct implementation takes as input a vector $\hat y$ with 10 elements and you want to select the $k$th element from it, where $k$ is the correct class: $-\log \hat {y}\_k$
  
  
  ---
  
  