Answers.Body: <p><em>Loss reduction means model improvement,</em> it does not in the wrong setup, wher random choise produces least loss. So it is some critical setup error. What classes do you have?
  I got also thet recently experimenting with an encoder with too narrow coding layer - it just EQUILIZES the output with average values cause this state has minimum loss.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-16T23:31:08.330Z
Answers.LastActivityDate: 2019-06-16T23:31:08.330Z
Answers.UserDisplayName: user8426627
Answers.Body: <p>It's important to remember what exactly the loss is measuring, and have some typical values in mind. </p>
  
  <p>The cross-entropy loss is <span class="math-container">$-\mathbb{E}_{x,y\sim p}\left[\log q(y|x)\right]$</span>, where <span class="math-container">$p$</span> is the data distribution and <span class="math-container">$q$</span> is the model distribution. 
  A couple of points about the loss:</p>
  
  <ul>
  <li>It's nonnegative, in the range <span class="math-container">$[0, \infty)$</span>.</li>
  <li>Predicting randomly (assuming balanced classes) gives loss equal to <span class="math-container">$\log k$</span>.</li>
  </ul>
  
  <p>In your case with four classes, the loss for a random classifier is <span class="math-container">$\log 4 \approx 1.39$</span>. So the story for what happened with your model is probably that initially (due to initialization, etc) it predicted high but wrong confidence, such as giving 99% probabilities to certain classes. This gives very high loss, but then after training for a while it reduces its loss to just under the random loss by predicting 25% on all examples. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-13T19:20:57.417Z
Answers.Text: In your definition of cross-entropy, the CE will always be negative, given that log of a number between 0 and 1 (excluded) is always negative, but, in practice, the implementation of cross-entropy may use the negative of your formulation, so that you can minimize it (i.e. you want to conceptually minimize a "loss") rather than maximizing (i.e. the CE is defined in terms of the likelihood, so you want to maximise the likelihood).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-13T19:26:13.060Z
Answers.Text: Yep that's a good point!
Answers.UserDisplayName: Chris Cundy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-13T16:59:49.730Z
Answers.LastActivityDate: 2020-04-13T19:27:24.330Z
Answers.LastEditDate: 2020-04-13T19:27:24.330Z
Answers.UserDisplayName: Chris Cundy
Body: <h2>Problem Statement</h2>
  
  <p>I've built a classifier to classify a dataset consisting of n samples and four classes of data. To this end, I've used pre-trained VGG-19, pre-trained Alexnet and even LeNet (with cross-entropy loss). However, I just changed the softmax layer's architecture and placed just four neurons for that (because my dataset includes just four classes). Since the dataset classes have a striking resemblance to each other, this classifier was unable to classify them and I was forced to use other methods.
  During the training section, after some epochs, loss decreased from approximately 7 to approximately 1.2, but there were no changes in accuracy and it was frozen on 25% (random precision). In the best epochs, the accuracy just reached near 27% but it was completely unstable.</p>
  
  <h2>Question</h2>
  
  <p>How is it justifiable? If loss reduction means model improvement, why doesn't accuracy increase? How is it possible to the loss decreases near 6 points (approximately from 7 to 1) but nothing happens to accuracy at all?</p>
  
Tags: <deep-learning><classification><training><objective-functions>
Title: If loss reduction means model improvement, why doesn't accuracy increase?
UserDisplayName: Arashsyh
ThreadText: # If loss reduction means model improvement, why doesn't accuracy increase?
  
  (Asked by: username_0 on 2019-06-16)
  
  ## Problem Statement
  
  I've built a classifier to classify a dataset consisting of n samples and four classes of data. To this end, I've used pre-trained VGG-19, pre-trained Alexnet and even LeNet (with cross-entropy loss). However, I just changed the softmax layer's architecture and placed just four neurons for that (because my dataset includes just four classes). Since the dataset classes have a striking resemblance to each other, this classifier was unable to classify them and I was forced to use other methods.
  During the training section, after some epochs, loss decreased from approximately 7 to approximately 1.2, but there were no changes in accuracy and it was frozen on 25% (random precision). In the best epochs, the accuracy just reached near 27% but it was completely unstable.
  
  ## Question
  
  How is it justifiable? If loss reduction means model improvement, why doesn't accuracy increase? How is it possible to the loss decreases near 6 points (approximately from 7 to 1) but nothing happens to accuracy at all?
  
  **Tags:** deep-learning, classification, training, objective-functions
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-06-16. Score: 1
  
  *Loss reduction means model improvement,* it does not in the wrong setup, wher random choise produces least loss. So it is some critical setup error. What classes do you have?
  I got also thet recently experimenting with an encoder with too narrow coding layer - it just EQUILIZES the output with average values cause this state has minimum loss.
  
  
  ---
  
  ### Answer by username_2 on 2020-04-13. Score: 1
  
  It's important to remember what exactly the loss is measuring, and have some typical values in mind.
  
  The cross-entropy loss is $-\mathbb{E}\_{x,y\sim p}\left[\log q(y|x)\right]$, where $p$ is the data distribution and $q$ is the model distribution.
  A couple of points about the loss:
  
  * It's nonnegative, in the range $[0, \infty)$.
  * Predicting randomly (assuming balanced classes) gives loss equal to $\log k$.
  
  In your case with four classes, the loss for a random classifier is $\log 4 \approx 1.39$. So the story for what happened with your model is probably that initially (due to initialization, etc) it predicted high but wrong confidence, such as giving 99% probabilities to certain classes. This gives very high loss, but then after training for a while it reduces its loss to just under the random loss by predicting 25% on all examples.
  
  > **Comments:**
  > **username_3**: In your definition of cross-entropy, the CE will always be negative, given that log of a number between 0 and 1 (excluded) is always negative, but, in practice, the implementation of cross-entropy may use the negative of your formulation, so that you can minimize it (i.e. you want to conceptually minimize a "loss") rather than maximizing (i.e. the CE is defined in terms of the likelihood, so you want to maximise the likelihood).  
  > **username_2**: Yep that's a good point!  
  
  ---
  
  