Answers.Body: <p><strong>Short answer: larger gradients</strong></p>
  <p>That is not the derivative of the softmax function. <span class="math-container">$t - o$</span> is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to simplify the derivative of the softmax function. In the end, you do end up with a different gradients. It would be like if you ignored the sigmoid derivative when using MSE loss and the outputs are different. Using softmax and cross entropy loss has different uses and benefits compared to using sigmoid and MSE. It will help prevent gradient vanishing because the derivative of the sigmoid function only has a large value in a very small space of it. It is similar to using a different cross entropy loss where the combined derivative of the loss and sigmoid is <span class="math-container">$t - o$</span>.</p>
  <p>Information on derivatives of cross entropy <a href="http://neuralnetworksanddeeplearning.com/chap3.html" rel="nofollow noreferrer">with sigmoid function</a> and <a href="https://peterroelants.github.io/posts/cross-entropy-softmax/" rel="nofollow noreferrer">with softmax function</a>. I would also suggest some more research on cross entropy loss functions beyond my links.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-11T02:36:08.270Z
Answers.Text: so cross entropy makes loss convergence faster, compared to mse?
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-14T19:26:50.350Z
Answers.Text: @datdinhqouc When the last layer is the sigmoid function then cross entropy should make loss convergence faster and prevent vanishing gradients.
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T22:44:05.630Z
Answers.LastActivityDate: 2020-10-11T22:57:12.587Z
Answers.LastEditDate: 2020-10-11T22:57:12.587Z
Answers.UserDisplayName: S2673
Answers.Body: <p>If you look at the definition of the cross-entropy (e.g. <a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="nofollow noreferrer">here</a>), you will see that it is defined for probability distributions (in fact, it comes from information theory). You can also show that the maximization of the (binomial/Bernoulli) log-likelihood is equivalent to the minimization of the cross-entropy, i.e. when you minimize the cross-entropy you actually maximize the log-likelihood of the parameters given your labelled data. Hence the use of the softmax is theoretically founded.</p>
  <p>Regarding the supposed derivative of the cross-entropy loss function preceded by the softmax, even if that derivative is correct (<a href="https://www.ics.uci.edu/%7Epjsadows/notes.pdf" rel="nofollow noreferrer">I didn't think about it and I don't want to think about it now</a>), note that then <span class="math-container">$t - o$</span> is different depending on whether <span class="math-container">$o$</span> is a probability vector or an unnormalized vector (which can take arbitrarily large numbers). If <span class="math-container">$o$</span> is a probability vector and <span class="math-container">$t$</span> a one-hot encoded vector (i.e. also a probability vector), then all numbers of <span class="math-container">$t - o$</span> will be in the range <span class="math-container">$[-1, 1]$</span>. However, if <span class="math-container">$o_i$</span> can be arbitrarily large, e.g. <span class="math-container">$o_i = 10$</span>, then <span class="math-container">$t_i - o_i \in [-10, -9]$</span>. So, the propagated error would be different if <span class="math-container">$o$</span> was not a probability vector.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T09:46:36.553Z
Answers.LastActivityDate: 2020-10-11T20:46:23.627Z
Answers.LastEditDate: 2020-10-11T20:46:23.627Z
Answers.UserDisplayName: nbro
Body: <p>I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found &amp; fixed some errors, but I had a few more ideas. For one, I am using the sigmoid activation function and MSE as an error function, but the internet suggests that I should rather use softmax for the output layer, and cross entropy loss as an error function. Now I get that softmax is a nice activation function for this task, because you can treat the output as a propability vector. But, while being a nice thing to have, that's more of a convinience thing, isn't it? Easier to visualize?</p>
  <p>But when I looked at what the derivative of softmax &amp; CEL combined is (my plan was to compute that in one step and then treat the activation function of the last layer as linear, as not to apply the softmax derivative again), I found:</p>
  <p><span class="math-container">$\frac{δE}{δi}$</span> = <span class="math-container">$t$</span> − <span class="math-container">$o$</span></p>
  <p>(With <span class="math-container">$i$</span> being the input of the last layer, <span class="math-container">$t$</span> the one hot target vector and <span class="math-container">$o$</span> the prediction vector).</p>
  <p>That is the same as the MSE derivative. So what benefits does softmax + CEL actually have when propagating, if the gradients produced by them are exactly the same?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-11T09:29:00.730Z
Comments.Text: Where did you find that derivative? Also, even if that's the correct derivative (I didn't think about it and I don't want to think about it now), note that, if $o$ is the prediction vector, then $t - o$ is different if $o$ is a probability vector or an unnormalised one.
Comments.UserDisplayName: nbro
Tags: <neural-networks><gradient-descent><cross-entropy><mean-squared-error><softmax>
Title: What is the advantage of using cross entropy loss & softmax?
UserDisplayName: Ben
ThreadText: # What is the advantage of using cross entropy loss & softmax?
  
  (Asked by: username_0 on 2020-10-11)
  
  I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some errors, but I had a few more ideas. For one, I am using the sigmoid activation function and MSE as an error function, but the internet suggests that I should rather use softmax for the output layer, and cross entropy loss as an error function. Now I get that softmax is a nice activation function for this task, because you can treat the output as a propability vector. But, while being a nice thing to have, that's more of a convinience thing, isn't it? Easier to visualize?
  
  But when I looked at what the derivative of softmax & CEL combined is (my plan was to compute that in one step and then treat the activation function of the last layer as linear, as not to apply the softmax derivative again), I found:
  
  $\frac{δE}{δi}$ = $t$ − $o$
  
  (With $i$ being the input of the last layer, $t$ the one hot target vector and $o$ the prediction vector).
  
  That is the same as the MSE derivative. So what benefits does softmax + CEL actually have when propagating, if the gradients produced by them are exactly the same?
  
  **Tags:** neural-networks, gradient-descent, cross-entropy, mean-squared-error, softmax
  > **Comments:**
  > **username_1**: Where did you find that derivative? Also, even if that's the correct derivative (I didn't think about it and I don't want to think about it now), note that, if $o$ is the prediction vector, then $t - o$ is different if $o$ is a probability vector or an unnormalised one.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-10-11. Score: 1
  
  **Short answer: larger gradients**
  
  That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to simplify the derivative of the softmax function. In the end, you do end up with a different gradients. It would be like if you ignored the sigmoid derivative when using MSE loss and the outputs are different. Using softmax and cross entropy loss has different uses and benefits compared to using sigmoid and MSE. It will help prevent gradient vanishing because the derivative of the sigmoid function only has a large value in a very small space of it. It is similar to using a different cross entropy loss where the combined derivative of the loss and sigmoid is $t - o$.
  
  Information on derivatives of cross entropy [with sigmoid function](http://neuralnetworksanddeeplearning.com/chap3.html) and [with softmax function](https://peterroelants.github.io/posts/cross-entropy-softmax/). I would also suggest some more research on cross entropy loss functions beyond my links.
  
  > **Comments:**
  > **username_3**: so cross entropy makes loss convergence faster, compared to mse?  
  > **username_2**: @datdinhqouc When the last layer is the sigmoid function then cross entropy should make loss convergence faster and prevent vanishing gradients.  
  
  ---
  
  ### Answer by username_1 on 2020-10-11. Score: -1
  
  If you look at the definition of the cross-entropy (e.g. [here](https://en.wikipedia.org/wiki/Cross_entropy)), you will see that it is defined for probability distributions (in fact, it comes from information theory). You can also show that the maximization of the (binomial/Bernoulli) log-likelihood is equivalent to the minimization of the cross-entropy, i.e. when you minimize the cross-entropy you actually maximize the log-likelihood of the parameters given your labelled data. Hence the use of the softmax is theoretically founded.
  
  Regarding the supposed derivative of the cross-entropy loss function preceded by the softmax, even if that derivative is correct ([I didn't think about it and I don't want to think about it now](https://www.ics.uci.edu/%7Epjsadows/notes.pdf)), note that then $t - o$ is different depending on whether $o$ is a probability vector or an unnormalized vector (which can take arbitrarily large numbers). If $o$ is a probability vector and $t$ a one-hot encoded vector (i.e. also a probability vector), then all numbers of $t - o$ will be in the range $[-1, 1]$. However, if $o\_i$ can be arbitrarily large, e.g. $o\_i = 10$, then $t\_i - o\_i \in [-10, -9]$. So, the propagated error would be different if $o$ was not a probability vector.
  
  
  ---
  
  