Answers.Body: <p>It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone <code>crossentropy</code> module the over all loss will be unstable (~1/x so it will diverge for small input values) whereas if you use a <code>softmax + crossentropy</code> module all-in-one, then it becomes numerically stable (y-p)</p>
  
  <p>Slides from DeepMind's Simon Osindero lecture at UCL in 2016:</p>
  
  <p><a href="https://i.sstatic.net/OyGix.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OyGix.jpg" alt="CE"></a></p>
  
  <p><a href="https://i.sstatic.net/uGw1c.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uGw1c.jpg" alt="CE+Softmax"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T00:16:50.360Z
Answers.Text: You say "if you have a standalone crossentropy module the over all loss will be unstable (~1/x)", but you don't really explain WHY. Maybe you can explain why using softmax is unstable.That's the actual question. Also, what does the CE have to do with the softmax?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T01:50:16.253Z
Answers.Text: The slides actually do explain *why*. If the question were about mathematical gradient derivations I'd agree with your comment, it's not. I'll add a few words though
Answers.UserDisplayName: ted
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-11T15:18:30.293Z
Answers.Text: There is a typo in the gradient on the second slide. Check here: https://xeonqq.github.io/machine%20learning/softmax/
  
  The $i$-th element in the gradient should be $\frac{e^{x_i}}{\sum_m e^{x_m}} - p_i$.
Answers.UserDisplayName: Carlos H. Mendoza-Cardenas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-13T14:23:48.843Z
Answers.LastActivityDate: 2020-04-14T01:51:20.540Z
Answers.LastEditDate: 2020-04-14T01:51:20.540Z
Answers.UserDisplayName: ted
Answers.Body: <p>This is also a question I stumble upon, thanks for the explaination from ted, it is very helpfull, I will try to elaborate a little bit. Let's still use DeepMind's Simon Osindero's <a href="https://docs.google.com/presentation/d/e/2PACX-1vQwrivdqqBR8teLQ7prKtiDyMLSqgGBzTxfQ6BKXPVvpFpLRUQOmqTm57LEMIy3IIK14RTLcBcT-PCO/pub?start=false&amp;loop=false&amp;delayms=60000&amp;slide=id.g1a727d4a2c_0_814" rel="nofollow noreferrer">slide</a>:
  <a href="https://i.sstatic.net/lV7Ty.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lV7Ty.jpg" alt="" /></a>
  The grey block on the left we are looking at is only a cross entropy operation, the input <span class="math-container">$x$</span> (a vector) could be the softmax output from previous layer (not the input for the neutral network), and <span class="math-container">$y$</span> (a scalar) is the cross entropy result of <span class="math-container">$x$</span>. To propagate the gradient back, we need to calculate the gradient of <span class="math-container">$dy/dx_i$</span>, which is <span class="math-container">$-p_i/x_i$</span> for each element in <span class="math-container">$x$</span>.
  As we know the softmax function scale the logits into the range [0,1], so if in one training step, the neutral network becomes super confident and predict one of the probabilties <span class="math-container">$x_i$</span> to be 0 then we have a numerical problem in calculting <span class="math-container">$dy/dx_i$</span>.</p>
  <p>While in the other case where we take the logits and calculate the softmax and crossentropy at one shot (XentLogits function), we don't have this problem. Because the derivative of XentLogits is <span class="math-container">$dy/dx_i = y - p_i$</span>, a more elaborated derivation can be found <a href="http://machinelearningmechanic.com/deep_learning/2019/09/04/cross-entropy-loss-derivative.html" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T11:56:48.277Z
Answers.LastActivityDate: 2020-12-17T12:09:45.357Z
Answers.LastEditDate: 2020-12-17T12:09:45.357Z
Answers.UserDisplayName: xeonqq
Body: <p>The <a href="https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/beginner.ipynb#scrollTo=he5u_okAYS4a&amp;line=1&amp;uniqifier=1" rel="noreferrer">beginner colab example</a> for tensorflow states:</p>
  
  <blockquote>
    <p>Note: It is possible to bake this <code>tf.nn.softmax</code> in as the activation function for the last layer of the network. While this can make the model output more directly interpretable, this approach is discouraged as it's impossible to
    provide an exact and numerically stable loss calculation for all models when using a softmax output. </p>
  </blockquote>
  
  <p>My question is, then, why? What do they mean by <em>impossible to provide an exact and numerically stable loss calculation</em>?</p>
  
Tags: <tensorflow><objective-functions><softmax>
Title: Why does TensorFlow docs discourage using softmax as activation for the last layer?
UserDisplayName: galah92
ThreadText: # Why does TensorFlow docs discourage using softmax as activation for the last layer?
  
  (Asked by: username_0 on 2020-04-13)
  
  The [beginner colab example](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/beginner.ipynb#scrollTo=he5u_okAYS4a&line=1&uniqifier=1) for tensorflow states:
  
  > Note: It is possible to bake this `tf.nn.softmax` in as the activation function for the last layer of the network. While this can make the model output more directly interpretable, this approach is discouraged as it's impossible to
  > provide an exact and numerically stable loss calculation for all models when using a softmax output.
  
  My question is, then, why? What do they mean by *impossible to provide an exact and numerically stable loss calculation*?
  
  **Tags:** tensorflow, objective-functions, softmax
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-04-13. Score: 2 | Accepted answer
  
  It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone `crossentropy` module the over all loss will be unstable (~1/x so it will diverge for small input values) whereas if you use a `softmax + crossentropy` module all-in-one, then it becomes numerically stable (y-p)
  
  Slides from DeepMind's Simon Osindero lecture at UCL in 2016:
  
  [![CE](https://i.sstatic.net/OyGix.jpg)](https://i.sstatic.net/OyGix.jpg)
  
  [![CE+Softmax](https://i.sstatic.net/uGw1c.jpg)](https://i.sstatic.net/uGw1c.jpg)
  
  > **Comments:**
  > **username_2**: You say "if you have a standalone crossentropy module the over all loss will be unstable (~1/x)", but you don't really explain WHY. Maybe you can explain why using softmax is unstable.That's the actual question. Also, what does the CE have to do with the softmax?  
  > **username_1**: The slides actually do explain *why*. If the question were about mathematical gradient derivations I'd agree with your comment, it's not. I'll add a few words though  
  > **username_3**: There is a typo in the gradient on the second slide. Check here: https://username_4.github.io/machine%20learning/softmax/  The $i$-th element in the gradient should be $\frac{e^{x_i}}{\sum_m e^{x_m}} - p_i$.  
  
  ---
  
  ### Answer by username_4 on 2020-12-17. Score: 2
  
  This is also a question I stumble upon, thanks for the explaination from ted, it is very helpfull, I will try to elaborate a little bit. Let's still use DeepMind's Simon Osindero's [slide](https://docs.google.com/presentation/d/e/2PACX-1vQwrivdqqBR8teLQ7prKtiDyMLSqgGBzTxfQ6BKXPVvpFpLRUQOmqTm57LEMIy3IIK14RTLcBcT-PCO/pub?start=false&loop=false&delayms=60000&slide=id.g1a727d4a2c_0_814):
  [![](https://i.sstatic.net/lV7Ty.jpg)](https://i.sstatic.net/lV7Ty.jpg)
  The grey block on the left we are looking at is only a cross entropy operation, the input $x$ (a vector) could be the softmax output from previous layer (not the input for the neutral network), and $y$ (a scalar) is the cross entropy result of $x$. To propagate the gradient back, we need to calculate the gradient of $dy/dx\_i$, which is $-p\_i/x\_i$ for each element in $x$.
  As we know the softmax function scale the logits into the range [0,1], so if in one training step, the neutral network becomes super confident and predict one of the probabilties $x\_i$ to be 0 then we have a numerical problem in calculting $dy/dx\_i$.
  
  While in the other case where we take the logits and calculate the softmax and crossentropy at one shot (XentLogits function), we don't have this problem. Because the derivative of XentLogits is $dy/dx\_i = y - p\_i$, a more elaborated derivation can be found [here](http://machinelearningmechanic.com/deep_learning/2019/09/04/cross-entropy-loss-derivative.html).
  
  
  ---
  
  