Answers.Body: <p><strong>Short answer</strong>: Generally, you don't need to do <code>softmax</code> if you don't need probabilities. And using raw logits leads to more numerically stable code.</p>
  <p><strong>Long answer</strong>: First of all, the inputs of the <code>softmax</code> layer are called <a href="https://en.wikipedia.org/wiki/Multinomial_logistic_regression" rel="nofollow noreferrer">logits</a>.</p>
  <p><em>During evaluation</em>, if you are only interested in the highest-probability class, then you can do <code>argmax(vec)</code> on the logits. If you want probability distribution over classes, then you'll need to exponentiate and normalize to 1 - that's what <code>softmax</code> does.</p>
  <p><em>During training</em>, you'd need to have a loss function to optimize. Your training data contains true classes, so you have your target probability distribution <span class="math-container">$p_i$</span>, which is 1 at your true class and 0 at all other classes. You train the network to produce a probability distribution <span class="math-container">$q_i$</span> as an output. It should be as close to the target distribution <span class="math-container">$p_i$</span> as possible. The &quot;distance&quot; measure between two probability distribution is called <a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="nofollow noreferrer">cross-entropy</a>:</p>
  <p><span class="math-container">$$ H = - \sum p_i \log q_i $$</span>
  As you can see, you only need logs of the output probabilities - so the logits will suffice to compute the loss. For example, the <code>keras</code> standard <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses/CategoricalCrossentropy" rel="nofollow noreferrer"><code>CategoricalCrossentropy</code> loss</a> can be configured to compute it <code>from_logits</code> and it mentions that:</p>
  <blockquote>
  <p>Using from_logits=True is more numerically stable.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-07T19:09:55.033Z
Answers.Text: Note that the cross-entropy is really defined for 2 **probability distributions**, and I would say that's the reason why we use the softmax. Note that, if the logits are zero or negative, then the cross-entropy is not defined because of the logarithm. I think that TensorFlow accepts logits because it probably performs a (stable) conversion of the logits to a probability distribution under the hood, but I have not looked at the source code. So, it's a matter of diving into the TF implementation of the CE loss.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-07T19:21:24.323Z
Answers.Text: If you directly use the logits as "the logarithms of the probability", then what you say should make sense. However, why should you interpret the output of the neural network as the logarithm of the probability of belonging to a class? Still, it would be nice to show what TF exactly does when you set `from_logits=True`.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-07T19:28:21.373Z
Answers.Text: @nbro I kinda started explaining why use of logits is more numerically stable, but it turned out to be too mush of a tangent. So that deserves a separate question and answer in my opinion.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-07T17:11:32.823Z
Answers.LastActivityDate: 2021-05-07T17:22:03.900Z
Answers.LastEditDate: 2021-05-07T17:22:03.900Z
Answers.UserDisplayName: Kostya
Body: <p>Why do we use the softmax activation function on the last layer?</p>
  <p>Suppose <span class="math-container">$i$</span> is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take <span class="math-container">$i$</span>th value, it would be the highest value because <span class="math-container">$e$</span> is an increasing function, so that's why I am asking this question. Taking <code>argmax(vec)</code> and <code>argmax(softmax(vec))</code> would give us the same value.</p>
  
Tags: <neural-networks><activation-functions><softmax><multiclass-classification>
Title: Why do we use the softmax instead of no activation function?
UserDisplayName: dato nefaridze
ThreadText: # Why do we use the softmax instead of no activation function?
  
  (Asked by: username_0 on 2021-05-07)
  
  Why do we use the softmax activation function on the last layer?
  
  Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th value, it would be the highest value because $e$ is an increasing function, so that's why I am asking this question. Taking `argmax(vec)` and `argmax(softmax(vec))` would give us the same value.
  
  **Tags:** neural-networks, activation-functions, softmax, multiclass-classification
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-07. Score: 4 | Accepted answer
  
  **Short answer**: Generally, you don't need to do `softmax` if you don't need probabilities. And using raw logits leads to more numerically stable code.
  
  **Long answer**: First of all, the inputs of the `softmax` layer are called [logits](https://en.wikipedia.org/wiki/Multinomial_logistic_regression).
  
  *During evaluation*, if you are only interested in the highest-probability class, then you can do `argmax(vec)` on the logits. If you want probability distribution over classes, then you'll need to exponentiate and normalize to 1 - that's what `softmax` does.
  
  *During training*, you'd need to have a loss function to optimize. Your training data contains true classes, so you have your target probability distribution $p\_i$, which is 1 at your true class and 0 at all other classes. You train the network to produce a probability distribution $q\_i$ as an output. It should be as close to the target distribution $p\_i$ as possible. The "distance" measure between two probability distribution is called [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy):
  
  $$ H = - \sum p\_i \log q\_i $$
  As you can see, you only need logs of the output probabilities - so the logits will suffice to compute the loss. For example, the `keras` standard [`CategoricalCrossentropy` loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses/CategoricalCrossentropy) can be configured to compute it `from_logits` and it mentions that:
  
  > Using from\_logits=True is more numerically stable.
  
  > **Comments:**
  > **username_2**: Note that the cross-entropy is really defined for 2 **probability distributions**, and I would say that's the reason why we use the softmax. Note that, if the logits are zero or negative, then the cross-entropy is not defined because of the logarithm. I think that TensorFlow accepts logits because it probably performs a (stable) conversion of the logits to a probability distribution under the hood, but I have not looked at the source code. So, it's a matter of diving into the TF implementation of the CE loss.  
  > **username_2**: If you directly use the logits as "the logarithms of the probability", then what you say should make sense. However, why should you interpret the output of the neural network as the logarithm of the probability of belonging to a class? Still, it would be nice to show what TF exactly does when you set `from_logits=True`.  
  > **username_1**: @nbro I kinda started explaining why use of logits is more numerically stable, but it turned out to be too mush of a tangent. So that deserves a separate question and answer in my opinion.  
  
  ---
  
  