Answers.Body: <p>First of all, check out <a href="https://ai.stackexchange.com/a/27698/20538">this question</a>. Generally, you don't need to apply <code>softmax</code> and using raw logits leads to better numerical stability.</p>
  <p>The numerical issue that you are talking about is well known and dealt with the so-called <a href="https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations" rel="nofollow noreferrer"><code>logsumexp</code> trick</a>. This usually is already incorporated in standard NN libraries. For example <code>keras</code>
  <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses/CategoricalCrossentropy" rel="nofollow noreferrer">CategoricalCrossentropy loss</a> can be configured to compute it <code>from_logits</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-03T17:29:15.027Z
Answers.LastActivityDate: 2021-06-03T17:29:15.027Z
Answers.UserDisplayName: Kostya
Body: <p>I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN evaluation.</p>
  <p>Currently, I'm using a non-linear activation function for hidden units. Because of its probability interpretation, I am forced to use the softmax (or the sigmoid for 2-class case) for the output layer. However, because of the use of integers, the linear combination of the activations and weights can easily be too large, and this causes a problem to the exponential in the softmax evaluation.</p>
  <p>Any solution?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-05T12:42:16.063Z
Comments.Text: Hello. Welcome to AI SE. I've edited your post in order to improve its clarity and, in particular, to add a more descriptive title. Let me know if I changed the meaning of your post with my edit. If yes, you can edit again your post to fix this.
Comments.UserDisplayName: nbro
Tags: <neural-networks><convolutional-neural-networks><activation-functions><weights><softmax>
Title: Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?
UserDisplayName: Liuuuuk
ThreadText: # Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?
  
  (Asked by: username_0 on 2021-06-03)
  
  I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN evaluation.
  
  Currently, I'm using a non-linear activation function for hidden units. Because of its probability interpretation, I am forced to use the softmax (or the sigmoid for 2-class case) for the output layer. However, because of the use of integers, the linear combination of the activations and weights can easily be too large, and this causes a problem to the exponential in the softmax evaluation.
  
  Any solution?
  
  **Tags:** neural-networks, convolutional-neural-networks, activation-functions, weights, softmax
  > **Comments:**
  > **username_1**: Hello. Welcome to AI SE. I've edited your post in order to improve its clarity and, in particular, to add a more descriptive title. Let me know if I changed the meaning of your post with my edit. If yes, you can edit again your post to fix this.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-06-03. Score: 1
  
  First of all, check out [this question](https://ai.stackexchange.com/a/27698/20538). Generally, you don't need to apply `softmax` and using raw logits leads to better numerical stability.
  
  The numerical issue that you are talking about is well known and dealt with the so-called [`logsumexp` trick](https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations). This usually is already incorporated in standard NN libraries. For example `keras`
  [CategoricalCrossentropy loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses/CategoricalCrossentropy) can be configured to compute it `from_logits`.
  
  
  ---
  
  