Answers.Body: <p>Can't you just divide the numbers by their sum? If your numbers could be negative, you can clamp them to a probability of zero by passing them through a RELU activation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-16T11:28:00.683Z
Answers.Text: I am asking for reputed ones in application.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-17T14:41:28.420Z
Answers.Text: I don't think there is any other function which preserves the ratios of numbers `[0, a, b]` and scales them to sum to 1. You explicitly ask if there is such function, and I answered that yes there is and provided an example.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-16T09:42:18.593Z
Answers.LastActivityDate: 2022-02-16T09:42:18.593Z
Answers.UserDisplayName: NikoNyrh
Body: <p>Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are intended for classification.</p>
  <p>But, the <code>softmax()</code> does not satisfy the property of scale invariance i.e., the ratio of inputs and the ratio of outputs does not remain the same.</p>
  <p>For example if we give the input <code>[1.4285, 0.3815]</code> to <code>softmax</code>, the function will give <code>[0.7402, 0.2598]</code> as output.We can calculate ratios <code>1.4285: 0.3815</code> and <code>0.7402: 0.2598</code> and find that they are not the same.</p>
  <p>Are there any scale-invariant versions of softmax?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-16T07:07:13.053Z
Comments.Text: Could you explian why you need a scale-invariant function? Unmodified logits in a neural network can be, and often are negative. They cannot be simply scaled into a discrete probability distribution. However, if you are willing to accept other non-linearities to adjust this, then something may be possible. But it not clear whether what has driven your question, thus whether such a thing would be a good idea.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-16T23:55:35.283Z
Comments.Text: @NeilSlater The text book I am reading mentioned about this property explicitly. So, I think there might be some significance and hence asked it.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-17T07:07:43.097Z
Comments.Text: Can you give a reference to where it is mentiioned and in which content? As far as I know there is no function that could turn arbitrary logits into a discrete probability distribution without impacting their ratios. Non-linearity is required to deal with negative inputs.
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><activation-functions><probability-distribution><softmax>
Title: Are there any scale invariant activation functions that outputs probability distribution?
UserDisplayName: hanugm
ThreadText: # Are there any scale invariant activation functions that outputs probability distribution?
  
  (Asked by: username_0 on 2022-02-16)
  
  Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are intended for classification.
  
  But, the `softmax()` does not satisfy the property of scale invariance i.e., the ratio of inputs and the ratio of outputs does not remain the same.
  
  For example if we give the input `[1.4285, 0.3815]` to `softmax`, the function will give `[0.7402, 0.2598]` as output.We can calculate ratios `1.4285: 0.3815` and `0.7402: 0.2598` and find that they are not the same.
  
  Are there any scale-invariant versions of softmax?
  
  **Tags:** neural-networks, activation-functions, probability-distribution, softmax
  
  > **Comments:**
  > **username_1**: Could you explian why you need a scale-invariant function? Unmodified logits in a neural network can be, and often are negative. They cannot be simply scaled into a discrete probability distribution. However, if you are willing to accept other non-linearities to adjust this, then something may be possible. But it not clear whether what has driven your question, thus whether such a thing would be a good idea.  
  > **username_0**: @username_1 The text book I am reading mentioned about this property explicitly. So, I think there might be some significance and hence asked it.  
  > **username_1**: Can you give a reference to where it is mentiioned and in which content? As far as I know there is no function that could turn arbitrary logits into a discrete probability distribution without impacting their ratios. Non-linearity is required to deal with negative inputs.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-16. Score: 1
  
  Can't you just divide the numbers by their sum? If your numbers could be negative, you can clamp them to a probability of zero by passing them through a RELU activation.
  
  > **Comments:**
  > **username_0**: I am asking for reputed ones in application.  
  > **username_2**: I don't think there is any other function which preserves the ratios of numbers `[0, a, b]` and scales them to sum to 1. You explicitly ask if there is such function, and I answered that yes there is and provided an example.  
  
  ---
  
  