Answers.Body: <p>I see no problem in regarding the softmax as a particular activation function which takes a vector input and produces a vector output. In fact, the sigmoid function can be viewed as a two-dimensional softmax in which one of the two inputs is hardwired to zero while the corresponding output is neglected.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-22T11:11:39.643Z
Answers.Text: Thanks for your answer. Indeed, there is no problem with regarding it as such. The question was mostly aimed at clarification.
Answers.UserDisplayName: TimvanSch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-13T10:46:22.313Z
Answers.LastActivityDate: 2019-02-13T14:26:05.213Z
Answers.LastEditDate: 2019-02-13T14:26:05.213Z
Answers.UserDisplayName: ssegvic
Body: <p>An activation function is a function from <span class="math-container">$R \rightarrow R$</span>. It takes as input the inner products of weights and activations in the previous layer. It outputs the activation. </p>
  
  <p>A softmax however, is a function that takes input from <span class="math-container">$R^p$</span>, where <span class="math-container">$p$</span> is the number of possible outcomes that need to be classified. Therefore, strictly speaking, it cannot be an activation function.</p>
  
  <p>Yet everywhere on the net it says the softmax is an activation function. Am I wrong or are they?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-13T11:56:47.080Z
Comments.Text: An activation function isn't necessarily defined as $\mathbb{R} \rightarrow \mathbb{R}$.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-13T14:27:26.350Z
Comments.Text: I would be interested to know where you are taking your definition of an activation function from in your first paragraph? I am not sure if it has a formal definition, and even if it did, this would maybe become an issue of mis-naming or mis-classifying something, but the engineering of it - what it is and does - would remain exactly the same.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-22T11:09:11.360Z
Comments.Text: Coming to think of it, I did not see a formal definition. Instead, I saw pictures such as those [link](https://stackoverflow.com/questions/38850538/single-neuron-in-neural-network-using-c), where it seems that they are functions that take from & $\mathbb{R}$.
Comments.UserDisplayName: TimvanSch
Tags: <neural-networks><artificial-neuron><activation-functions>
Title: Is it a great misconception that the softmax is an activation function?
UserDisplayName: TimvanSch
ThreadText: # Is it a great misconception that the softmax is an activation function?
  
  (Asked by: username_0 on 2019-02-13)
  
  An activation function is a function from $R \rightarrow R$. It takes as input the inner products of weights and activations in the previous layer. It outputs the activation.
  
  A softmax however, is a function that takes input from $R^p$, where $p$ is the number of possible outcomes that need to be classified. Therefore, strictly speaking, it cannot be an activation function.
  
  Yet everywhere on the net it says the softmax is an activation function. Am I wrong or are they?
  
  **Tags:** neural-networks, artificial-neuron, activation-functions
  
  > **Comments:**
  > **username_1**: An activation function isn't necessarily defined as $\mathbb{R} \rightarrow \mathbb{R}$.  
  > **username_2**: I would be interested to know where you are taking your definition of an activation function from in your first paragraph? I am not sure if it has a formal definition, and even if it did, this would maybe become an issue of mis-naming or mis-classifying something, but the engineering of it - what it is and does - would remain exactly the same.  
  > **username_0**: Coming to think of it, I did not see a formal definition. Instead, I saw pictures such as those [link](https://stackoverflow.com/questions/38850538/single-neuron-in-neural-network-using-c), where it seems that they are functions that take from & $\mathbb{R}$.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-02-13. Score: 1
  
  I see no problem in regarding the softmax as a particular activation function which takes a vector input and produces a vector output. In fact, the sigmoid function can be viewed as a two-dimensional softmax in which one of the two inputs is hardwired to zero while the corresponding output is neglected.
  
  > **Comments:**
  > **username_0**: Thanks for your answer. Indeed, there is no problem with regarding it as such. The question was mostly aimed at clarification.  
  
  ---
  
  