Answers.Body: <p>There is at least <a href="https://arxiv.org/abs/1710.05941" rel="nofollow noreferrer">Swish</a>, which is defined as <span class="math-container">$f(x) = x \cdot \text{sigmoid}(\beta x)$</span>.</p>
  <blockquote>
  <p>...This suggests that Swish can be loosely viewed as a smooth function
  which nonlinearly interpolates between the linear function and the
  ReLU function. The degree of interpolation can be controlled by the
  model if β is set as a trainable parameter.</p>
  </blockquote>
  <p>There is an other paper which introduces <a href="https://arxiv.org/abs/2108.12943" rel="nofollow noreferrer">Growing Cosine Unit</a>, defined as <span class="math-container">$f(x) = x \cdot \text{cos}(x)$</span>.</p>
  <blockquote>
  <p>The experiments (Table 1-3) show that use of GCU activation for
  convolutional layers and ReLU for the dense layers provides the best
  performance among all architectures considered. This is particularly
  evident on the VGG-16 network trained on the Imagenette dataset, where
  the GCU models outperform all ReLU architectures by 7+%. The models
  with GCU in the convolutional layers also converge faster during
  training as highlighted by Fig 7-8.</p>
  </blockquote>
  <p>In their experiments this GCU performed significantly better than Swish, or any other activation function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T22:59:22.250Z
Answers.Text: Hm interesting. Maybe bad question as I don't have experience with academics: On Google scholar it only shows 2 citations of this paper. Is this inaccurate, or do responses just take some time, or is this result not as important as it seems?
Answers.UserDisplayName: Moritz Groß
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T22:41:15.033Z
Answers.LastActivityDate: 2022-01-24T22:41:15.033Z
Answers.UserDisplayName: NikoNyrh
Body: <p>All activation functions I know of are injective, which I think makes sense.</p>
  <p>But are there cases where non-injective activations can be useful?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T19:18:04.570Z
Comments.Text: Do you care about clarifying what you mean by "injective" and which activation functions do you have in mind (because maybe there's a misunderstanding)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T20:38:19.660Z
Comments.Text: Injective like in https://en.wikipedia.org/wiki/Injective_function. And I don't have examples in mind. Like of course |x| or sin(x) would be possible, but I'm unsure of what would be gained when using them.
Comments.UserDisplayName: Moritz Groß
Tags: <neural-networks><deep-learning><activation-functions><relu><softmax>
Title: Are any non-injective activation functions used?
UserDisplayName: Moritz Groß
ThreadText: # Are any non-injective activation functions used?
  
  (Asked by: username_0 on 2022-01-24)
  
  All activation functions I know of are injective, which I think makes sense.
  
  But are there cases where non-injective activations can be useful?
  
  **Tags:** neural-networks, deep-learning, activation-functions, relu, softmax
  
  > **Comments:**
  > **username_1**: Do you care about clarifying what you mean by "injective" and which activation functions do you have in mind (because maybe there's a misunderstanding)?  
  > **username_0**: Injective like in https://en.wikipedia.org/wiki/Injective_function. And I don't have examples in mind. Like of course |x| or sin(x) would be possible, but I'm unsure of what would be gained when using them.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-24. Score: 1 | Accepted answer
  
  There is at least [Swish](https://arxiv.org/abs/1710.05941), which is defined as $f(x) = x \cdot \text{sigmoid}(\beta x)$.
  
  > ...This suggests that Swish can be loosely viewed as a smooth function
  > which nonlinearly interpolates between the linear function and the
  > ReLU function. The degree of interpolation can be controlled by the
  > model if β is set as a trainable parameter.
  
  There is an other paper which introduces [Growing Cosine Unit](https://arxiv.org/abs/2108.12943), defined as $f(x) = x \cdot \text{cos}(x)$.
  
  > The experiments (Table 1-3) show that use of GCU activation for
  > convolutional layers and ReLU for the dense layers provides the best
  > performance among all architectures considered. This is particularly
  > evident on the VGG-16 network trained on the Imagenette dataset, where
  > the GCU models outperform all ReLU architectures by 7+%. The models
  > with GCU in the convolutional layers also converge faster during
  > training as highlighted by Fig 7-8.
  
  In their experiments this GCU performed significantly better than Swish, or any other activation function.
  
  > **Comments:**
  > **username_0**: Hm interesting. Maybe bad question as I don't have experience with academics: On Google scholar it only shows 2 citations of this paper. Is this inaccurate, or do responses just take some time, or is this result not as important as it seems?  
  
  ---
  
  