<p>There is at least <a href="https://arxiv.org/abs/1710.05941" rel="nofollow noreferrer">Swish</a>, which is defined as <span class="math-container">$f(x) = x \cdot \text{sigmoid}(\beta x)$</span>.</p>
<blockquote>
<p>...This suggests that Swish can be loosely viewed as a smooth function
which nonlinearly interpolates between the linear function and the
ReLU function. The degree of interpolation can be controlled by the
model if β is set as a trainable parameter.</p>
</blockquote>
<p>There is an other paper which introduces <a href="https://arxiv.org/abs/2108.12943" rel="nofollow noreferrer">Growing Cosine Unit</a>, defined as <span class="math-container">$f(x) = x \cdot \text{cos}(x)$</span>.</p>
<blockquote>
<p>The experiments (Table 1-3) show that use of GCU activation for
convolutional layers and ReLU for the dense layers provides the best
performance among all architectures considered. This is particularly
evident on the VGG-16 network trained on the Imagenette dataset, where
the GCU models outperform all ReLU architectures by 7+%. The models
with GCU in the convolutional layers also converge faster during
training as highlighted by Fig 7-8.</p>
</blockquote>
<p>In their experiments this GCU performed significantly better than Swish, or any other activation function.</p>

CC BY-SA 4.0
2022-01-24T22:59:22.250Z
Hm interesting. Maybe bad question as I don't have experience with academics: On Google scholar it only shows 2 citations of this paper. Is this inaccurate, or do responses just take some time, or is this result not as important as it seems?
Moritz Groß
CC BY-SA 4.0
2022-01-24T22:41:15.033Z
2022-01-24T22:41:15.033Z
NikoNyrh
<p>All activation functions I know of are injective, which I think makes sense.</p>
<p>But are there cases where non-injective activations can be useful?</p>

CC BY-SA 4.0
2022-01-24T19:18:04.570Z
Do you care about clarifying what you mean by "injective" and which activation functions do you have in mind (because maybe there's a misunderstanding)?
nbro
CC BY-SA 4.0
2022-01-24T20:38:19.660Z
Injective like in https://en.wikipedia.org/wiki/Injective_function. And I don't have examples in mind. Like of course |x| or sin(x) would be possible, but I'm unsure of what would be gained when using them.
Moritz Groß
<neural-networks><deep-learning><activation-functions><relu><softmax>
Are any non-injective activation functions used?
Moritz Groß
# Are any non-injective activation functions used?

(Asked by: username_0 on 2022-01-24)

All activation functions I know of are injective, which I think makes sense.

But are there cases where non-injective activations can be useful?

**Tags:** neural-networks, deep-learning, activation-functions, relu, softmax

> **Comments:**
> **username_1**: Do you care about clarifying what you mean by "injective" and which activation functions do you have in mind (because maybe there's a misunderstanding)?  
> **username_0**: Injective like in https://en.wikipedia.org/wiki/Injective_function. And I don't have examples in mind. Like of course |x| or sin(x) would be possible, but I'm unsure of what would be gained when using them.  

---

## 1 answers

### Answer by username_2 on 2022-01-24. Score: 1 | Accepted answer

There is at least [Swish](https://arxiv.org/abs/1710.05941), which is defined as $f(x) = x \cdot \text{sigmoid}(\beta x)$.

> ...This suggests that Swish can be loosely viewed as a smooth function
> which nonlinearly interpolates between the linear function and the
> ReLU function. The degree of interpolation can be controlled by the
> model if β is set as a trainable parameter.

There is an other paper which introduces [Growing Cosine Unit](https://arxiv.org/abs/2108.12943), defined as $f(x) = x \cdot \text{cos}(x)$.

> The experiments (Table 1-3) show that use of GCU activation for
> convolutional layers and ReLU for the dense layers provides the best
> performance among all architectures considered. This is particularly
> evident on the VGG-16 network trained on the Imagenette dataset, where
> the GCU models outperform all ReLU architectures by 7+%. The models
> with GCU in the convolutional layers also converge faster during
> training as highlighted by Fig 7-8.

In their experiments this GCU performed significantly better than Swish, or any other activation function.

> **Comments:**
> **username_0**: Hm interesting. Maybe bad question as I don't have experience with academics: On Google scholar it only shows 2 citations of this paper. Is this inaccurate, or do responses just take some time, or is this result not as important as it seems?  

---

