<p>By definition from wikipedia:</p>
<blockquote>
<p>Let <span class="math-container">$\varphi$</span> be a nonconstant, bounded, and continuous function.</p>
</blockquote>
<p>This is also being proven for ReLU and other functions that violets some of those properties... however, if you take <code>1 - sigmoid(x)</code> as your activation, all the properties are satisfied, and the derivative is strictly negative</p>
<p>This can be more &quot;popularly&quot; be seen also in the max-out activation function</p>

CC BY-SA 4.0
2022-08-17T00:59:57.307Z
2022-08-17T00:59:57.307Z
Alberto
<p>All the activation functions I see have positive derivatives.
Will negative ReLU work as well as its positive counterpart or will it lead to instability?</p>

<neural-networks><backpropagation><activation-functions>
Why is the derivative of activation function all positive?
user33869
# Why is the derivative of activation function all positive?

(Asked by: username_0 on 2022-08-12)

All the activation functions I see have positive derivatives.
Will negative ReLU work as well as its positive counterpart or will it lead to instability?

**Tags:** neural-networks, backpropagation, activation-functions


---

## 1 answers

### Answer by username_1 on 2022-08-17. Score: 1

By definition from wikipedia:

> Let $\varphi$ be a nonconstant, bounded, and continuous function.

This is also being proven for ReLU and other functions that violets some of those properties... however, if you take `1 - sigmoid(x)` as your activation, all the properties are satisfied, and the derivative is strictly negative

This can be more "popularly" be seen also in the max-out activation function


---

