Answers.Body: <p>By definition from wikipedia:</p>
  <blockquote>
  <p>Let <span class="math-container">$\varphi$</span> be a nonconstant, bounded, and continuous function.</p>
  </blockquote>
  <p>This is also being proven for ReLU and other functions that violets some of those properties... however, if you take <code>1 - sigmoid(x)</code> as your activation, all the properties are satisfied, and the derivative is strictly negative</p>
  <p>This can be more &quot;popularly&quot; be seen also in the max-out activation function</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-17T00:59:57.307Z
Answers.LastActivityDate: 2022-08-17T00:59:57.307Z
Answers.UserDisplayName: Alberto
Body: <p>All the activation functions I see have positive derivatives.
  Will negative ReLU work as well as its positive counterpart or will it lead to instability?</p>
  
Tags: <neural-networks><backpropagation><activation-functions>
Title: Why is the derivative of activation function all positive?
UserDisplayName: user33869
ThreadText: # Why is the derivative of activation function all positive?
  
  (Asked by: username_0 on 2022-08-12)
  
  All the activation functions I see have positive derivatives.
  Will negative ReLU work as well as its positive counterpart or will it lead to instability?
  
  **Tags:** neural-networks, backpropagation, activation-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-17. Score: 1
  
  By definition from wikipedia:
  
  > Let $\varphi$ be a nonconstant, bounded, and continuous function.
  
  This is also being proven for ReLU and other functions that violets some of those properties... however, if you take `1 - sigmoid(x)` as your activation, all the properties are satisfied, and the derivative is strictly negative
  
  This can be more "popularly" be seen also in the max-out activation function
  
  
  ---
  
  