Answers.Body: <p>The ReLU function has a parameter that determines the slope of the function when <span class="math-container">$x &lt; 0$</span>. If you want that constant to be <span class="math-container">$1/20$</span>, then the function that you have mentioned gets the required derivative.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-25T02:16:44.230Z
Answers.Text: So, using that example, for $x < 0$, the derivative would be $x * 0.5$. Is that correct?
Answers.UserDisplayName: Mike AI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-25T08:20:41.647Z
Answers.Text: It would be $0.05$ because $1/20 = 0.05$.
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-02-24T21:53:42.937Z
Answers.LastActivityDate: 2020-05-30T12:56:41.543Z
Answers.LastEditDate: 2020-05-30T12:56:41.543Z
Answers.UserDisplayName: Mohsin
Answers.Body: <p>Derivative gives the rate of change in <span class="math-container">$y$</span> for a small change in <span class="math-container">$x$</span> or the slope of a function at point <span class="math-container">$x$</span>.</p>
  
  <p>In the above function,</p>
  
  <pre><code>y = x      for x &gt;= 0,     i.e. y/x = 1
  y = x/20   for x &lt; 0,      i.e.  y/x = 1/20
  </code></pre>
  
  <p>The following function returns the derivative of leaky ReLU as explained</p>
  
  <pre><code>private double leaky_relu_derivative(double x)
  {
      if (x &gt;= 0)
          return 1;
      else
          return 1.0 / 20;
  }
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T14:29:40.780Z
Answers.LastActivityDate: 2020-05-30T13:01:46.290Z
Answers.LastEditDate: 2020-05-30T13:01:46.290Z
Answers.UserDisplayName: Mahesh Nepal
Answers.Body: <p>In terms of the Heaviside step function <span class="math-container">$$H(x)=\frac{1+sign(x)}{2}$$</span></p>
  <p>the leaky ReLU can be written as
  <span class="math-container">$$ (\alpha + (1-\alpha)H(x))x $$</span></p>
  <p>so its derivative is</p>
  <p><span class="math-container">$$ \alpha + (1-\alpha)H(x) $$</span></p>
  <p>In your example, <span class="math-container">$\alpha = \frac{1}{20}$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-22T23:36:21.603Z
Answers.LastActivityDate: 2022-11-22T23:36:21.603Z
Answers.UserDisplayName: Jaume Oliver Lafont
Body: <p>I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the derivative of the Leaky ReLU is.</p>
  
  <p>Here is the C# code for the leaky RELU function which I got from <a href="https://www.codeproject.com/Articles/1220276/ReInventing-Neural-Networks" rel="nofollow noreferrer">this site</a>:</p>
  
  <pre><code>private double leaky_relu(double x)
  {
      if (x &gt;= 0)
          return x;
      else
          return x / 20;
  }
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-30T12:54:41.683Z
Comments.Text: See also [What is the derivative of Leaky ReLU?](https://stats.stackexchange.com/q/275521/82135).
Comments.UserDisplayName: nbro
Tags: <neural-networks><backpropagation><math><activation-functions><relu>
Title: What is the derivative of the Leaky ReLU activation function?
UserDisplayName: Mike AI
ThreadText: # What is the derivative of the Leaky ReLU activation function?
  
  (Asked by: username_0 on 2018-02-19)
  
  I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the derivative of the Leaky ReLU is.
  
  Here is the C# code for the leaky RELU function which I got from [this site](https://www.codeproject.com/Articles/1220276/ReInventing-Neural-Networks):
  
  ```
  private double leaky_relu(double x)
  {
      if (x >= 0)
          return x;
      else
          return x / 20;
  }
  
  ```
  
  **Tags:** neural-networks, backpropagation, math, activation-functions, relu
  
  > **Comments:**
  > **username_1**: See also [What is the derivative of Leaky ReLU?](https://stats.stackexchange.com/q/275521/82135).  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2018-02-24. Score: 1 | Accepted answer
  
  The ReLU function has a parameter that determines the slope of the function when $x < 0$. If you want that constant to be $1/20$, then the function that you have mentioned gets the required derivative.
  
  > **Comments:**
  > **username_0**: So, using that example, for $x < 0$, the derivative would be $x * 0.5$. Is that correct?  
  > **username_2**: It would be $0.05$ because $1/20 = 0.05$.  
  
  ---
  
  ### Answer by username_3 on 2018-06-21. Score: 2
  
  Derivative gives the rate of change in $y$ for a small change in $x$ or the slope of a function at point $x$.
  
  In the above function,
  
  ```
  y = x      for x >= 0,     i.e. y/x = 1
  y = x/20   for x < 0,      i.e.  y/x = 1/20
  
  ```
  
  The following function returns the derivative of leaky ReLU as explained
  
  ```
  private double leaky_relu_derivative(double x)
  {
      if (x >= 0)
          return 1;
      else
          return 1.0 / 20;
  }
  
  ```
  
  
  ---
  
  ### Answer by username_4 on 2022-11-22. Score: 0
  
  In terms of the Heaviside step function $$H(x)=\frac{1+sign(x)}{2}$$
  
  the leaky ReLU can be written as
  $$ (\alpha + (1-\alpha)H(x))x $$
  
  so its derivative is
  
  $$ \alpha + (1-\alpha)H(x) $$
  
  In your example, $\alpha = \frac{1}{20}$
  
  
  ---
  
  