<p>The ReLU function has a parameter that determines the slope of the function when <span class="math-container">$x &lt; 0$</span>. If you want that constant to be <span class="math-container">$1/20$</span>, then the function that you have mentioned gets the required derivative.</p>

CC BY-SA 3.0
2018-02-25T02:16:44.230Z
So, using that example, for $x < 0$, the derivative would be $x * 0.5$. Is that correct?
Mike AI
CC BY-SA 3.0
2018-02-25T08:20:41.647Z
It would be $0.05$ because $1/20 = 0.05$.
Mohsin
CC BY-SA 4.0
2018-02-24T21:53:42.937Z
2020-05-30T12:56:41.543Z
2020-05-30T12:56:41.543Z
Mohsin
<p>Derivative gives the rate of change in <span class="math-container">$y$</span> for a small change in <span class="math-container">$x$</span> or the slope of a function at point <span class="math-container">$x$</span>.</p>

<p>In the above function,</p>

<pre><code>y = x      for x &gt;= 0,     i.e. y/x = 1
y = x/20   for x &lt; 0,      i.e.  y/x = 1/20
</code></pre>

<p>The following function returns the derivative of leaky ReLU as explained</p>

<pre><code>private double leaky_relu_derivative(double x)
{
    if (x &gt;= 0)
        return 1;
    else
        return 1.0 / 20;
}
</code></pre>

CC BY-SA 4.0
2018-06-21T14:29:40.780Z
2020-05-30T13:01:46.290Z
2020-05-30T13:01:46.290Z
Mahesh Nepal
<p>In terms of the Heaviside step function <span class="math-container">$$H(x)=\frac{1+sign(x)}{2}$$</span></p>
<p>the leaky ReLU can be written as
<span class="math-container">$$ (\alpha + (1-\alpha)H(x))x $$</span></p>
<p>so its derivative is</p>
<p><span class="math-container">$$ \alpha + (1-\alpha)H(x) $$</span></p>
<p>In your example, <span class="math-container">$\alpha = \frac{1}{20}$</span></p>

CC BY-SA 4.0
2022-11-22T23:36:21.603Z
2022-11-22T23:36:21.603Z
Jaume Oliver Lafont
<p>I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the derivative of the Leaky ReLU is.</p>

<p>Here is the C# code for the leaky RELU function which I got from <a href="https://www.codeproject.com/Articles/1220276/ReInventing-Neural-Networks" rel="nofollow noreferrer">this site</a>:</p>

<pre><code>private double leaky_relu(double x)
{
    if (x &gt;= 0)
        return x;
    else
        return x / 20;
}
</code></pre>

CC BY-SA 4.0
2020-05-30T12:54:41.683Z
See also [What is the derivative of Leaky ReLU?](https://stats.stackexchange.com/q/275521/82135).
nbro
<neural-networks><backpropagation><math><activation-functions><relu>
What is the derivative of the Leaky ReLU activation function?
Mike AI
# What is the derivative of the Leaky ReLU activation function?

(Asked by: username_0 on 2018-02-19)

I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the derivative of the Leaky ReLU is.
Here is the C# code for the leaky RELU function which I got from this site:
private double leaky_relu(double x)
{
    if (x >= 0)
        return x;
    else
        return x / 20;
}

**Tags:** neural-networks, backpropagation, math, activation-functions, relu

> **Comments:**
> **username_1**: See also [What is the derivative of Leaky ReLU?](https://stats.stackexchange.com/q/275521/82135).  

---

## 3 answers

### Answer by username_2 on 2018-02-24. Score: 1 | Accepted answer

The ReLU function has a parameter that determines the slope of the function when $x < 0$. If you want that constant to be $1/20$, then the function that you have mentioned gets the required derivative.

> **Comments:**
> **username_0**: So, using that example, for $x < 0$, the derivative would be $x * 0.5$. Is that correct?  
> **username_2**: It would be $0.05$ because $1/20 = 0.05$.  

---

### Answer by username_3 on 2018-06-21. Score: 2

Derivative gives the rate of change in $y$ for a small change in $x$ or the slope of a function at point $x$.
In the above function,
y = x      for x >= 0,     i.e. y/x = 1
y = x/20   for x < 0,      i.e.  y/x = 1/20

The following function returns the derivative of leaky ReLU as explained
private double leaky_relu_derivative(double x)
{
    if (x >= 0)
        return 1;
    else
        return 1.0 / 20;
}


---

### Answer by username_4 on 2022-11-22. Score: 0

In terms of the Heaviside step function $$H(x)=\frac{1+sign(x)}{2}$$

the leaky ReLU can be written as
$$ (\alpha + (1-\alpha)H(x))x $$

so its derivative is

$$ \alpha + (1-\alpha)H(x) $$

In your example, $\alpha = \frac{1}{20}$


---

