Answers.Body: <h1>ReLU is non-linear by definition</h1>
  
  <blockquote>
    <p>In calculus and related areas, a <a href="https://en.wikipedia.org/wiki/Linear_function" rel="nofollow noreferrer">linear function</a> is a function whose graph is a straight line, that is a polynomial function of degree one or zero.</p>
  </blockquote>
  
  <p>Since the graph of the ReLU function <span class="math-container">$f(x) = \max(0,x)$</span> is not a straight line (equivalently, it cannot be expressed in the form <span class="math-container">$f(x) = mx + c$</span>), by definition it is not linear.</p>
  
  <h1>ReLU <em>is</em> piecewise linear</h1>
  
  <p>ReLU <em>is</em> <a href="https://en.wikipedia.org/wiki/Piecewise_linear_function" rel="nofollow noreferrer">piecewise linear</a> on the bounds <span class="math-container">$(-\inf,0]$</span> and <span class="math-container">$[0,\inf)$</span>:</p>
  
  <p><span class="math-container">$$
  f(x) = \max(0,x) = \begin{cases}
                 0               &amp; x \le 0\\
                 x               &amp; x \gt 0\\
         \end{cases}
  $$</span></p>
  
  <p>But this is still non-linear on the entire domain:</p>
  
  <p><a href="https://i.sstatic.net/Gg4bV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Gg4bV.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-26T16:15:50.760Z
Answers.LastActivityDate: 2020-05-26T16:34:38.610Z
Answers.LastEditDate: 2020-05-26T16:34:38.610Z
Answers.UserDisplayName: brazofuerte
Answers.Body: <p><strong>Short Answer:</strong> Yes</p>
  
  <p><strong>Visually:</strong></p>
  
  <p><a href="https://i.sstatic.net/Yqgclm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Yqgclm.png" alt="enter image description here"></a></p>
  
  <p>if you see the image from <a href="https://en.wikipedia.org/wiki/Linear_function" rel="nofollow noreferrer">wikipedia</a>, it shown that ReLU (the blue line) is non-Linear (the line is not straight, it turns in 0). You can also check "visual" definition of linear function in <a href="https://en.wikipedia.org/wiki/Linear_function" rel="nofollow noreferrer">wikipedia</a>:</p>
  
  <blockquote>
    <p>"In calculus and related areas, a linear function is a function whose graph is a straight line"</p>
  </blockquote>
  
  <p><strong>Mathematically:</strong></p>
  
  <p>Linear function of one variable can be defined as: </p>
  
  <p><span class="math-container">$ f(x) = ax + b $</span> </p>
  
  <p>If you plot that function in 2D, it will give you a straight line. Then, the form of linear function with multi variables:</p>
  
  <p><span class="math-container">$ f(x_1, x_2, ..., x_n) = a_1x_1 + a_2x_2 + ... + a_nx_n + b $</span></p>
  
  <p>If you again plot that function in the correct dimension it also give you a straight line. And if you that function carefully, it similar with calculation that happen in a neuron. That's why neuron addition and multiplication is a linear function:</p>
  
  <p><span class="math-container">$ f(x_1, x_2, ..., x_n) = w_1x_1 + w_2x_2 + ... + w_nx_n + b $</span></p>
  
  <p>Adding more layer of linear functions doesn't make the function become "complex" for example, if you have <span class="math-container">$f(x)$</span> like below and then you put another layer of linear function <span class="math-container">$g(x)$</span> on top of it:</p>
  
  <p><span class="math-container">$f(x) = ax + b$</span></p>
  
  <p><span class="math-container">$g(x) = cf(x) + d = cax + cb + d$</span></p>
  
  <p>as the neural network is trained to find the value of <span class="math-container">$a,b,c,d$</span>, we can group the constant from the formula above, and then rewrite to:</p>
  
  <p><span class="math-container">$h(x) = mx + n$</span></p>
  
  <p>with <span class="math-container">$m=ca$</span> and <span class="math-container">$n=cb+d$</span>. So without non-linear function the layer of neural network is useless, it only give you another "simple" linear function</p>
  
  <p>ReLU formula is a <span class="math-container">$f(x)=max(0,x)$</span>, it produces non-linearity as you can't write to linear function format. Using this function will give you "complexity" when you add more layer on top of it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-29T02:27:13.590Z
Answers.LastActivityDate: 2020-03-29T02:55:43.340Z
Answers.LastEditDate: 2020-03-29T02:55:43.340Z
Answers.UserDisplayName: malioboro
Body: <p>According to <a href="https://www.deeplearning-academy.com/p/ai-wiki-activation-functions" rel="nofollow noreferrer">this blog post</a></p>
  
  <blockquote>
    <p>The purpose of an activation function is to add some kind of non-linear property to the function</p>
  </blockquote>
  
  <p>The sigmoid is typically used as an activation function of a unit of a neural network in order to introduce non-linearity.</p>
  
  <p>Is ReLU a non-linear activation function? And why? If not, then why is it used as an activation function of neural networks?</p>
  
Tags: <neural-networks><deep-learning><math><activation-functions><relu>
Title: Is ReLU a non-linear activation function?
UserDisplayName: slowmonk
ThreadText: # Is ReLU a non-linear activation function?
  
  (Asked by: username_0 on 2020-03-29)
  
  According to [this blog post](https://www.deeplearning-academy.com/p/ai-wiki-activation-functions)
  
  > The purpose of an activation function is to add some kind of non-linear property to the function
  
  The sigmoid is typically used as an activation function of a unit of a neural network in order to introduce non-linearity.
  
  Is ReLU a non-linear activation function? And why? If not, then why is it used as an activation function of neural networks?
  
  **Tags:** neural-networks, deep-learning, math, activation-functions, relu
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-05-26. Score: 3
  
  # ReLU is non-linear by definition
  
  > In calculus and related areas, a [linear function](https://en.wikipedia.org/wiki/Linear_function) is a function whose graph is a straight line, that is a polynomial function of degree one or zero.
  
  Since the graph of the ReLU function $f(x) = \max(0,x)$ is not a straight line (equivalently, it cannot be expressed in the form $f(x) = mx + c$), by definition it is not linear.
  
  # ReLU *is* piecewise linear
  
  ReLU *is* [piecewise linear](https://en.wikipedia.org/wiki/Piecewise_linear_function) on the bounds $(-\inf,0]$ and $[0,\inf)$:
  
  $$
  f(x) = \max(0,x) = \begin{cases}
  0 & x \le 0\\
  x & x \gt 0\\
  \end{cases}
  $$
  
  But this is still non-linear on the entire domain:
  
  [![enter image description here](https://i.sstatic.net/Gg4bV.png)](https://i.sstatic.net/Gg4bV.png)
  
  
  ---
  
  ### Answer by username_2 on 2020-03-29. Score: 2
  
  **Short Answer:** Yes
  
  **Visually:**
  
  [![enter image description here](https://i.sstatic.net/Yqgclm.png)](https://i.sstatic.net/Yqgclm.png)
  
  if you see the image from [wikipedia](https://en.wikipedia.org/wiki/Linear_function), it shown that ReLU (the blue line) is non-Linear (the line is not straight, it turns in 0). You can also check "visual" definition of linear function in [wikipedia](https://en.wikipedia.org/wiki/Linear_function):
  
  > "In calculus and related areas, a linear function is a function whose graph is a straight line"
  
  **Mathematically:**
  
  Linear function of one variable can be defined as:
  
  $ f(x) = ax + b $
  
  If you plot that function in 2D, it will give you a straight line. Then, the form of linear function with multi variables:
  
  $ f(x\_1, x\_2, ..., x\_n) = a\_1x\_1 + a\_2x\_2 + ... + a\_nx\_n + b $
  
  If you again plot that function in the correct dimension it also give you a straight line. And if you that function carefully, it similar with calculation that happen in a neuron. That's why neuron addition and multiplication is a linear function:
  
  $ f(x\_1, x\_2, ..., x\_n) = w\_1x\_1 + w\_2x\_2 + ... + w\_nx\_n + b $
  
  Adding more layer of linear functions doesn't make the function become "complex" for example, if you have $f(x)$ like below and then you put another layer of linear function $g(x)$ on top of it:
  
  $f(x) = ax + b$
  
  $g(x) = cf(x) + d = cax + cb + d$
  
  as the neural network is trained to find the value of $a,b,c,d$, we can group the constant from the formula above, and then rewrite to:
  
  $h(x) = mx + n$
  
  with $m=ca$ and $n=cb+d$. So without non-linear function the layer of neural network is useless, it only give you another "simple" linear function
  
  ReLU formula is a $f(x)=max(0,x)$, it produces non-linearity as you can't write to linear function format. Using this function will give you "complexity" when you add more layer on top of it.
  
  
  ---
  
  