Answers.Body: <p>ReLU is <a href="https://en.wikipedia.org/wiki/Piecewise_linear_function" rel="nofollow noreferrer">piecewise linear function</a> that outputs the received input directly if it's positive, or outputs a zero. i.e., <span class="math-container">$max(0, x)$</span></p>
  <p><img src="https://miro.medium.com/max/1000/1*3JUMOqugWKB2SDra6x6v0A.png" alt="img alt" /></p>
  <h3>How significant is adding relu to full connected layers?</h3>
  <p>ReLU, being an <a href="https://en.wikipedia.org/wiki/Activation_function" rel="nofollow noreferrer">activation function</a>, will determine what the output of the nodes in your FCs are. Since it's a non-linear function, one significance is it will allow the nodes in your model to learn  complex mappings between the inputs and the outputs. Compared to using a linear function, it will allow back-propagation since it has a derivative, allowing the neural network to have the advantage of stacking of multiple FC layers.</p>
  <h3>Is it necessary?</h3>
  <p>ReLU (and non-linear activation functions in general) will introduce non-linear properties in a neural network that enables it to learn more complex arbitrary structures in the inputs. Without activation functions <strong>between the layers</strong>, your neural network will simply be a linear function, regardless of the number of layers it has. Why? A linear function + linear function gives a linear function. Additionally, see this <a href="https://stackoverflow.com/a/9783865">answer</a>.</p>
  <h3>How is the performance affected by adding ReLU?</h3>
  <p>Compared to no activation function at all, it will be slower to train but <a href="https://stats.stackexchange.com/a/253394/296297">will only behave like a linear regression model</a>. So ReLU will increase the power of the model by making it non-linear.
  However, compared to other non-linear activation functions like tanH, ReLU will speed up training as (1) its computation step is cheaper i.e., <span class="math-container">$0.0$</span> or <span class="math-container">$x$</span> without additional operations (2) its gradient just depends on the sign of the input <span class="math-container">$x$</span>. See <a href="https://stats.stackexchange.com/questions/226923/why-do-we-use-relu-in-neural-networks-and-how-do-we-use-it">this answer</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T11:22:26.763Z
Answers.LastActivityDate: 2020-11-07T11:22:26.763Z
Answers.UserDisplayName: mugoh
Body: <p>How significant is adding a ReLU to fully connected (FC) layers? Is it necessary, or how is the performance of a model affected by adding ReLU to FC layers?</p>
  
Tags: <neural-networks><deep-neural-networks><activation-functions><performance><relu>
Title: How is the performance of a model affected by adding a ReLU to fully connected layers?
UserDisplayName: sai_varshittha
ThreadText: # How is the performance of a model affected by adding a ReLU to fully connected layers?
  
  (Asked by: username_0 on 2020-11-07)
  
  How significant is adding a ReLU to fully connected (FC) layers? Is it necessary, or how is the performance of a model affected by adding ReLU to FC layers?
  
  **Tags:** neural-networks, deep-neural-networks, activation-functions, performance, relu
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-07. Score: 2 | Accepted answer
  
  ReLU is [piecewise linear function](https://en.wikipedia.org/wiki/Piecewise_linear_function) that outputs the received input directly if it's positive, or outputs a zero. i.e., $max(0, x)$
  
  ![img alt](https://miro.medium.com/max/1000/1*3JUMOqugWKB2SDra6x6v0A.png)
  
  ### How significant is adding relu to full connected layers?
  
  ReLU, being an [activation function](https://en.wikipedia.org/wiki/Activation_function), will determine what the output of the nodes in your FCs are. Since it's a non-linear function, one significance is it will allow the nodes in your model to learn complex mappings between the inputs and the outputs. Compared to using a linear function, it will allow back-propagation since it has a derivative, allowing the neural network to have the advantage of stacking of multiple FC layers.
  
  ### Is it necessary?
  
  ReLU (and non-linear activation functions in general) will introduce non-linear properties in a neural network that enables it to learn more complex arbitrary structures in the inputs. Without activation functions **between the layers**, your neural network will simply be a linear function, regardless of the number of layers it has. Why? A linear function + linear function gives a linear function. Additionally, see this [answer](https://stackoverflow.com/a/9783865).
  
  ### How is the performance affected by adding ReLU?
  
  Compared to no activation function at all, it will be slower to train but [will only behave like a linear regression model](https://stats.stackexchange.com/a/253394/296297). So ReLU will increase the power of the model by making it non-linear.
  However, compared to other non-linear activation functions like tanH, ReLU will speed up training as (1) its computation step is cheaper i.e., $0.0$ or $x$ without additional operations (2) its gradient just depends on the sign of the input $x$. See [this answer](https://stats.stackexchange.com/questions/226923/why-do-we-use-relu-in-neural-networks-and-how-do-we-use-it)
  
  
  ---
  
  