Answers.Body: <p>Of course, it is possible, but why do you want to do this?</p>
  
  <p>Let's think about it. Imagine your weights for that layer is a matrix full of ones, so, if you have no bias, then your output of that layer would be the sum of all values in the <span class="math-container">$h_1$</span>, <span class="math-container">$h_j$</span>, <span class="math-container">$h_n$</span> neurons, right? So, it is possible to sum all the values together, give the output to an activation function, and then you'll have your output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-09T17:24:05.327Z
Answers.LastActivityDate: 2019-10-10T02:15:24.603Z
Answers.LastEditDate: 2019-10-10T02:15:24.603Z
Answers.UserDisplayName: basilisk
Answers.Body: <p><strong>TL;DR</strong>: This is possible but removing the activations will decrease the expressivity of the network because it will become mathematically equivalent to a single neuron.</p>
  
  <p><strong>Mathematical Explanation</strong></p>
  
  <p>The outputs of your intermediate neurons (in the absence of activation functions) are now:</p>
  
  <p><span class="math-container">$\text{(1)}\quad H_i(x) = \sum_{j=1}^nw_{ij}\cdot x_j+b_i$</span></p>
  
  <p>You are then summing each <span class="math-container">$H_i$</span>:</p>
  
  <p><span class="math-container">$\text{(2)}\quad\mathcal{H(x)}=\sum_{i=1}^nH_i(x)$</span></p>
  
  <p>You then pass <span class="math-container">$\mathcal{H(x)}$</span> to some activation say <span class="math-container">$g$</span>:</p>
  
  <p><span class="math-container">$\text{(3)}\quad\hat y = g(\mathcal{H(x)})$</span></p>
  
  <p>The trouble is that the inner term <span class="math-container">$\mathcal{H(x)}$</span> mathematically reduces to a single linear operation on <span class="math-container">$x$</span>. Proof:</p>
  
  <ol>
  <li><span class="math-container">$\mathcal{H(x)}=\sum_{i=1}^nH_i(x)$</span>. Substituting in (1):</li>
  <li><span class="math-container">$\mathcal{H(x)}=\sum_{i=1}^n((\sum_{j=1}^nw_{ij}\cdot x_j)+b_i)$</span>. This can be re-aranged :</li>
  <li><span class="math-container">$\mathcal{H(x)}=(\sum_{j=1}^n(\sum_{i=1}^nw_{ij})\cdot x_j)+\sum_{i=1}^nb_i$</span>. But this reduces to:</li>
  <li><span class="math-container">$\mathcal{H(x)}=\sum_{j=1}^n\tilde w_{j}\cdot x_j+\tilde b$</span>. Where <span class="math-container">$\tilde w_{j},\tilde b$</span> are scalars. </li>
  </ol>
  
  <p>Thus, without the non-linear activations (3) mathematically reduces to a single neuron. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-09T18:34:15.507Z
Answers.LastActivityDate: 2019-10-10T03:33:40.743Z
Answers.LastEditDate: 2019-10-10T03:33:40.743Z
Answers.UserDisplayName: respectful
Body: <p>I'm just started to learn deep learning and I have a question about this neural network:</p>
  
  <p><a href="https://i.sstatic.net/W6FuR.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/W6FuR.png" alt="enter image description here"></a></p>
  
  <p>I think <span class="math-container">$h_1$</span>, <span class="math-container">$h_j$</span> and <span class="math-container">$h_n$</span> are perceptrons. So, if they are perceptrons, all of them will have an activation function.</p>
  
  <p>I'm wondering if it is possible to have only one activation function, and sum the output to all of the perceptrons, and pass that sum to that activation function. And the output of this activation function will be <span class="math-container">$y$</span>.</p>
  
  <p>I will have this network, where <span class="math-container">$H1$</span>, <span class="math-container">$Hj$</span> and <span class="math-container">$Hn$</span> don't have activation function:
  <a href="https://i.sstatic.net/2OuoU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2OuoU.png" alt="enter image description here"></a></p>
  
  <p>The input for the activation function will be the sum of the outputs of <span class="math-container">$H1$</span>, <span class="math-container">$Hj$</span> and <span class="math-container">$Hn$</span> without been processed by an activation function.</p>
  
  <p>Is that possible (or is it a good idea)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-09T16:35:02.680Z
Comments.Text: Your question is a little unclear can you clarify: "I'm wondering if it is possible to have only one, and sum the output to all of the perceptrons, and pass that sum to an activation function." I think I understand but I want to make sure I truly understand your question. Do you mean is it possible to add another hidden layer with only one neuron?
Comments.UserDisplayName: respectful
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-09T17:15:15.273Z
Comments.Text: you mean one activation function for every neuron ?
Comments.UserDisplayName: basilisk
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-09T17:19:32.563Z
Comments.Text: I have updated my question.
Comments.UserDisplayName: VansFannel
Tags: <deep-learning><activation-functions>
Title: Can multiple activation functions be replaced with a single activation function?
UserDisplayName: VansFannel
ThreadText: # Can multiple activation functions be replaced with a single activation function?
  
  (Asked by: username_0 on 2019-10-09)
  
  I'm just started to learn deep learning and I have a question about this neural network:
  
  [![enter image description here](https://i.sstatic.net/W6FuR.png)](https://i.sstatic.net/W6FuR.png)
  
  I think $h\_1$, $h\_j$ and $h\_n$ are perceptrons. So, if they are perceptrons, all of them will have an activation function.
  
  I'm wondering if it is possible to have only one activation function, and sum the output to all of the perceptrons, and pass that sum to that activation function. And the output of this activation function will be $y$.
  
  I will have this network, where $H1$, $Hj$ and $Hn$ don't have activation function:
  [![enter image description here](https://i.sstatic.net/2OuoU.png)](https://i.sstatic.net/2OuoU.png)
  
  The input for the activation function will be the sum of the outputs of $H1$, $Hj$ and $Hn$ without been processed by an activation function.
  
  Is that possible (or is it a good idea)?
  
  **Tags:** deep-learning, activation-functions
  > **Comments:**
  > **username_1**: Your question is a little unclear can you clarify: "I'm wondering if it is possible to have only one, and sum the output to all of the perceptrons, and pass that sum to an activation function." I think I understand but I want to make sure I truly understand your question. Do you mean is it possible to add another hidden layer with only one neuron?  
  > **username_2**: you mean one activation function for every neuron ?  
  > **username_0**: I have updated my question.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-10-09. Score: 2 | Accepted answer
  
  **TL;DR**: This is possible but removing the activations will decrease the expressivity of the network because it will become mathematically equivalent to a single neuron.
  
  **Mathematical Explanation**
  
  The outputs of your intermediate neurons (in the absence of activation functions) are now:
  
  $\text{(1)}\quad H\_i(x) = \sum\_{j=1}^nw\_{ij}\cdot x\_j+b\_i$
  
  You are then summing each $H\_i$:
  
  $\text{(2)}\quad\mathcal{H(x)}=\sum\_{i=1}^nH\_i(x)$
  
  You then pass $\mathcal{H(x)}$ to some activation say $g$:
  
  $\text{(3)}\quad\hat y = g(\mathcal{H(x)})$
  
  The trouble is that the inner term $\mathcal{H(x)}$ mathematically reduces to a single linear operation on $x$. Proof:
  
  1. $\mathcal{H(x)}=\sum\_{i=1}^nH\_i(x)$. Substituting in (1):
  2. $\mathcal{H(x)}=\sum\_{i=1}^n((\sum\_{j=1}^nw\_{ij}\cdot x\_j)+b\_i)$. This can be re-aranged :
  3. $\mathcal{H(x)}=(\sum\_{j=1}^n(\sum\_{i=1}^nw\_{ij})\cdot x\_j)+\sum\_{i=1}^nb\_i$. But this reduces to:
  4. $\mathcal{H(x)}=\sum\_{j=1}^n\tilde w\_{j}\cdot x\_j+\tilde b$. Where $\tilde w\_{j},\tilde b$ are scalars.
  
  Thus, without the non-linear activations (3) mathematically reduces to a single neuron.
  
  
  ---
  
  ### Answer by username_2 on 2019-10-09. Score: 0
  
  Of course, it is possible, but why do you want to do this?
  
  Let's think about it. Imagine your weights for that layer is a matrix full of ones, so, if you have no bias, then your output of that layer would be the sum of all values in the $h\_1$, $h\_j$, $h\_n$ neurons, right? So, it is possible to sum all the values together, give the output to an activation function, and then you'll have your output.
  
  
  ---
  
  