Answers.Body: <p>As you stated, it's popular to have some form of a <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">rectified linear unit (ReLU) activation</a> in hidden layers and the output layer is often a <a href="https://en.wikipedia.org/wiki/Softmax_function" rel="nofollow noreferrer">softmax</a> or sigmoid (depending also on the problem: multi-class or binary classification, respectively), which provides an output that can be viewed as a probability distribution.</p>
  <p>You could generalize this further to blocks of different activation functions within the same layer. This is something I've thought about, haven't done, but imagine has been attempted. In some sense, the idea here would be to allow for a subsection of the network to develop a representation that may not be feasible otherwise. These different representations within the same layer would then be unified by subsequent layers as we move closer to the output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-23T21:14:24.443Z
Answers.Text: regarding your last paragraph that's what I'm thinking about. Is there ever a benefit to have a layer where each Perceptron has a different activation function?
Answers.UserDisplayName: user8714896
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-23T21:16:19.007Z
Answers.Text: I updated the answer, maybe it addresses the question a bit better. I think the larger point is that anything is possible; neural networks are highly flexible models afterall. But the motivation for doing so is crucial – what would you hope to gain by having a block of neurons with alternate activations in the same layer? Why is this gain not possible without it? I suspect that answers that motivate such an architecture are hard to find. Further, as far as I'm aware, these aren't features that are readily available in existing software libraries.
Answers.UserDisplayName: Greenstick
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-23T23:10:27.350Z
Answers.Text: @Greenstick I suspect with some problems where characteristics of the objective function are not obvious that experimenting with a mixture of activation functions might afford a better approximation to the function trying to be learned.
Answers.UserDisplayName: user8714896
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T20:07:47.710Z
Answers.Text: @user8714896 This is in effect done. Certain activations (e.g. PReLU) take in parameters that have their own associated weights (multiple params could exist for a layer); the neural net then learns how to parameterize these activations to minimize the objective function. You lose some interpretability and (maybe) motivation doing this (e.g. the biological motivation discussed above), but the nice part is that you don't have to precisely *know* the activation ahead of time – the neural net figures it out. Hope that helps; if this is in line with the answer you're looking for, please accept it.
Answers.UserDisplayName: Greenstick
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T16:26:37.073Z
Answers.Text: This answer would improve considerably if you could do a little bit of research to find some research papers that show that, indeed, different activation functions can be used for different hidden neurons.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-23T21:08:58.547Z
Answers.LastActivityDate: 2021-01-17T16:21:04.440Z
Answers.LastEditDate: 2021-01-17T16:21:04.440Z
Answers.UserDisplayName: Greenstick
Body: <p>Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output layer, which typically has a different activation function) of the MLP? Is this a requirement, are there any advantages, or maybe is it just a rule of thumb?</p>
  
Tags: <neural-networks><deep-learning><activation-functions><multilayer-perceptrons><hyper-parameters>
Title: Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function?
UserDisplayName: user8714896
ThreadText: # Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function?
  
  (Asked by: username_0 on 2020-07-23)
  
  Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output layer, which typically has a different activation function) of the MLP? Is this a requirement, are there any advantages, or maybe is it just a rule of thumb?
  
  **Tags:** neural-networks, deep-learning, activation-functions, multilayer-perceptrons, hyper-parameters
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-23. Score: 2 | Accepted answer
  
  As you stated, it's popular to have some form of a [rectified linear unit (ReLU) activation](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) in hidden layers and the output layer is often a [softmax](https://en.wikipedia.org/wiki/Softmax_function) or sigmoid (depending also on the problem: multi-class or binary classification, respectively), which provides an output that can be viewed as a probability distribution.
  
  You could generalize this further to blocks of different activation functions within the same layer. This is something I've thought about, haven't done, but imagine has been attempted. In some sense, the idea here would be to allow for a subsection of the network to develop a representation that may not be feasible otherwise. These different representations within the same layer would then be unified by subsequent layers as we move closer to the output.
  
  > **Comments:**
  > **username_0**: regarding your last paragraph that's what I'm thinking about. Is there ever a benefit to have a layer where each Perceptron has a different activation function?  
  > **username_1**: I updated the answer, maybe it addresses the question a bit better. I think the larger point is that anything is possible; neural networks are highly flexible models afterall. But the motivation for doing so is crucial – what would you hope to gain by having a block of neurons with alternate activations in the same layer? Why is this gain not possible without it? I suspect that answers that motivate such an architecture are hard to find. Further, as far as I'm aware, these aren't features that are readily available in existing software libraries.  
  > **username_0**: @Greenstick I suspect with some problems where characteristics of the objective function are not obvious that experimenting with a mixture of activation functions might afford a better approximation to the function trying to be learned.  
  > **username_1**: @user8714896 This is in effect done. Certain activations (e.g. PReLU) take in parameters that have their own associated weights (multiple params could exist for a layer); the neural net then learns how to parameterize these activations to minimize the objective function. You lose some interpretability and (maybe) motivation doing this (e.g. the biological motivation discussed above), but the nice part is that you don't have to precisely *know* the activation ahead of time – the neural net figures it out. Hope that helps; if this is in line with the answer you're looking for, please accept it.  
  > **username_2**: This answer would improve considerably if you could do a little bit of research to find some research papers that show that, indeed, different activation functions can be used for different hidden neurons.  
  
  ---
  
  