Answers.Body: <p>From <a href="https://www.quora.com/Why-does-every-node-in-a-layer-use-the-same-activation-function" rel="nofollow noreferrer">here</a>:</p>
  
  <blockquote>
    <p>Using other activation functions don’t provide significant improvement in performance and tweaking them doesn’t provide any big improvement. So as per simplicity we use same activation function for most of the case in Deep Neural Networks.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-30T00:37:03.877Z
Answers.Text: Many thanks, so should the same activation function be used throughout the entire network/model, or is there still merit in using different activation functions for different layers in the same model?
Answers.UserDisplayName: lfgtm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-30T10:29:54.740Z
Answers.Text: @lfgtm Normally, there is an improvement in mixing activation functions across a network (keeping layers to just 1 though). An example of this, is in a CNN, where the net is all ReLU activations until the last layer where, for classification, it is a softmax activation.
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-30T12:56:33.833Z
Answers.Text: Many thanks this answers my questions. Kind Regards.
Answers.UserDisplayName: lfgtm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-03T11:28:29.383Z
Answers.Text: To me, this answer is not satisfactory, because you're quoting an answer from Quora, which may not even be reliable. There's no reference to a research paper that really supports the truth of this claim. What I suggest is that you read some literature about the topic, then edit this answer to improve it with references to good research material.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-30T00:27:56.933Z
Answers.LastActivityDate: 2019-12-30T00:27:56.933Z
Answers.UserDisplayName: Recessive
Body: <p>I'm new to machine learning (so excuse my nomenclature), and not being a python developer, I decided to jump in at the deep (no pun intended) end writing my own framework in C++.</p>
  <p>In my current design, I have given each neuron/cell the possibility to have a different activation function. Is this a plausible design for a neural network? A lot of the examples I see use the same activation function for all neurons in a given layer.</p>
  <p>Is there a model which may require this, or should all neurons in a layer use the same activation function? Would I be correct in using different activation functions for different layers in the same model, or would all layers have the same activation function within a model?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-02T01:18:49.633Z
Comments.Text: It's interesting that you take the challenge, but knowing python will help you a lot in the field of deep learning. If you already know C++ learning python will not take you much.
Comments.UserDisplayName: Y K
Tags: <neural-networks><activation-functions><hidden-layers><network-design>
Title: Do all neurons in a layer have the same activation function?
UserDisplayName: lfgtm
ThreadText: # Do all neurons in a layer have the same activation function?
  
  (Asked by: username_0 on 2019-12-30)
  
  I'm new to machine learning (so excuse my nomenclature), and not being a python developer, I decided to jump in at the deep (no pun intended) end writing my own framework in C++.
  
  In my current design, I have given each neuron/cell the possibility to have a different activation function. Is this a plausible design for a neural network? A lot of the examples I see use the same activation function for all neurons in a given layer.
  
  Is there a model which may require this, or should all neurons in a layer use the same activation function? Would I be correct in using different activation functions for different layers in the same model, or would all layers have the same activation function within a model?
  
  **Tags:** neural-networks, activation-functions, hidden-layers, network-design
  > **Comments:**
  > **username_1**: It's interesting that you take the challenge, but knowing python will help you a lot in the field of deep learning. If you already know C++ learning python will not take you much.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-12-30. Score: 2 | Accepted answer
  
  From [here](https://www.quora.com/Why-does-every-node-in-a-layer-use-the-same-activation-function):
  
  > Using other activation functions don’t provide significant improvement in performance and tweaking them doesn’t provide any big improvement. So as per simplicity we use same activation function for most of the case in Deep Neural Networks.
  
  > **Comments:**
  > **username_0**: Many thanks, so should the same activation function be used throughout the entire network/model, or is there still merit in using different activation functions for different layers in the same model?  
  > **username_2**: @lfgtm Normally, there is an improvement in mixing activation functions across a network (keeping layers to just 1 though). An example of this, is in a CNN, where the net is all ReLU activations until the last layer where, for classification, it is a softmax activation.  
  > **username_0**: Many thanks this answers my questions. Kind Regards.  
  > **username_3**: To me, this answer is not satisfactory, because you're quoting an answer from Quora, which may not even be reliable. There's no reference to a research paper that really supports the truth of this claim. What I suggest is that you read some literature about the topic, then edit this answer to improve it with references to good research material.  
  
  ---
  
  