Answers.Body: <p>To know the form of your non-linear function,  firstly you should define the type of problem you are dealing with such as an image classification task. Secondly, pick the activation functions based on your task such as sigmoid, Tanh, ReLu, LeadyRelu, Softmax etc. Overall, your ANN performance mainly depends on the number of hidden layers (hidden units), selection of activation functions,  weight decay, momentum and dropout etc. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-12T14:03:00.303Z
Answers.LastActivityDate: 2019-10-12T14:03:00.303Z
Answers.UserDisplayName: Ali Usyd
Answers.Body: <p><strong>TL;DR</strong>:One does not know ahead of time what hyper-parameters will achieve optimal performance. So what you need is an <em>iterative</em> implementation strategy:</p>
  
  <p><strong>Implementation Strategy</strong></p>
  
  <p>When working with neural networks it is key to make sure that you spend your time wisely. It is possible to spend lots of time on a dead end simply because you made an assumption about your model at the very beginning. </p>
  
  <p>So when selecting activation functions and other hyper parameters don't over think things. That is, get a quick and dirty model up and running and tune from there. From this model you can iterate. For example, you could start with ReLU activations in the hidden layers and as you tune your model you could experiment with other other activations. </p>
  
  <p>That is, the data and the task at hand along with your tuning shape your model. A highly recommended video on this is A. Ng's lecture <a href="https://www.youtube.com/embed/JUJNGv_sb4Y" rel="nofollow noreferrer">here</a> and <a href="https://www.coursera.org/lecture/deep-neural-network/tuning-process-dknSn" rel="nofollow noreferrer">this video</a> from the A. Ng deep learning specialization.</p>
  
  <p>Some content not in the video is how to use <a href="https://www.ritchieng.com/machinelearning-learning-curve/" rel="nofollow noreferrer">learning curves</a> to help define your iterations. These help you decide what you should do next when your model is not achieving desired performance. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-24T22:51:37.527Z
Answers.Text: Do you know if this things have changed to nowadays?
Answers.UserDisplayName: Verónica Rmz.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-26T13:35:21.110Z
Answers.Text: @VerónicaRmz. Deeplearning.ai has just released an updated version of the deep learning specialization. They have one course specifically dedicated to tuning neural networks. I recommend checking it out on Coursera.
Answers.UserDisplayName: respectful
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-12T17:50:20.937Z
Answers.LastActivityDate: 2019-10-12T17:56:48.653Z
Answers.LastEditDate: 2019-10-12T17:56:48.653Z
Answers.UserDisplayName: respectful
Body: <p>I am new to neural networks. I would like to use them as a fitting or forecasting method.</p>
  
  <p>A simple NN model that does not contain hidden layers, that is, the input nodes are directly connected to the outputs nodes, represents a linear model.  Nonlinearity begins to appear in an ANN model when we have hidden nodes, in which a nonlinear function is assigned to the hidden nodes, and using minimization their weights are determined.</p>
  
  <p>How do we choose the non-linear activation function that should be assigned to each hidden neuron?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-13T14:31:58.500Z
Comments.Text: Choice of activation functions depends a lot on the task that you want to accomplish.
Comments.UserDisplayName: naive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-13T14:34:58.953Z
Comments.Text: [This](https://stats.stackexchange.com/questions/115258/comprehensive-list-of-activation-functions-in-neural-networks-with-pros-cons/) will be of help.
Comments.UserDisplayName: naive
Tags: <neural-networks><linear-regression><hidden-layers><regression><forecasting>
Title: How do we choose the activation function for each hidden node?
UserDisplayName: Nizar
ThreadText: # How do we choose the activation function for each hidden node?
  
  (Asked by: username_0 on 2019-10-12)
  
  I am new to neural networks. I would like to use them as a fitting or forecasting method.
  
  A simple NN model that does not contain hidden layers, that is, the input nodes are directly connected to the outputs nodes, represents a linear model. Nonlinearity begins to appear in an ANN model when we have hidden nodes, in which a nonlinear function is assigned to the hidden nodes, and using minimization their weights are determined.
  
  How do we choose the non-linear activation function that should be assigned to each hidden neuron?
  
  **Tags:** neural-networks, linear-regression, hidden-layers, regression, forecasting
  > **Comments:**
  > **username_1**: Choice of activation functions depends a lot on the task that you want to accomplish.  
  > **username_1**: [This](https://stats.stackexchange.com/questions/115258/comprehensive-list-of-activation-functions-in-neural-networks-with-pros-cons/) will be of help.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-10-12. Score: 1
  
  To know the form of your non-linear function, firstly you should define the type of problem you are dealing with such as an image classification task. Secondly, pick the activation functions based on your task such as sigmoid, Tanh, ReLu, LeadyRelu, Softmax etc. Overall, your ANN performance mainly depends on the number of hidden layers (hidden units), selection of activation functions, weight decay, momentum and dropout etc.
  
  
  ---
  
  ### Answer by username_3 on 2019-10-12. Score: 1
  
  **TL;DR**:One does not know ahead of time what hyper-parameters will achieve optimal performance. So what you need is an *iterative* implementation strategy:
  
  **Implementation Strategy**
  
  When working with neural networks it is key to make sure that you spend your time wisely. It is possible to spend lots of time on a dead end simply because you made an assumption about your model at the very beginning.
  
  So when selecting activation functions and other hyper parameters don't over think things. That is, get a quick and dirty model up and running and tune from there. From this model you can iterate. For example, you could start with ReLU activations in the hidden layers and as you tune your model you could experiment with other other activations.
  
  That is, the data and the task at hand along with your tuning shape your model. A highly recommended video on this is A. Ng's lecture [here](https://www.youtube.com/embed/JUJNGv_sb4Y) and [this video](https://www.coursera.org/lecture/deep-neural-network/tuning-process-dknSn) from the A. Ng deep learning specialization.
  
  Some content not in the video is how to use [learning curves](https://www.ritchieng.com/machinelearning-learning-curve/) to help define your iterations. These help you decide what you should do next when your model is not achieving desired performance.
  
  > **Comments:**
  > **username_4**: Do you know if this things have changed to nowadays?  
  > **username_3**: @VerónicaRmz. Deeplearning.ai has just released an updated version of the deep learning specialization. They have one course specifically dedicated to tuning neural networks. I recommend checking it out on Coursera.  
  
  ---
  
  