Answers.Body: <p>Activation function is a non-linear function. Operation in a neuron without activation function is just a linear function. If we don't put activation function between operation of neurons, then the function "Layer" is useless.</p>
  
  <p>for example if you have two layer network, when you are doing forward-propagation, your output (without activation function) of your first layer will be calculated as:</p>
  
  <p><span class="math-container">$O_1 = W_1X+b_1 $</span></p>
  
  <p>Then your output of your second layer will be:</p>
  
  <p><span class="math-container">$O_2 = W_2O_1+b_2 $</span></p>
  
  <p>If we substitute <span class="math-container">$O_1$</span>, so the output of your second layer can be calculated as:</p>
  
  <p><span class="math-container">$O_2 = W_2(W_1X+b_1)+b_2 $</span></p>
  
  <p>or simply</p>
  
  <p><span class="math-container">$O_2 = W_2W_1X+W_2b_1+b_2 $</span></p>
  
  <p>As we train neural network to optimize the value of <span class="math-container">$W$</span> and <span class="math-container">$b$</span> (we train to find the best value of it) so instead of training neural network with two layers, we actually just train a one layer network. From the latter formula we can said <span class="math-container">$W_2W_1 = W_3$</span> and <span class="math-container">$W_2b_1+b_2 = b_3$</span> so our two layer network is just another linear model:</p>
  
  <p><span class="math-container">$O_2 = W_3X+b_3 $</span></p>
  
  <p>We don't want that, we add layers to get more complex model. That's why we use Activation function that is non-linear function. To prevent our deep model is just become a simple linear function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T07:52:08.067Z
Answers.Text: @DuttaA that's true, I'm sorry for answered it too hastily, after re-read the question I realize the question is unclear, now I'll wait additional explanation from OP
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T08:16:46.857Z
Answers.Text: I understand that a linear line cannot separate data scattered in complex manner but Then why we don't use simple polynomials. why specifically sigmoid, or tanh or ReLu what exactly they are doing ?
Answers.UserDisplayName: Harshal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T06:59:41.220Z
Answers.LastActivityDate: 2019-02-01T06:59:41.220Z
Answers.UserDisplayName: malioboro
Body: <p>Is the role played by activation function significant only during the training of neural network or they play their role during testing (after training we supply data for prediction) the network.</p>
  
  <p>I understand that a linear line cannot separate data scattered in complex manner but
  Then why we don't used simple polynomials.</p>
  
  <p>why specifically sigmoid, or tanh or ReLu what exactly they are doing ?</p>
  
  <p>What Activation functions do when we are supplying data during training and </p>
  
  <p>And when we supply test data once we have trained the network and we input test data for prediction?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-29T11:43:07.547Z
Comments.Text: Possible duplicate of [What is the purpose of an activation function in neural networks?](https://ai.stackexchange.com/questions/5493/what-is-the-purpose-of-an-activation-function-in-neural-networks)
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-03T09:33:11.000Z
Comments.Text: You're asking too many questions in the same post.
Comments.UserDisplayName: nbro
Tags: <neural-networks>
Title: What role the activation function plays in the forward pass and how it is different from backpropagation
UserDisplayName: Harshal
ThreadText: # What role the activation function plays in the forward pass and how it is different from backpropagation
  
  (Asked by: username_0 on 2019-02-01)
  
  Is the role played by activation function significant only during the training of neural network or they play their role during testing (after training we supply data for prediction) the network.
  
  I understand that a linear line cannot separate data scattered in complex manner but
  Then why we don't used simple polynomials.
  
  why specifically sigmoid, or tanh or ReLu what exactly they are doing ?
  
  What Activation functions do when we are supplying data during training and
  
  And when we supply test data once we have trained the network and we input test data for prediction?
  
  **Tags:** neural-networks
  
  > **Comments:**
  > **username_1**: You're asking too many questions in the same post.  
  > **username_1**: Possible duplicate of [What is the purpose of an activation function in neural networks?](https://ai.stackexchange.com/questions/5493/what-is-the-purpose-of-an-activation-function-in-neural-networks)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-02-01. Score: 2
  
  Activation function is a non-linear function. Operation in a neuron without activation function is just a linear function. If we don't put activation function between operation of neurons, then the function "Layer" is useless.
  
  for example if you have two layer network, when you are doing forward-propagation, your output (without activation function) of your first layer will be calculated as:
  
  $O\_1 = W\_1X+b\_1 $
  
  Then your output of your second layer will be:
  
  $O\_2 = W\_2O\_1+b\_2 $
  
  If we substitute $O\_1$, so the output of your second layer can be calculated as:
  
  $O\_2 = W\_2(W\_1X+b\_1)+b\_2 $
  
  or simply
  
  $O\_2 = W\_2W\_1X+W\_2b\_1+b\_2 $
  
  As we train neural network to optimize the value of $W$ and $b$ (we train to find the best value of it) so instead of training neural network with two layers, we actually just train a one layer network. From the latter formula we can said $W\_2W\_1 = W\_3$ and $W\_2b\_1+b\_2 = b\_3$ so our two layer network is just another linear model:
  
  $O\_2 = W\_3X+b\_3 $
  
  We don't want that, we add layers to get more complex model. That's why we use Activation function that is non-linear function. To prevent our deep model is just become a simple linear function.
  
  > **Comments:**
  > **username_2**: @DuttaA that's true, I'm sorry for answered it too hastily, after re-read the question I realize the question is unclear, now I'll wait additional explanation from OP  
  > **username_0**: I understand that a linear line cannot separate data scattered in complex manner but Then why we don't use simple polynomials. why specifically sigmoid, or tanh or ReLu what exactly they are doing ?  
  
  ---
  
  