Answers.Body: <p>By itself, I'm not sure it's possible to know. It's possible the slides were old. Or, the intended purpose was to mention how as sigmoid ranges from 0 to 1. Mostly, it looks like it was intended to bring up gradient descent. But it could also be an entry point to the discussion of other methods such as ReLU. Either that or perhaps some sort of norming function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-24T04:38:01.610Z
Answers.LastActivityDate: 2019-10-24T04:38:01.610Z
Answers.UserDisplayName: hrokr
Answers.Body: <p>Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e</p>
  
  <pre><code>layer_output = Weights.layer_input + bias
  </code></pre>
  
  <p>Consider a 2 layer neural network, the outputs from layer one will be:
      x2 = W1*x1 + b1
  Now we pass the same input to the second layer, which will be</p>
  
  <pre><code>x3 = W2x*2 + b2
  Also x2 = W1*x1 + b1
  Substituting back, we have:
  x3 = W2(W1*x1 + b1) + b2
  x3 = (W2W1)*x1 + (W2*b1 + b2)
  x3 = W*x1 + b
  </code></pre>
  
  <p>Oh no! We still got a linear function. No matter how many layers we add, we will still get a linear function. In that case, our network will never be able to approximate any non linear functions.</p>
  
  <p>So what is the solution?</p>
  
  <p>We will simply add some non linear functions in between. These functions are called activation functions. Some of these functions include:</p>
  
  <ul>
  <li>ReLU</li>
  <li>Sigmoid</li>
  <li>tanh</li>
  <li>Softmax</li>
  </ul>
  
  <p>and there are a lot more of them.</p>
  
  <p>Yay! Our network is no more linear!</p>
  
  <p>We have a lot of different non linear functions, and each of them serve a different purpose.</p>
  
  <p>For example, ReLU is simple and computationally cheap.
      ReLU(x) = max(0, x)
  Sigmoid outputs are between 0 and 1.
  tanh is similar to sigmoid, but zero centered, with outputs from -1 to 1
  Softmax is usually used if you want to represent any vector as a discrete probability distribution.</p>
  
  <p>Hope you are having a great day!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-28T19:31:25.873Z
Answers.LastActivityDate: 2019-10-28T19:31:25.873Z
Answers.UserDisplayName: Sooryakiran Pallikulathil
Body: <p><a href="https://i.sstatic.net/hKGIu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hKGIu.png" alt="enter image description here"></a></p>
  
  <p>I got this slide from CMU's lecture notes. The <span class="math-container">$x_i$</span>s on the right are inputs and the <span class="math-container">$w_i$</span>s are weights that get multiplied together then summed up at each hidden layer node. So I'm assuming this is a node in the hidden layer.  </p>
  
  <p>What is the mathematical reason for taking the sum of the weights and inputs and inputting that into a sigmoid function? Is there something the sigmoid function provides mathematically or provides some sort of intuition useful for the next layer?</p>
  
Tags: <neural-networks><activation-functions><hidden-layers><function-approximation>
Title: Why is there a sigmoid function in the hidden layer of a neural network?
UserDisplayName: Gooby
ThreadText: # Why is there a sigmoid function in the hidden layer of a neural network?
  
  (Asked by: username_0 on 2019-10-23)
  
  [![enter image description here](https://i.sstatic.net/hKGIu.png)](https://i.sstatic.net/hKGIu.png)
  
  I got this slide from CMU's lecture notes. The $x\_i$s on the right are inputs and the $w\_i$s are weights that get multiplied together then summed up at each hidden layer node. So I'm assuming this is a node in the hidden layer.
  
  What is the mathematical reason for taking the sum of the weights and inputs and inputting that into a sigmoid function? Is there something the sigmoid function provides mathematically or provides some sort of intuition useful for the next layer?
  
  **Tags:** neural-networks, activation-functions, hidden-layers, function-approximation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-10-28. Score: 5 | Accepted answer
  
  Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e
  layer_output = Weights.layer_input + bias
  
  Consider a 2 layer neural network, the outputs from layer one will be:
      x2 = W1*x1 + b1
  Now we pass the same input to the second layer, which will be
  x3 = W2x*2 + b2
  Also x2 = W1*x1 + b1
  Substituting back, we have:
  x3 = W2(W1*x1 + b1) + b2
  x3 = (W2W1)*x1 + (W2*b1 + b2)
  x3 = W*x1 + b
  
  Oh no! We still got a linear function. No matter how many layers we add, we will still get a linear function. In that case, our network will never be able to approximate any non linear functions.
  So what is the solution?
  We will simply add some non linear functions in between. These functions are called activation functions. Some of these functions include:
  
  ReLU
  Sigmoid
  tanh
  Softmax
  
  and there are a lot more of them.
  Yay! Our network is no more linear!
  We have a lot of different non linear functions, and each of them serve a different purpose.
  For example, ReLU is simple and computationally cheap.
      ReLU(x) = max(0, x)
  Sigmoid outputs are between 0 and 1.
  tanh is similar to sigmoid, but zero centered, with outputs from -1 to 1
  Softmax is usually used if you want to represent any vector as a discrete probability distribution.
  Hope you are having a great day!
  
  
  ---
  
  ### Answer by username_1 on 2019-10-24. Score: 2
  
  By itself, I'm not sure it's possible to know. It's possible the slides were old. Or, the intended purpose was to mention how as sigmoid ranges from 0 to 1. Mostly, it looks like it was intended to bring up gradient descent. But it could also be an entry point to the discussion of other methods such as ReLU. Either that or perhaps some sort of norming function.
  
  
  ---
  
  