Answers.Body: <p>If you only had linear layers in a neural network, all the layers would essentially collapse to one linear layer, and, therefore, a "deep" neural network architecture effectively wouldn't be deep anymore but just a linear classifier. </p>
  
  <p><span class="math-container">$$y = f(W_1 W_2 W_3x) = f(Wx)$$</span></p>
  
  <p>where <span class="math-container">$W$</span> corresponds to the matrix that represents the network weights and biases for one layer, and <span class="math-container">$f()$</span> to the activation function.</p>
  
  <p>Now, with the introduction of a non-linear activation unit after every linear transformation, this won't happen anymore.</p>
  
  <p><span class="math-container">$$y = f_1( W_1 f_2( W_2f_3( W_3x)))$$</span></p>
  
  <p>Each layer can now build up on the results of the preceding non-linear layer which essentially leads to a complex non-linear function that is able to approximate every possible function with the right weighting and enough depth/width.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-02T15:45:33.167Z
Answers.Text: It should be noted that although a composition of multiple linear operators (on a Euclidean space) _can_ always be collapsed to a single matrix $W$, this doesn't mean keeping instead separate matrices $W_1, W_2...$ never makes sense. In particular, if $W_2$ maps from a high-dimensional space to a low-dimensional one and $W_1$ back to the high-dimensional one, then $W_1(W_2\:x)$ is cheaper to compute than $W(x)$. So, “else it would be equivalent to a single layer” is _not_ really and argument for why nonlinearities in between the layers are needed. What's actually needed is _the nonlinearity_.
Answers.UserDisplayName: leftaroundabout
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-04T11:01:37.483Z
Answers.LastActivityDate: 2019-10-23T22:32:37.820Z
Answers.LastEditDate: 2019-10-23T22:32:37.820Z
Answers.UserDisplayName: Marcel_marcel1991
Answers.Body: <p>Let's first talk about <a href="https://en.wikipedia.org/wiki/Linear_map" rel="nofollow noreferrer">linearity</a>. Linearity means the map (a function), <span class="math-container">$f: V \rightarrow W$</span>, used is a linear map, that is, it satisfies the following two conditions </p>
  
  <ol>
  <li><span class="math-container">$f(x + y) = f(x) + f(y), \; x, y \in V$</span> </li>
  <li><span class="math-container">$f(c x) = cf(x), \; c \in \mathbb{R}$</span></li>
  </ol>
  
  <p>You should be familiar with this definition if you have studied linear algebra in the past.</p>
  
  <p>However, it's more important to think of linearity in terms of linear separability of data, which means the data can be separated into different classes by drawing a line (or hyperplane, if more than two dimensions), which represents a linear decision boundary, through the data. If we cannot do that, then the data is not linearly separable. Often times, data from a more complex (and thus more relevant) problem setting is not linearly separable, so it is in our interest to model these.</p>
  
  <p>To model nonlinear decision boundaries of data, we can utilize a neural network that introduces non-linearity. Neural networks classify data that is not linearly separable by transforming data using some nonlinear function (or our activation function), so the resulting transformed points become linearly separable.</p>
  
  <p>Different activation functions are used for different problem setting contexts. You can read more about that in the book <a href="http://www.deeplearningbook.org" rel="nofollow noreferrer">Deep Learning (Adaptive Computation and Machine Learning series)</a>.</p>
  
  <p>For an example of non linearly separable data, see the XOR data set.</p>
  
  <p><a href="https://i.sstatic.net/JdfWD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JdfWD.png" alt="enter image description here"></a></p>
  
  <p>Can you draw a single line to separate the two classes?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T06:50:17.343Z
Answers.Text: So, without activation functions, outputs of NNs would always be linear, since output from previous layer will be multiplied with weights and added to bias, at each layer. So, in order for a NN to learn or approximate complex functions, different activation functions are being used depending on the purpose. Purpose of an activation function is to introduce non-linearity which those multiplications did. Is my intuition correct?
Answers.UserDisplayName: Naveen Reddy Marthala
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-21T17:34:45.190Z
Answers.Text: Yup that is correct - different activations functions may work better depending on the problem context.
Answers.UserDisplayName: sma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-03T00:18:36.437Z
Answers.LastActivityDate: 2019-10-24T01:54:18.380Z
Answers.LastEditDate: 2019-10-24T01:54:18.380Z
Answers.UserDisplayName: sma
Answers.Body: <p>Consider a very simple neural network, with just 2 layers, where the first has 2 neurons and the last 1 neuron, and the input size is 2. The inputs are <span class="math-container">$x_1$</span> and <span class="math-container">$x_1$</span>.</p>
  
  <p>The weights of the first layer are <span class="math-container">$w_{11}, w_{12}, w_{21}$</span> and <span class="math-container">$w_{22}$</span>. We do not have activations, so the outputs of the neurons in the first layer are</p>
  
  <p><span class="math-container">\begin{align}
  o_1 = w_{11}x_1 + w_{12}x_2 \\
  o_2 = w_{21}x_1 + w_{22}x_2
  \end{align}</span></p>
  
  <p>Let's calculate the output of the last layer with weights <span class="math-container">$z_1$</span> and <span class="math-container">$z_2$</span></p>
  
  <p><span class="math-container">$$out = z_1o_1 + z_2o_2$$</span></p>
  
  <p>Just substitute <span class="math-container">$o_1$</span> and <span class="math-container">$o_2$</span> and you will get:</p>
  
  <p><span class="math-container">$$out = z_1(w_{11}x_1 + w_{12}x_2) + z_2(w_{21}x_1 + w_{22}x_2)$$</span> </p>
  
  <p>or</p>
  
  <p><span class="math-container">$$out = (z_1w_{11} + z_2 w_{21})x_1 + (z_2w_{22} + z_1w_{12})x_2$$</span></p>
  
  <p>And look at this! If we create NN just with one layer with weights <span class="math-container">$z_1w_{11} + z_2 w_{21}$</span> and <span class="math-container">$z_2w_{22} + z_1w_{12}$</span> it will be equivalent to our 2 layers NN.</p>
  
  <p>The conclusion: without nonlinearity, the computational power of a multilayer NN is equal to 1-layer NN.</p>
  
  <p>Also, you can think of the sigmoid function as differentiable IF the statement that gives a probability. And adding new layers can create new, more complex combinations of IF statements. For example, the first layer combines features and gives probabilities that there are eyes, tail, and ears on the picture, the second combines new, more complex features from the last layer and gives probability that there is a cat.</p>
  
  <p>For more information: <a href="http://karpathy.github.io/neuralnets/" rel="nofollow noreferrer">Hacker's guide to Neural Networks</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-03T00:19:05.840Z
Answers.LastActivityDate: 2019-10-23T23:43:51.703Z
Answers.LastEditDate: 2019-10-23T23:43:51.703Z
Answers.UserDisplayName: user2674414
Answers.Body: <p><strong>First Degree Linear Polynomials</strong></p>
  
  <p>Non-linearity is not the correct mathematical term.  Those that use it probably intend to refer to a first degree polynomial relationship between input and output, the kind of relationship that would be graphed as a straight line, a flat plane, or a higher degree surface with no curvature.</p>
  
  <p>To model relations more complex than <em>y = a<sub>1</sub>x<sub>1</sub> + a<sub>2</sub>x<sub>2</sub> + ... + b</em>, more than just those two terms of a Taylor series approximation is needed.</p>
  
  <p><strong>Tune-able Functions with Non-zero Curvature</strong></p>
  
  <p>Artificial networks such as the multi-layer perceptron and its variants are matrices of functions with non-zero curvature that, when taken collectively as a circuit, can be tuned with attenuation grids to approximate more complex functions of non-zero curvature.  These more complex functions generally have multiple inputs (independent variables).</p>
  
  <p>The attenuation grids are simply matrix-vector products, the matrix being the parameters that are tuned to create a circuit that approximates the more complex curved, multivariate function with simpler curved functions.</p>
  
  <p>Oriented with the multi-dimensional signal entering at the left and the result appearing on the right (left-to-right causality), as in the electrical engineering convention, the vertical columns are called layers of activations, mostly for historical reasons.  They are actually arrays of simple curved functions.  The most commonly used activations today are these.</p>
  
  <ul>
  <li>ReLU</li>
  <li>Leaky ReLU</li>
  <li>ELU</li>
  <li>Threshold (binary step)</li>
  <li>Logistic</li>
  </ul>
  
  <p>The identity function is sometimes used to pass through signals untouched for various structural convenience reasons.</p>
  
  <p>These are less used but were in vogue at one point or another.  They are still used but have lost popularity because they place additional overhead on back propagation computations and tend to lose in contests for speed and accuracy.</p>
  
  <ul>
  <li>Softmax</li>
  <li>Sigmoid</li>
  <li>TanH</li>
  <li>ArcTan</li>
  </ul>
  
  <p>The more complex of these can be parametrized and all of them can be perturbed with pseudo-random noise to improve reliability.</p>
  
  <p><strong>Why Bother With All of That?</strong></p>
  
  <p>Artificial networks are not necessary for tuning well developed classes of relationships between input and desired output.  For instance, these are easily optimized using well developed optimization techniques.</p>
  
  <ul>
  <li>Higher degree polynomials &mdash; Often directly solvable using techniques derived directly from linear algebra</li>
  <li>Periodic functions &mdash; Can be treated with Fourier methods</li>
  <li>Curve fitting &mdash; converges well using the Levenberg–Marquardt algorithm, a damped least-squares approach</li>
  </ul>
  
  <p>For these, approaches developed long before the advent of artificial networks can often arrive at an optimal solution with less computational overhead and more precision and reliability.</p>
  
  <p>Where artificial networks excel is in the acquisition of functions about which the practitioner is largely ignorant or the tuning of the parameters of known functions for which specific convergence methods have not yet been devised.</p>
  
  <p>Multi-layer perceptrons (ANNs) tune the parameters (attenuation matrix) during training.  Tuning is directed by gradient descent or one of its variants to produce a digital approximation of an analog circuit that models the unknown functions.  The gradient descent is driven by some criteria toward which circuit behavior is driven by comparing outputs with that criteria.  The criteria can be any of these.</p>
  
  <ul>
  <li>Matching labels (the desired output values corresponding to the training example inputs)</li>
  <li>The need to pass information through narrow signal paths and reconstruct from that limited information</li>
  <li>Another criteria inherent in the network</li>
  <li>Another criteria arising from a signal source from outside the network</li>
  </ul>
  
  <p><strong>In Summary</strong></p>
  
  <p>In summary, activation functions provide the building blocks that can be used repeatedly in two dimensions of the network structure so that, combined with an attenuation matrix to vary the weight of signaling from layer to layer, is known to be able to approximate an arbitrary and complex function.</p>
  
  <p><strong>Deeper Network Excitement</strong></p>
  
  <p>The post-millenial excitement about deeper networks is because the patterns in two distinct classes of complex inputs have been successfully identified and put into use within larger business, consumer, and scientific markets.</p>
  
  <ol>
  <li>Heterogeneous and semantically complex structures</li>
  <li>Media files and streams (images, video, audio)</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-04T03:07:28.227Z
Answers.LastActivityDate: 2018-08-04T09:22:54.237Z
Answers.LastEditDate: 2018-08-04T09:22:54.237Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>It is said that activation functions in neural networks help introduce <strong><em>non-linearity</em></strong>.</p>
  
  <ul>
  <li>What does this mean?</li>
  <li>What does <strong><em>non-linearity</em></strong> mean in this context?</li>
  <li>How does the introduction of this <strong><em>non-linearity</em></strong> help?</li>
  <li>Are there any other purposes of <strong><em>activation functions</em></strong>?</li>
  </ul>
  
Tags: <neural-networks><deep-learning><terminology><activation-functions>
Title: What is the purpose of an activation function in neural networks?
UserDisplayName: Mohsin
ThreadText: # What is the purpose of an activation function in neural networks?
  
  (Asked by: username_0 on 2018-03-02)
  
  It is said that activation functions in neural networks help introduce ***non-linearity***.
  
  * What does this mean?
  * What does ***non-linearity*** mean in this context?
  * How does the introduction of this ***non-linearity*** help?
  * Are there any other purposes of ***activation functions***?
  
  **Tags:** neural-networks, deep-learning, terminology, activation-functions
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2018-03-04. Score: 11
  
  If you only had linear layers in a neural network, all the layers would essentially collapse to one linear layer, and, therefore, a "deep" neural network architecture effectively wouldn't be deep anymore but just a linear classifier.
  
  $$y = f(W\_1 W\_2 W\_3x) = f(Wx)$$
  
  where $W$ corresponds to the matrix that represents the network weights and biases for one layer, and $f()$ to the activation function.
  
  Now, with the introduction of a non-linear activation unit after every linear transformation, this won't happen anymore.
  
  $$y = f\_1( W\_1 f\_2( W\_2f\_3( W\_3x)))$$
  
  Each layer can now build up on the results of the preceding non-linear layer which essentially leads to a complex non-linear function that is able to approximate every possible function with the right weighting and enough depth/width.
  
  > **Comments:**
  > **username_2**: It should be noted that although a composition of multiple linear operators (on a Euclidean space) _can_ always be collapsed to a single matrix $W$, this doesn't mean keeping instead separate matrices $W_1, W_2...$ never makes sense. In particular, if $W_2$ maps from a high-dimensional space to a low-dimensional one and $W_1$ back to the high-dimensional one, then $W_1(W_2\:x)$ is cheaper to compute than $W(x)$. So, “else it would be equivalent to a single layer” is _not_ really and argument for why nonlinearities in between the layers are needed. What's actually needed is _the nonlinearity_.  
  
  ---
  
  ### Answer by username_3 on 2018-03-03. Score: 6
  
  Let's first talk about [linearity](https://en.wikipedia.org/wiki/Linear_map). Linearity means the map (a function), $f: V \rightarrow W$, used is a linear map, that is, it satisfies the following two conditions
  
  1. $f(x + y) = f(x) + f(y), \; x, y \in V$
  2. $f(c x) = cf(x), \; c \in \mathbb{R}$
  
  You should be familiar with this definition if you have studied linear algebra in the past.
  
  However, it's more important to think of linearity in terms of linear separability of data, which means the data can be separated into different classes by drawing a line (or hyperplane, if more than two dimensions), which represents a linear decision boundary, through the data. If we cannot do that, then the data is not linearly separable. Often times, data from a more complex (and thus more relevant) problem setting is not linearly separable, so it is in our interest to model these.
  
  To model nonlinear decision boundaries of data, we can utilize a neural network that introduces non-linearity. Neural networks classify data that is not linearly separable by transforming data using some nonlinear function (or our activation function), so the resulting transformed points become linearly separable.
  
  Different activation functions are used for different problem setting contexts. You can read more about that in the book [Deep Learning (Adaptive Computation and Machine Learning series)](http://www.deeplearningbook.org).
  
  For an example of non linearly separable data, see the XOR data set.
  
  [![enter image description here](https://i.sstatic.net/JdfWD.png)](https://i.sstatic.net/JdfWD.png)
  
  Can you draw a single line to separate the two classes?
  
  > **Comments:**
  > **username_4**: So, without activation functions, outputs of NNs would always be linear, since output from previous layer will be multiplied with weights and added to bias, at each layer. So, in order for a NN to learn or approximate complex functions, different activation functions are being used depending on the purpose. Purpose of an activation function is to introduce non-linearity which those multiplications did. Is my intuition correct?  
  > **username_3**: Yup that is correct - different activations functions may work better depending on the problem context.  
  
  ---
  
  ### Answer by username_5 on 2018-03-03. Score: 5
  
  Consider a very simple neural network, with just 2 layers, where the first has 2 neurons and the last 1 neuron, and the input size is 2. The inputs are $x\_1$ and $x\_1$.
  
  The weights of the first layer are $w\_{11}, w\_{12}, w\_{21}$ and $w\_{22}$. We do not have activations, so the outputs of the neurons in the first layer are
  
  \begin{align}
  o\_1 = w\_{11}x\_1 + w\_{12}x\_2 \\
  o\_2 = w\_{21}x\_1 + w\_{22}x\_2
  \end{align}
  
  Let's calculate the output of the last layer with weights $z\_1$ and $z\_2$
  
  $$out = z\_1o\_1 + z\_2o\_2$$
  
  Just substitute $o\_1$ and $o\_2$ and you will get:
  
  $$out = z\_1(w\_{11}x\_1 + w\_{12}x\_2) + z\_2(w\_{21}x\_1 + w\_{22}x\_2)$$
  
  or
  
  $$out = (z\_1w\_{11} + z\_2 w\_{21})x\_1 + (z\_2w\_{22} + z\_1w\_{12})x\_2$$
  
  And look at this! If we create NN just with one layer with weights $z\_1w\_{11} + z\_2 w\_{21}$ and $z\_2w\_{22} + z\_1w\_{12}$ it will be equivalent to our 2 layers NN.
  
  The conclusion: without nonlinearity, the computational power of a multilayer NN is equal to 1-layer NN.
  
  Also, you can think of the sigmoid function as differentiable IF the statement that gives a probability. And adding new layers can create new, more complex combinations of IF statements. For example, the first layer combines features and gives probabilities that there are eyes, tail, and ears on the picture, the second combines new, more complex features from the last layer and gives probability that there is a cat.
  
  For more information: [Hacker's guide to Neural Networks](http://karpathy.github.io/neuralnets/).
  
  
  ---
  
  ### Answer by username_6 on 2018-08-04. Score: 3
  
  **First Degree Linear Polynomials**
  
  Non-linearity is not the correct mathematical term. Those that use it probably intend to refer to a first degree polynomial relationship between input and output, the kind of relationship that would be graphed as a straight line, a flat plane, or a higher degree surface with no curvature.
  
  To model relations more complex than *y = a1x1 + a2x2 + ... + b*, more than just those two terms of a Taylor series approximation is needed.
  
  **Tune-able Functions with Non-zero Curvature**
  
  Artificial networks such as the multi-layer perceptron and its variants are matrices of functions with non-zero curvature that, when taken collectively as a circuit, can be tuned with attenuation grids to approximate more complex functions of non-zero curvature. These more complex functions generally have multiple inputs (independent variables).
  
  The attenuation grids are simply matrix-vector products, the matrix being the parameters that are tuned to create a circuit that approximates the more complex curved, multivariate function with simpler curved functions.
  
  Oriented with the multi-dimensional signal entering at the left and the result appearing on the right (left-to-right causality), as in the electrical engineering convention, the vertical columns are called layers of activations, mostly for historical reasons. They are actually arrays of simple curved functions. The most commonly used activations today are these.
  
  * ReLU
  * Leaky ReLU
  * ELU
  * Threshold (binary step)
  * Logistic
  
  The identity function is sometimes used to pass through signals untouched for various structural convenience reasons.
  
  These are less used but were in vogue at one point or another. They are still used but have lost popularity because they place additional overhead on back propagation computations and tend to lose in contests for speed and accuracy.
  
  * Softmax
  * Sigmoid
  * TanH
  * ArcTan
  
  The more complex of these can be parametrized and all of them can be perturbed with pseudo-random noise to improve reliability.
  
  **Why Bother With All of That?**
  
  Artificial networks are not necessary for tuning well developed classes of relationships between input and desired output. For instance, these are easily optimized using well developed optimization techniques.
  
  * Higher degree polynomials — Often directly solvable using techniques derived directly from linear algebra
  * Periodic functions — Can be treated with Fourier methods
  * Curve fitting — converges well using the Levenberg–Marquardt algorithm, a damped least-squares approach
  
  For these, approaches developed long before the advent of artificial networks can often arrive at an optimal solution with less computational overhead and more precision and reliability.
  
  Where artificial networks excel is in the acquisition of functions about which the practitioner is largely ignorant or the tuning of the parameters of known functions for which specific convergence methods have not yet been devised.
  
  Multi-layer perceptrons (ANNs) tune the parameters (attenuation matrix) during training. Tuning is directed by gradient descent or one of its variants to produce a digital approximation of an analog circuit that models the unknown functions. The gradient descent is driven by some criteria toward which circuit behavior is driven by comparing outputs with that criteria. The criteria can be any of these.
  
  * Matching labels (the desired output values corresponding to the training example inputs)
  * The need to pass information through narrow signal paths and reconstruct from that limited information
  * Another criteria inherent in the network
  * Another criteria arising from a signal source from outside the network
  
  **In Summary**
  
  In summary, activation functions provide the building blocks that can be used repeatedly in two dimensions of the network structure so that, combined with an attenuation matrix to vary the weight of signaling from layer to layer, is known to be able to approximate an arbitrary and complex function.
  
  **Deeper Network Excitement**
  
  The post-millenial excitement about deeper networks is because the patterns in two distinct classes of complex inputs have been successfully identified and put into use within larger business, consumer, and scientific markets.
  
  1. Heterogeneous and semantically complex structures
  2. Media files and streams (images, video, audio)
  
  
  ---
  
  