Answers.Body: <p>Almost never. The sum of linear functions is another linear function, so if neurons were only linear transformations there would be basically no point to having more than one neuron per layer. Instead, every neuron applies some kind of <em>nonlinear</em> function to its input. There are lots of different variations, but in the end the combination of the nonlinear activation function at each layer with the linear matrix multiplication connecting the outputs of each layer to the inputs of the next, creates something that has much more intricate behavior while still being reasonably efficient to compute.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-13T12:53:49.963Z
Answers.Text: I know that the sum of a set of linear functions is linear...  Perhaps I mislead you with my question.  I was simply asking about the connection between how kernel and bias values are applied to produce weights;  Obviously, we apply non-linear functions to the data during transformations.  I appreciate your thoughts thought.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-13T04:07:24.240Z
Answers.LastActivityDate: 2019-10-13T04:07:24.240Z
Answers.UserDisplayName: hobbs
Answers.Body: <p>Taking the question from <a href="https://ai.stackexchange.com/questions/15877/do-neurons-of-a-neural-network-model-a-linear-relationship/15886#comment24003_15879">comments</a> on nbro's answer.</p>
  
  <hr>
  
  <blockquote>
    <p>Am I wrong to see a clear relationship between how we are currently training networks and the classic function that defines a line? </p>
  </blockquote>
  
  <p>You are right about it. This is an intuitive way to understand neural networks. You can create a neural network that only does simple linear regression, by using linear activations functions in all the layers, such as the neural network (model) output is a linear combination of the inputs. And, this seems like a great way to introduce neural networks to students. </p>
  
  <p>But, one must also look at the fact that neural networks provide the flexibility to model many kinds of non-linear relationships.</p>
  
  <hr>
  
  <p>A <a href="https://stats.stackexchange.com/questions/115258/comprehensive-list-of-activation-functions-in-neural-networks-with-pros-cons/">list of activation functions</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-12T21:17:19.920Z
Answers.LastActivityDate: 2019-10-15T07:08:00.847Z
Answers.LastEditDate: 2019-10-15T07:08:00.847Z
Answers.UserDisplayName: naive
Answers.Body: <p>You're quite right in your interpretation, but I'll answer in two parts in order to avoid confusion with respect to activation functions.</p>
  
  <p>Part 1. (TLDR: a neurons weights is the normal vector of a hyperplane that divides input space in two parts. The neuron's preactivation is proportional to the distance of the input point to the plane.) Every artificial neuron learns a linear relationship between its inputs. The most recalled equation of a line is <span class="math-container">$y=m \cdot x+b$</span>, but that's actually a very specific form that allows us going through values of X of the line and seeing to what values of Y it corresponds. A most general form would be <span class="math-container">$0=n \cdot y + m \cdot x + b$</span>. This tells us that the line is formed by the points (X,Y) whose values make that series of calculations be zero. We can explore different values of (X,Y) and see that most of them give non-zero values, and that they give positive values at one side of the line and negative values at the other side. Only if you land just on the line it will give you a zero. This is a very important interpretation, because it's what allows neurons to find divisions of the input space (into a positive side and a negative side). Of course it probably won't be a 2d space, so it will be a hyperplane instead of a line, but I hope you get the idea.</p>
  
  <p>Part 2. However, if we only use linear transformations we couldn't learn non-linear functions. Here's where the activation function plays a very important role: it distorts the neuron's preactivation value (which is linear) in a non-linear way (what makes it a non-linear function). Activation functions have lots of bells and whistles, which are too much to write here, but you can start thinking about them as distortions applied to that distance of the input point to the neuron's hyperplane. The one you saw is called ReLU, and it basically truncates the negative values, thus focusing only on the positive side of the hyperplane (it may be interpreted as measuring how far the point has crossed a frontier).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-15T18:07:29.330Z
Answers.LastActivityDate: 2019-10-15T18:07:29.330Z
Answers.UserDisplayName: David
Answers.Body: <p>In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron <span class="math-container">$i$</span> in layer <span class="math-container">$l$</span> of an NN is </p>
  
  <p><span class="math-container">$$o_i^l = \sigma(\mathbf{x}_i^l \cdot  \mathbf{w}_i^l + b_i^l),$$</span> </p>
  
  <p>where <span class="math-container">$\sigma$</span> is a so-called <a href="https://en.wikipedia.org/wiki/Activation_function" rel="noreferrer"><em>activation function</em></a>, which is usually a non-linearity, but it can also be the <a href="https://en.wikipedia.org/wiki/Identity_function" rel="noreferrer">identity function</a>, <span class="math-container">$\mathbf{x}_i^l$</span> and <span class="math-container">$\mathbf{w}_i^l$</span> are the vectors that respectively contain the inputs and the weights for neuron <span class="math-container">$i$</span> in layer <span class="math-container">$l$</span>, and <span class="math-container">$b_i^l \in \mathbb{R}$</span> is a <a href="https://en.wikipedia.org/wiki/Inductive_bias" rel="noreferrer">bias</a>. Similarly, the output of a layer of a feed-forward neural network (FFNN) is computed as</p>
  
  <p><span class="math-container">$$\mathbf{o}^l = \sigma(\mathbf{X}^l \mathbf{W}^l + \mathbf{b}^l).$$</span></p>
  
  <p>In your specific example, you set the new weight to <span class="math-container">$0$</span>, if the output of the linear combination is less than <span class="math-container">$0$</span>, else you use the output of the linear combination. This is the definition of the <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="noreferrer">ReLU activation function</a>, which is a non-linear function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-12T16:27:16.520Z
Answers.Text: Yes, I understand that... And I realize I was using ReLU rather than sigmoid...  but am I wrong to see a clear relationship between how we are currently training networks and the classic function that defines a line?
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-12T16:28:13.487Z
Answers.Text: Oh, and a follow-up... From my reading over the last week, I had the impression that sigmoid was out of favor in current thought.  Is that not true?  I realize it was the standard just a few years ago.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-12T16:28:35.753Z
Answers.Text: @DavidHoelzer In the equation of a line, $\sigma$ is always the identity function, while in the case of NNs, $\sigma$ is almost never the identity function. This is the main difference.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-12T16:31:08.717Z
Answers.Text: @DavidHoelzer It seems to me that ReLU (or variations) is used more often (than sigmoids), but I think you can find several discussions on the web related to this topic.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-13T13:56:39.960Z
Answers.Text: @DuttaA Yes, but I think the equations have just been transcribed incorrectly. Maybe we can ask for clarifications regarding the passage of the book where these equations are shown.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-14T14:32:07.720Z
Answers.Text: The code does not appear as written, and I really don't know how to properly mark up a dot product in the SO Markdown..  I'm paraphrasing the discussion and function representation of the "What are activation functions" in the box on page 72.  Also, I suspect Mr. Chollet would not be pleased to find that someone has posted his book publicly with DRM removed.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-12T16:23:44.157Z
Answers.LastActivityDate: 2019-10-12T16:34:19.923Z
Answers.LastEditDate: 2019-10-12T16:34:19.923Z
Answers.UserDisplayName: nbro
Body: <p>I'm certain that this is a very naive question, but I am just beginning to look more deeply at neural networks, having only used decision tree approaches in the past.  Also, my formal mathematics training is more than 30 years in the past, so please be kind. :)</p>
  
  <p>As I'm reading François Chollet's book on <a href="https://rads.stackoverflow.com/amzn/click/com/1617294438" rel="nofollow noreferrer" rel="nofollow noreferrer">Deep Learning with Python</a>, I'm struck that it appears that we are effectively treating the weights (kernel and biases) as terms in the standard linear equation (<span class="math-container">$y=mx+b$</span>). At page 72 of the book, the author writes</p>
  
  <pre><code>output = dot(W, input) + b
  output = (output &lt; 0 ? 0 : output)
  </code></pre>
  
  <p>Am I reading too much into this, or is this correct (and so fundamental I shouldn't be asking about it)?</p>
  
Tags: <neural-networks><activation-functions><neurons>
Title: Do neurons of a neural network model a linear relationship?
UserDisplayName: David Hoelzer
ThreadText: # Do neurons of a neural network model a linear relationship?
  
  (Asked by: username_0 on 2019-10-12)
  
  I'm certain that this is a very username_2 question, but I am just beginning to look more deeply at neural networks, having only used decision tree approaches in the past. Also, my formal mathematics training is more than 30 years in the past, so please be kind. :)
  
  As I'm reading François Chollet's book on [Deep Learning with Python](https://rads.stackoverflow.com/amzn/click/com/1617294438), I'm struck that it appears that we are effectively treating the weights (kernel and biases) as terms in the standard linear equation ($y=mx+b$). At page 72 of the book, the author writes
  
  ```
  output = dot(W, input) + b
  output = (output < 0 ? 0 : output)
  
  ```
  
  Am I reading too much into this, or is this correct (and so fundamental I shouldn't be asking about it)?
  
  **Tags:** neural-networks, activation-functions, neurons
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2019-10-12. Score: 10 | Accepted answer
  
  In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron $i$ in layer $l$ of an NN is
  
  $$o\_i^l = \sigma(\mathbf{x}\_i^l \cdot \mathbf{w}\_i^l + b\_i^l),$$
  
  where $\sigma$ is a so-called [*activation function*](https://en.wikipedia.org/wiki/Activation_function), which is usually a non-linearity, but it can also be the [identity function](https://en.wikipedia.org/wiki/Identity_function), $\mathbf{x}\_i^l$ and $\mathbf{w}\_i^l$ are the vectors that respectively contain the inputs and the weights for neuron $i$ in layer $l$, and $b\_i^l \in \mathbb{R}$ is a [bias](https://en.wikipedia.org/wiki/Inductive_bias). Similarly, the output of a layer of a feed-forward neural network (FFNN) is computed as
  
  $$\mathbf{o}^l = \sigma(\mathbf{X}^l \mathbf{W}^l + \mathbf{b}^l).$$
  
  In your specific example, you set the new weight to $0$, if the output of the linear combination is less than $0$, else you use the output of the linear combination. This is the definition of the [ReLU activation function](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)), which is a non-linear function.
  
  > **Comments:**
  > **username_0**: Yes, I understand that... And I realize I was using ReLU rather than sigmoid...  but am I wrong to see a clear relationship between how we are currently training networks and the classic function that defines a line?  
  > **username_0**: Oh, and a follow-up... From my reading over the last week, I had the impression that sigmoid was out of favor in current thought.  Is that not true?  I realize it was the standard just a few years ago.  
  > **username_4**: @username_0 In the equation of a line, $\sigma$ is always the identity function, while in the case of NNs, $\sigma$ is almost never the identity function. This is the main difference.  
  > **username_4**: @username_0 It seems to me that ReLU (or variations) is used more often (than sigmoids), but I think you can find several discussions on the web related to this topic.  
  > **username_4**: @DuttaA Yes, but I think the equations have just been transcribed incorrectly. Maybe we can ask for clarifications regarding the passage of the book where these equations are shown.  
  > **username_0**: The code does not appear as written, and I really don't know how to properly mark up a dot product in the SO Markdown..  I'm paraphrasing the discussion and function representation of the "What are activation functions" in the box on page 72.  Also, I suspect Mr. Chollet would not be pleased to find that someone has posted his book publicly with DRM removed.  
  
  ---
  
  ### Answer by username_1 on 2019-10-13. Score: 5
  
  Almost never. The sum of linear functions is another linear function, so if neurons were only linear transformations there would be basically no point to having more than one neuron per layer. Instead, every neuron applies some kind of *nonlinear* function to its input. There are lots of different variations, but in the end the combination of the nonlinear activation function at each layer with the linear matrix multiplication connecting the outputs of each layer to the inputs of the next, creates something that has much more intricate behavior while still being reasonably efficient to compute.
  
  > **Comments:**
  > **username_0**: I know that the sum of a set of linear functions is linear...  Perhaps I mislead you with my question.  I was simply asking about the connection between how kernel and bias values are applied to produce weights;  Obviously, we apply non-linear functions to the data during transformations.  I appreciate your thoughts thought.  
  
  ---
  
  ### Answer by username_2 on 2019-10-12. Score: 3
  
  Taking the question from [comments](https://ai.stackexchange.com/questions/15877/do-neurons-of-a-neural-network-model-a-linear-relationship/15886#comment24003_15879) on username_4's answer.
  
  ---
  
  > Am I wrong to see a clear relationship between how we are currently training networks and the classic function that defines a line?
  
  You are right about it. This is an intuitive way to understand neural networks. You can create a neural network that only does simple linear regression, by using linear activations functions in all the layers, such as the neural network (model) output is a linear combination of the inputs. And, this seems like a great way to introduce neural networks to students.
  
  But, one must also look at the fact that neural networks provide the flexibility to model many kinds of non-linear relationships.
  
  ---
  
  A [list of activation functions](https://stats.stackexchange.com/questions/115258/comprehensive-list-of-activation-functions-in-neural-networks-with-pros-cons/).
  
  
  ---
  
  ### Answer by username_3 on 2019-10-15. Score: 1
  
  You're quite right in your interpretation, but I'll answer in two parts in order to avoid confusion with respect to activation functions.
  
  Part 1. (TLDR: a neurons weights is the normal vector of a hyperplane that divides input space in two parts. The neuron's preactivation is proportional to the distance of the input point to the plane.) Every artificial neuron learns a linear relationship between its inputs. The most recalled equation of a line is $y=m \cdot x+b$, but that's actually a very specific form that allows us going through values of X of the line and seeing to what values of Y it corresponds. A most general form would be $0=n \cdot y + m \cdot x + b$. This tells us that the line is formed by the points (X,Y) whose values make that series of calculations be zero. We can explore different values of (X,Y) and see that most of them give non-zero values, and that they give positive values at one side of the line and negative values at the other side. Only if you land just on the line it will give you a zero. This is a very important interpretation, because it's what allows neurons to find divisions of the input space (into a positive side and a negative side). Of course it probably won't be a 2d space, so it will be a hyperplane instead of a line, but I hope you get the idea.
  
  Part 2. However, if we only use linear transformations we couldn't learn non-linear functions. Here's where the activation function plays a very important role: it distorts the neuron's preactivation value (which is linear) in a non-linear way (what makes it a non-linear function). Activation functions have lots of bells and whistles, which are too much to write here, but you can start thinking about them as distortions applied to that distance of the input point to the neuron's hyperplane. The one you saw is called ReLU, and it basically truncates the negative values, thus focusing only on the positive side of the hyperplane (it may be interpreted as measuring how far the point has crossed a frontier).
  
  
  ---
  
  