Answers.Body: <p>I think your confusion comes from the fact that you are calling those two hidden nodes "perceptrons". You shouldn't call the hidden nodes in your network perceptrons. You should call them "nodes" or "neurons", although the term "multilayer perceptron" comes exactly from that.</p>
  
  <p>You should think of a perceptron as something like</p>
  
  <p><a href="https://i.sstatic.net/DWFga.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DWFga.png" alt="enter image description here"></a></p>
  
  <p>A perceptron simply computes a linear combination of the inputs using the corresponding weights. A single perceptron can (only) <strong>linearly</strong> separate a set of points. </p>
  
  <p>However, if you have two "stacked perceptrons" (like the hidden nodes in your network), then, provided that both "perceptrons" (or hidden nodes) <em>only compute a linear combination of the inputs</em>, the output of each hidden node can be interpreted as line (like the violet and green lines in your picture).</p>
  
  <blockquote>
    <p>I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron?</p>
  </blockquote>
  
  <p>This is because the two perceptrons will eventually have different weights. You can see from your diagram that <span class="math-container">$p_0$</span> has weights <span class="math-container">$w_{0, 0}$</span> and <span class="math-container">$w_{0, 1}$</span>, whereas perceptron <span class="math-container">$p_1$</span> has weights <span class="math-container">$w_{1, 1}$</span> and <span class="math-container">$w_{1, 0}$</span>. Note that weights are learned, during training, depending on your loss function (at the output node <span class="math-container">$p_2$</span>) and the labeled dataset (that you use to train that simple neural network).</p>
  
  <blockquote>
    <p>I know the weights can be initialized differently but does this second perceptron classify something else?</p>
  </blockquote>
  
  <p>If you consider the two hidden nodes (in your network) separately, then (as I said above) the output of these two hidden nodes can be interpreted as a line, and these lines might be different (as illustrated in your picture), because these two hidden nodes will eventually learn something different about the input. However, you should also think of the output of two hidden nodes as contributing to the input of the output node. So, in a certain way, you're combining the decisions of the two hidden nodes (or, as you call them, "perceptrons").</p>
  
  <p>Note that you will need to introduce "non-linearity" in those hidden perceptrons in order to be able to non-linearly separate a set of points.</p>
  
  <blockquote>
    <p>Shouldn't the decision boundaries both converge to the same ultimately after training?</p>
  </blockquote>
  
  <p>No, not necessarily. The decision boundary found by the perceptron learning algorithm depends on the initialization of the weights and the order that the inputs are presented. See chapter 4 (specifically, pages 192-196) of <a href="http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf" rel="nofollow noreferrer">Pattern Recognition and Machine Learning</a> by C. Bishop. Two perceptrons would be guaranteed to converge to the same solution if the loss function was convex (i.e. there's only one optimum).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-03T21:43:50.520Z
Answers.LastActivityDate: 2020-04-08T14:48:34.510Z
Answers.LastEditDate: 2020-04-08T14:48:34.510Z
Answers.UserDisplayName: nbro
Body: <p>I've learned that you can use two perceptrons to ultimately create a classifier for non-linearly separable data. I'm trying to understand how / if  these two perceptrons converge to two different decision boundaries though.</p>
  
  <p><a href="https://i.sstatic.net/z9iMR.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/z9iMR.png" alt="enter image description here"></a></p>
  
  <p>Source: <a href="https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html" rel="nofollow noreferrer">https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html</a></p>
  
  <p>I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron? I know the weights can be initialized differently but does this second perceptron classify something else? Shouldn't the decision boundaries be converge to be the the same ultimately after training?</p>
  
  <p><a href="https://i.sstatic.net/gqiw6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gqiw6.png" alt="enter image description here"></a></p>
  
  <p>Source: <a href="https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html" rel="nofollow noreferrer">https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html</a></p>
  
Tags: <neural-networks><activation-functions><perceptron>
Title: How do two perceptrons produce different linear decision boundaries when learning?
UserDisplayName: ashar
ThreadText: # How do two perceptrons produce different linear decision boundaries when learning?
  
  (Asked by: username_0 on 2019-03-03)
  
  I've learned that you can use two perceptrons to ultimately create a classifier for non-linearly separable data. I'm trying to understand how / if these two perceptrons converge to two different decision boundaries though.
  
  [![enter image description here](https://i.sstatic.net/z9iMR.png)](https://i.sstatic.net/z9iMR.png)
  
  Source: <https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html>
  
  I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron? I know the weights can be initialized differently but does this second perceptron classify something else? Shouldn't the decision boundaries be converge to be the the same ultimately after training?
  
  [![enter image description here](https://i.sstatic.net/gqiw6.png)](https://i.sstatic.net/gqiw6.png)
  
  Source: <https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html>
  
  **Tags:** neural-networks, activation-functions, perceptron
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-03. Score: 3
  
  I think your confusion comes from the fact that you are calling those two hidden nodes "perceptrons". You shouldn't call the hidden nodes in your network perceptrons. You should call them "nodes" or "neurons", although the term "multilayer perceptron" comes exactly from that.
  
  You should think of a perceptron as something like
  
  [![enter image description here](https://i.sstatic.net/DWFga.png)](https://i.sstatic.net/DWFga.png)
  
  A perceptron simply computes a linear combination of the inputs using the corresponding weights. A single perceptron can (only) **linearly** separate a set of points.
  
  However, if you have two "stacked perceptrons" (like the hidden nodes in your network), then, provided that both "perceptrons" (or hidden nodes) *only compute a linear combination of the inputs*, the output of each hidden node can be interpreted as line (like the violet and green lines in your picture).
  
  > I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron?
  
  This is because the two perceptrons will eventually have different weights. You can see from your diagram that $p\_0$ has weights $w\_{0, 0}$ and $w\_{0, 1}$, whereas perceptron $p\_1$ has weights $w\_{1, 1}$ and $w\_{1, 0}$. Note that weights are learned, during training, depending on your loss function (at the output node $p\_2$) and the labeled dataset (that you use to train that simple neural network).
  
  > I know the weights can be initialized differently but does this second perceptron classify something else?
  
  If you consider the two hidden nodes (in your network) separately, then (as I said above) the output of these two hidden nodes can be interpreted as a line, and these lines might be different (as illustrated in your picture), because these two hidden nodes will eventually learn something different about the input. However, you should also think of the output of two hidden nodes as contributing to the input of the output node. So, in a certain way, you're combining the decisions of the two hidden nodes (or, as you call them, "perceptrons").
  
  Note that you will need to introduce "non-linearity" in those hidden perceptrons in order to be able to non-linearly separate a set of points.
  
  > Shouldn't the decision boundaries both converge to the same ultimately after training?
  
  No, not necessarily. The decision boundary found by the perceptron learning algorithm depends on the initialization of the weights and the order that the inputs are presented. See chapter 4 (specifically, pages 192-196) of [Pattern Recognition and Machine Learning](http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf) by C. Bishop. Two perceptrons would be guaranteed to converge to the same solution if the loss function was convex (i.e. there's only one optimum).
  
  
  ---
  
  