Answers.Body: <p>There's a few reasons I can think of, though I have not read an explicit description of why it is done this way. It's likely that people just started doing it this way because it's most logical, and people who have attempted to try your method of having reduced connections have seen a performance hit and so no change was made.</p>
  <p>The first reason is that if you allow all nodes from one layer to connect to all others in the next, the network will optimise unnecessary connections out. Essentially, the weighting of these connections will become 0. This, however, does not mean you can trim these connections, as ignoring them in this local minima might be optimal, but later it might be really important these connections remain. As such, you can never truly know if a connection between one layer and the next is necessary, so it's just better to leave it in case it helps improve network performance.</p>
  <p>The second reason is it's just simpler mathematically. Networks are implemented specifically so it's very easy to apply a series of matrix calculations to perform all computations. Trimming connections means either:</p>
  <ul>
  <li>A matrix must contain 0 values, wasting computation time</li>
  <li>A custom script must be written to calculate this networks structure, which in the real world can take a very long time as it must be implemented using something like CUDA (on a GPU level, making it very complicated)</li>
  </ul>
  <p>Overall, it's just a lot simpler to have all nodes connected between layers, rather than on connection per node.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T02:23:13.043Z
Answers.LastActivityDate: 2020-07-30T02:23:13.043Z
Answers.UserDisplayName: Recessive
Answers.Body: <p><strong>It doesn't.</strong></p>
  <p>Whether or not this is useful is another story, but it is totally fine to do that neural net you have with just one input value. Perhaps you choose one pixel of the photo and make your classification based on the intensity in that one pixel (I guess I'm assuming a black-and-white photo), or you have some method to condense an entire photograph into one value that summarizes the photo. Then each neuron in the hidden layer only has one input connection.</p>
  <p>Likewise, you are allowed to decide that the top neuron in the hidden layer should have only one input connection; just drop the other two.</p>
  <p>Again, this might not give useful results, but they're still neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T16:19:18.293Z
Answers.Text: so in short, was it made so that it could be better represented mathematically? So as a unified model? Isn't there some document or something that takes up my question and maybe even explains a meaning?
Answers.UserDisplayName: iwab
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T16:22:49.003Z
Answers.Text: I don't follow how you get from my answer to "better represented mathematically" and "a unified model". Please explain.
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T16:26:32.207Z
Answers.Text: ahh, i read your answer a bit wrong at first, but you actually say that it would also work with one input, but i would still like to know why someone came up with this architecture.
Answers.UserDisplayName: iwab
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T16:27:50.410Z
Answers.Text: Which architecture?
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T16:29:05.603Z
Answers.Text: the architecture that each neuron in one layer is connected to each in the next.
Answers.UserDisplayName: iwab
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T16:29:57.423Z
Answers.Text: It seems like you're the one who came up with it.
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T16:31:25.830Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/111188/discussion-between-dave-and-iwab).
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T15:06:25.903Z
Answers.LastActivityDate: 2020-07-29T15:06:25.903Z
Answers.UserDisplayName: Dave
Answers.Body: <p>If you adopt a slightly different point-of-view, then a neural network of this static kind is just a big function with parameters, <span class="math-container">$y=F(x,P)$</span>, and the task of training the network is a non-linear fit of this function to the data set.</p>
  <p>That is, training the network is to reduce all of the residuals <span class="math-container">$y_k-F(x_k,P)$</span> simultaneously. This is a balancing act, just tuning one weight to adjust one residual will in general worsen some other residuals. Even if that is taken into account, methods that adjust one variable at a time are usually much slower than methods that adjust all variables simultaneously along some gradient or Newton direction.</p>
  <p>The usual back-propagation algorithm sequentializes the gradient descent method for the square sum of the residuals. Better variants improve that to a Newton-like method by some estimate of the Hessean of this square sum or following along the idea of the Gauß-Newton method.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T05:03:45.110Z
Answers.Text: But I donts see, why just tuning one weight worsen some other residuals? If I have a layer with 3 neurons, I just tune 3 weights (not 9) and ending with the same results. Why should it take more time? It would just look like that here: https://i.imgur.com/HvhaBrV.jpg
Answers.UserDisplayName: iwab
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T22:35:51.733Z
Answers.LastActivityDate: 2020-07-29T22:35:51.733Z
Answers.UserDisplayName: Lutz Lehmann
Body: <h2>For example, if I have the following architecture:</h2>
  <p><a href="https://i.sstatic.net/I1SWw.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/I1SWw.gif" alt="3 layer neural network" /></a></p>
  <ul>
  <li>Each neuron in the hidden layer has a connection from each one in the
  input layer.</li>
  <li>3 x 1 Input Matrix and a 4 x 3 weight matrix (for the backpropagation we have of course the transformed version 3 x 4)</li>
  </ul>
  <p>But until now, I still don't understand what the point is that a neuron has 3 inputs (in the hidden layer of the example). It would work the same way, if I would only adjust one weight of the 3 connections.</p>
  <p>But in the current case the information flows only distributed over several &quot;channels&quot;, but what is the point?</p>
  <p>With backpropagation, in some cases the weights are simply adjusted proportionally based on the error.</p>
  <p>Or is it just done that way, because then you can better mathematically implement everything (with matrix multiplication and so on)?</p>
  <p>Either my question is stupid or I have an error in my thinking and assume wrong ideas. Can someone please help me with the interpretation.</p>
  <p>In tensorflow playground for example, I cut the connections (by setting the weight to 0), it just compansated it by changing the other still existing connection a bit more:
  <a href="https://i.sstatic.net/lNHq4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lNHq4.png" alt="TensorflowImage" /></a></p>
  
Tags: <neural-networks><backpropagation><architecture><dense-layers>
Title: Why does a neuron in a multi-layer network need several input connections?
UserDisplayName: iwab
ThreadText: # Why does a neuron in a multi-layer network need several input connections?
  
  (Asked by: username_0 on 2020-07-29)
  
  ## For example, if I have the following architecture:
  
  [![3 layer neural network](https://i.sstatic.net/I1SWw.gif)](https://i.sstatic.net/I1SWw.gif)
  
  * Each neuron in the hidden layer has a connection from each one in the
    input layer.
  * 3 x 1 Input Matrix and a 4 x 3 weight matrix (for the backpropagation we have of course the transformed version 3 x 4)
  
  But until now, I still don't understand what the point is that a neuron has 3 inputs (in the hidden layer of the example). It would work the same way, if I would only adjust one weight of the 3 connections.
  
  But in the current case the information flows only distributed over several "channels", but what is the point?
  
  With backpropagation, in some cases the weights are simply adjusted proportionally based on the error.
  
  Or is it just done that way, because then you can better mathematically implement everything (with matrix multiplication and so on)?
  
  Either my question is stupid or I have an error in my thinking and assume wrong ideas. Can someone please help me with the interpretation.
  
  In tensorflow playground for example, I cut the connections (by setting the weight to 0), it just compansated it by changing the other still existing connection a bit more:
  [![TensorflowImage](https://i.sstatic.net/lNHq4.png)](https://i.sstatic.net/lNHq4.png)
  
  **Tags:** neural-networks, backpropagation, architecture, dense-layers
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-07-30. Score: 2
  
  There's a few reasons I can think of, though I have not read an explicit description of why it is done this way. It's likely that people just started doing it this way because it's most logical, and people who have attempted to try your method of having reduced connections have seen a performance hit and so no change was made.
  
  The first reason is that if you allow all nodes from one layer to connect to all others in the next, the network will optimise unnecessary connections out. Essentially, the weighting of these connections will become 0. This, however, does not mean you can trim these connections, as ignoring them in this local minima might be optimal, but later it might be really important these connections remain. As such, you can never truly know if a connection between one layer and the next is necessary, so it's just better to leave it in case it helps improve network performance.
  
  The second reason is it's just simpler mathematically. Networks are implemented specifically so it's very easy to apply a series of matrix calculations to perform all computations. Trimming connections means either:
  
  * A matrix must contain 0 values, wasting computation time
  * A custom script must be written to calculate this networks structure, which in the real world can take a very long time as it must be implemented using something like CUDA (on a GPU level, making it very complicated)
  
  Overall, it's just a lot simpler to have all nodes connected between layers, rather than on connection per node.
  
  
  ---
  
  ### Answer by username_2 on 2020-07-29. Score: 0
  
  **It doesn't.**
  
  Whether or not this is useful is another story, but it is totally fine to do that neural net you have with just one input value. Perhaps you choose one pixel of the photo and make your classification based on the intensity in that one pixel (I guess I'm assuming a black-and-white photo), or you have some method to condense an entire photograph into one value that summarizes the photo. Then each neuron in the hidden layer only has one input connection.
  
  Likewise, you are allowed to decide that the top neuron in the hidden layer should have only one input connection; just drop the other two.
  
  Again, this might not give useful results, but they're still neural networks.
  
  > **Comments:**
  > **username_0**: so in short, was it made so that it could be better represented mathematically? So as a unified model? Isn't there some document or something that takes up my question and maybe even explains a meaning?  
  > **username_2**: I don't follow how you get from my answer to "better represented mathematically" and "a unified model". Please explain.  
  > **username_0**: ahh, i read your answer a bit wrong at first, but you actually say that it would also work with one input, but i would still like to know why someone came up with this architecture.  
  > **username_2**: Which architecture?  
  > **username_0**: the architecture that each neuron in one layer is connected to each in the next.  
  > **username_2**: It seems like you're the one who came up with it.  
  > **username_2**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/111188/discussion-between-username_2-and-username_0).  
  
  ---
  
  ### Answer by username_3 on 2020-07-29. Score: 0
  
  If you adopt a slightly different point-of-view, then a neural network of this static kind is just a big function with parameters, $y=F(x,P)$, and the task of training the network is a non-linear fit of this function to the data set.
  
  That is, training the network is to reduce all of the residuals $y\_k-F(x\_k,P)$ simultaneously. This is a balancing act, just tuning one weight to adjust one residual will in general worsen some other residuals. Even if that is taken into account, methods that adjust one variable at a time are usually much slower than methods that adjust all variables simultaneously along some gradient or Newton direction.
  
  The usual back-propagation algorithm sequentializes the gradient descent method for the square sum of the residuals. Better variants improve that to a Newton-like method by some estimate of the Hessean of this square sum or following along the idea of the Gauß-Newton method.
  
  > **Comments:**
  > **username_0**: But I donts see, why just tuning one weight worsen some other residuals? If I have a layer with 3 neurons, I just tune 3 weights (not 9) and ending with the same results. Why should it take more time? It would just look like that here: https://i.imgur.com/HvhaBrV.jpg  
  
  ---
  
  