Answers.Body: <p>Actually, this already exists!</p>
  
  <p>I happened to make a presentation of a paper that talks about this topic. These networks are called <a href="https://arxiv.org/pdf/1608.06993.pdf" rel="nofollow noreferrer">DenseNets</a>, which stands for densely connected convolutional networks. Just like in your question, within a dense block, the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed-forward neural network the <span class="math-container">$l$</span>th layer is a function of the previous output <span class="math-container">$x_l = H(x_{l-1})$</span>, while in the dense net each layer is a function of all the previous outputs <span class="math-container">$x_l = H([x_0, x_1, \dots, x_{l-1}])$</span>.</p>
  
  <p><a href="https://i.sstatic.net/HVagY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HVagY.png" alt="(imagine)"></a></p>
  
  <p>However, since it is a CNN, there is a reduction in the size of the feature maps with each pooling layer, so to keep the dimensions constant, there is an alternation between the dense block and pooling layers.</p>
  
  <p>The results are clear: not only in almost all the tests the accuracy of the dense net is greater than that of the other methods, but they do so using up to 90% fewer parameters, i.e. they have a high efficiency of parameters. Moreover, as suggested by the authors themselves, the improved accuracy can be explained by the shorter connections between the layers, which allow acting during the training phase in a deep supervision fashion, solving the vanishing gradient problem. This is similar to how it was done in other methods, but with a less complicated gradient.</p>
  
  <p>If you're interested you should definitely check out their paper <a href="https://arxiv.org/pdf/1608.06993.pdf" rel="nofollow noreferrer">Densely Connected Convolutional Networks</a> (2018).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-11T00:55:07.393Z
Answers.Text: @NeilSlater Although this is an old question and residual networks are older than dense nets, I think it's worth adding your answer that talks about residual networks.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-02T11:07:11.130Z
Answers.Text: There is a similar concept called [Resnet](https://www.kaggle.com/keras/resnet50), that uses "skip connections" to achieve a similar result. The difference is that instead of direct deep connections, the skip connections are summed into the existings the next but one layer. Mathematically this creates a "short-circuit" for gradient flow to earlier layers when training, and also makes it very easy for later layers to learn the identity function, which intuitively means they can learn *improvements* over identity function.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-04-02T09:28:51.013Z
Answers.LastActivityDate: 2020-03-11T00:59:32.597Z
Answers.LastEditDate: 2020-03-11T00:59:32.597Z
Answers.UserDisplayName: Armando
Body: <p>Why aren't there neural networks that connect the output of each layer to all next layers?</p>
  
  <p>For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational power considerations, wouldn't this be better than only connecting layers 1 and 2, 3 and 4, etc? </p>
  
  <p>Also, wouldn't this solve the vanishing gradient problem?</p>
  
  <p>If computational power is the concern, perhaps you could connect layer 1 only to the next N layers.</p>
  
Tags: <neural-networks><feedforward-neural-networks><hidden-layers><residual-networks>
Title: Why aren't there neural networks that connect the output of each layer to all next layers?
UserDisplayName: Christopher Jernigan
ThreadText: # Why aren't there neural networks that connect the output of each layer to all next layers?
  
  (Asked by: username_0 on 2018-04-01)
  
  Why aren't there neural networks that connect the output of each layer to all next layers?
  
  For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational power considerations, wouldn't this be better than only connecting layers 1 and 2, 3 and 4, etc?
  
  Also, wouldn't this solve the vanishing gradient problem?
  
  If computational power is the concern, perhaps you could connect layer 1 only to the next N layers.
  
  **Tags:** neural-networks, feedforward-neural-networks, hidden-layers, residual-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-04-02. Score: 5
  
  Actually, this already exists!
  
  I happened to make a presentation of a paper that talks about this topic. These networks are called [DenseNets](https://arxiv.org/pdf/1608.06993.pdf), which stands for densely connected convolutional networks. Just like in your question, within a dense block, the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed-forward neural network the $l$th layer is a function of the previous output $x\_l = H(x\_{l-1})$, while in the dense net each layer is a function of all the previous outputs $x\_l = H([x\_0, x\_1, \dots, x\_{l-1}])$.
  
  [![(imagine)](https://i.sstatic.net/HVagY.png)](https://i.sstatic.net/HVagY.png)
  
  However, since it is a CNN, there is a reduction in the size of the feature maps with each pooling layer, so to keep the dimensions constant, there is an alternation between the dense block and pooling layers.
  
  The results are clear: not only in almost all the tests the accuracy of the dense net is greater than that of the other methods, but they do so using up to 90% fewer parameters, i.e. they have a high efficiency of parameters. Moreover, as suggested by the authors themselves, the improved accuracy can be explained by the shorter connections between the layers, which allow acting during the training phase in a deep supervision fashion, solving the vanishing gradient problem. This is similar to how it was done in other methods, but with a less complicated gradient.
  
  If you're interested you should definitely check out their paper [Densely Connected Convolutional Networks](https://arxiv.org/pdf/1608.06993.pdf) (2018).
  
  > **Comments:**
  > **username_3**: There is a similar concept called [Resnet](https://www.kaggle.com/keras/resnet50), that uses "skip connections" to achieve a similar result. The difference is that instead of direct deep connections, the skip connections are summed into the existings the next but one layer. Mathematically this creates a "short-circuit" for gradient flow to earlier layers when training, and also makes it very easy for later layers to learn the identity function, which intuitively means they can learn *improvements* over identity function.  
  > **username_2**: @NeilSlater Although this is an old question and residual networks are older than dense nets, I think it's worth adding your answer that talks about residual networks.  
  
  ---
  
  