Answers.Body: <p>Such a network could be either a <strong>Residual Network</strong> or a <strong>Highway Network</strong> depending upon the underlying architecture of the skip layers. They are primarily used to to tackle the problem of vanishing gradients in very deep networks by reusing activations from a previous layer and passing them to adjacent layers (two or three skips away).</p>
  
  <ul>
  <li><strong>Highway Network</strong>: (<a href="http://papers.nips.cc/paper/5850-training-very-deep-networks" rel="nofollow noreferrer">Training Very Deep Networks, 2015</a>)</li>
  </ul>
  
  <p><span class="math-container">$z$</span> acts as a <em>Gating Function</em>, which controls how much information is to be transmitted from a previous layer in the network. An additional weight matrix is used to estimate the skip weights.<a href="https://i.sstatic.net/zIlct.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zIlct.png" alt="Highway"></a></p>
  
  <ul>
  <li><strong>Residual Network</strong>:<a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">(Deep Residual Learning for Image Recognition ,2015)</a></li>
  </ul>
  
  <p>This architecture does not involve a Gate Controller. The shortcut connections are realized by adding the outputs of a previous layer to that of the connecting layer.</p>
  
  <p><a href="https://i.sstatic.net/FfU4q.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FfU4q.png" alt="Residual"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-04T20:49:57.457Z
Answers.LastActivityDate: 2020-02-04T20:49:57.457Z
Answers.UserDisplayName: s_bh
Answers.Body: <p>This could be called a <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">residual neural network (ResNet)</a>, which is a neural network with skip connections, that is, connections that skip layers. </p>
  
  <p>Here's a screenshot of a figure from the paper <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">Deep Residual Learning for Image Recognition</a> (2015), an important paper that shows the usefulness of these architectures.</p>
  
  <p><a href="https://i.sstatic.net/C5zjY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/C5zjY.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-03T22:06:55.493Z
Answers.LastActivityDate: 2020-02-03T22:06:55.493Z
Answers.UserDisplayName: nbro
Body: <p>Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer.  Is there a name for this architecture? A layer feeds forward around the layer after it?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-03T20:05:47.983Z
Comments.Text: How are you combining the two layers - concatenating the two vectors to make one longer vector size num_imputs + num_hidden?
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><terminology><feedforward-neural-networks><residual-networks>
Title: What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?
UserDisplayName: scorpdaddy
ThreadText: # What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?
  
  (Asked by: username_0 on 2020-02-03)
  
  Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer. Is there a name for this architecture? A layer feeds forward around the layer after it?
  
  **Tags:** machine-learning, terminology, feedforward-neural-networks, residual-networks
  
  > **Comments:**
  > **username_1**: How are you combining the two layers - concatenating the two vectors to make one longer vector size num_imputs + num_hidden?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-02-03. Score: 2 | Accepted answer
  
  This could be called a [residual neural network (ResNet)](https://arxiv.org/pdf/1512.03385.pdf), which is a neural network with skip connections, that is, connections that skip layers.
  
  Here's a screenshot of a figure from the paper [Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385.pdf) (2015), an important paper that shows the usefulness of these architectures.
  
  [![enter image description here](https://i.sstatic.net/C5zjY.png)](https://i.sstatic.net/C5zjY.png)
  
  
  ---
  
  ### Answer by username_2 on 2020-02-04. Score: 2
  
  Such a network could be either a **Residual Network** or a **Highway Network** depending upon the underlying architecture of the skip layers. They are primarily used to to tackle the problem of vanishing gradients in very deep networks by reusing activations from a previous layer and passing them to adjacent layers (two or three skips away).
  
  * **Highway Network**: ([Training Very Deep Networks, 2015](http://papers.nips.cc/paper/5850-training-very-deep-networks))
  
  $z$ acts as a *Gating Function*, which controls how much information is to be transmitted from a previous layer in the network. An additional weight matrix is used to estimate the skip weights.[![Highway](https://i.sstatic.net/zIlct.png)](https://i.sstatic.net/zIlct.png)
  
  * **Residual Network**:[(Deep Residual Learning for Image Recognition ,2015)](https://arxiv.org/pdf/1512.03385.pdf)
  
  This architecture does not involve a Gate Controller. The shortcut connections are realized by adding the outputs of a previous layer to that of the connecting layer.
  
  [![Residual](https://i.sstatic.net/FfU4q.png)](https://i.sstatic.net/FfU4q.png)
  
  
  ---
  
  