Answers.Body: <p>This type of connections are called skip or residual connections. There are numerous works which employs this type of mechanism, for example: <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">ResNet</a>, <a href="https://openreview.net/pdf?id=HkwVAXyCW" rel="nofollow noreferrer">SkipRNN</a>. In addition <a href="https://www.aclweb.org/anthology/C16-1020.pdf" rel="nofollow noreferrer">here</a> you can find a paper that empirically explores the skip connections for sequential tagging, or this <a href="https://link.springer.com/article/10.1007/s11265-020-01518-1" rel="nofollow noreferrer">one</a> for speech enhancement.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-29T09:03:06.773Z
Answers.LastActivityDate: 2020-10-29T09:03:06.773Z
Answers.UserDisplayName: razvanc92
Body: <p>Are there any architectures of deep neural networks that connect input neurons not only with the first hidden layer but also with deeper ones (red lines on the picture)?</p>
  <p><a href="https://i.sstatic.net/DHjHh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DHjHh.png" alt="image" /></a></p>
  <p>If so could you give some names or links to research papers?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-29T09:17:03.273Z
Comments.Text: This seems to be a duplicate of https://ai.stackexchange.com/q/17190/2444. Well, at least, I answer your question there too.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-29T09:41:56.957Z
Comments.Text: @nbro Well, I don't think this question is a duplicate, but yes there is an answer to my question in the linked one. Thanks!
Comments.UserDisplayName: GKozinski
Tags: <neural-networks><deep-learning><deep-neural-networks><architecture><hidden-layers>
Title: Are there deep neural networks that have inputs connected with deeper hidden layers?
UserDisplayName: GKozinski
ThreadText: # Are there deep neural networks that have inputs connected with deeper hidden layers?
  
  (Asked by: username_0 on 2020-10-29)
  
  Are there any architectures of deep neural networks that connect input neurons not only with the first hidden layer but also with deeper ones (red lines on the picture)?
  
  [![image](https://i.sstatic.net/DHjHh.png)](https://i.sstatic.net/DHjHh.png)
  
  If so could you give some names or links to research papers?
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks, architecture, hidden-layers
  
  > **Comments:**
  > **username_1**: This seems to be a duplicate of https://ai.stackexchange.com/q/17190/2444. Well, at least, I answer your question there too.  
  > **username_0**: @username_1 Well, I don't think this question is a duplicate, but yes there is an answer to my question in the linked one. Thanks!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-10-29. Score: 2 | Accepted answer
  
  This type of connections are called skip or residual connections. There are numerous works which employs this type of mechanism, for example: [ResNet](https://arxiv.org/pdf/1512.03385.pdf), [SkipRNN](https://openreview.net/pdf?id=HkwVAXyCW). In addition [here](https://www.aclweb.org/anthology/C16-1020.pdf) you can find a paper that empirically explores the skip connections for sequential tagging, or this [one](https://link.springer.com/article/10.1007/s11265-020-01518-1) for speech enhancement.
  
  
  ---
  
  