Answers.Body: <p>I think that the <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">official paper</a> that introduced residual connections named their network(s) <code>ResNet</code>, so when somebody says <code>ResNet</code> I would assume that they mean an architecture following the original, i.e. convolutional layers with residual connections in between (see <a href="https://pi-tau.github.io/posts/res-nets/#the-residual-block-empowering-deeper-networks" rel="nofollow noreferrer">here</a> if you want more details about the architecture).</p>
  <p>If I want to describe a network architecture with fully-connected layers that uses residual connections I would say exactly this.. fcnn with residual connections.</p>
  <p>Regarding RNNs, I think that the design of the LSTM cell greatly resembles the <a href="https://arxiv.org/pdf/1505.00387.pdf" rel="nofollow noreferrer">highway networks</a>. In fact I think that the highway networks were inspired by the LSTM (also both are authored by the same guy Jürgen Schmidhuber). For sequential tasks I would say that using gated summation makes more sense than the simple summation that residual connections use. However, I think that there are some ideas to apply residual connections between the layers of a multi-layer rnn.</p>
  <p><a href="https://i.sstatic.net/QLAkN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QLAkN.png" alt="multi-layer rnn with residuals" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T19:27:34.130Z
Answers.LastActivityDate: 2023-07-23T19:27:34.130Z
Answers.UserDisplayName: pi-tau
Body: <p>When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks or CNNs? Are there applications of ResNets to RNNs or is that pointless because they are conceptually similar (<a href="https://arxiv.org/abs/1604.03640v1" rel="nofollow noreferrer">https://arxiv.org/abs/1604.03640v1</a>)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-08T16:40:49.120Z
Comments.Text: I think the original implementation was with Deep CNN's, but I think the idea also makes sense for MLPs -- possibly other architectures, but I would need to think more about this.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-11T15:22:55.607Z
Comments.Text: @DavidIreland
  I looked it up and it has indeed been applied for MLPs as well (see e.g. here: https://arxiv.org/abs/2105.03404). Then probably CNNs and MLPs are the most reasonable application areas. Have you had the chance to think about it more or do you agree?
Comments.UserDisplayName: postnubilaphoebus
Tags: <convolutional-neural-networks><recurrent-neural-networks><residual-networks>
Title: What underlying network is typically meant with ResNET?
UserDisplayName: postnubilaphoebus
ThreadText: # What underlying network is typically meant with ResNET?
  
  (Asked by: username_0 on 2022-12-08)
  
  When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks or CNNs? Are there applications of ResNets to RNNs or is that pointless because they are conceptually similar (<https://arxiv.org/abs/1604.03640v1>)?
  
  **Tags:** convolutional-neural-networks, recurrent-neural-networks, residual-networks
  
  > **Comments:**
  > **username_1**: I think the original implementation was with Deep CNN's, but I think the idea also makes sense for MLPs -- possibly other architectures, but I would need to think more about this.  
  > **username_0**: @DavidIreland I looked it up and it has indeed been applied for MLPs as well (see e.g. here: https://arxiv.org/abs/2105.03404). Then probably CNNs and MLPs are the most reasonable application areas. Have you had the chance to think about it more or do you agree?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-07-23. Score: 0
  
  I think that the [official paper](https://arxiv.org/pdf/1512.03385.pdf) that introduced residual connections named their network(s) `ResNet`, so when somebody says `ResNet` I would assume that they mean an architecture following the original, i.e. convolutional layers with residual connections in between (see [here](https://username_2.github.io/posts/res-nets/#the-residual-block-empowering-deeper-networks) if you want more details about the architecture).
  
  If I want to describe a network architecture with fully-connected layers that uses residual connections I would say exactly this.. fcnn with residual connections.
  
  Regarding RNNs, I think that the design of the LSTM cell greatly resembles the [highway networks](https://arxiv.org/pdf/1505.00387.pdf). In fact I think that the highway networks were inspired by the LSTM (also both are authored by the same guy Jürgen Schmidhuber). For sequential tasks I would say that using gated summation makes more sense than the simple summation that residual connections use. However, I think that there are some ideas to apply residual connections between the layers of a multi-layer rnn.
  
  [![multi-layer rnn with residuals](https://i.sstatic.net/QLAkN.png)](https://i.sstatic.net/QLAkN.png)
  
  
  ---
  
  