<p>I think that the <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">official paper</a> that introduced residual connections named their network(s) <code>ResNet</code>, so when somebody says <code>ResNet</code> I would assume that they mean an architecture following the original, i.e. convolutional layers with residual connections in between (see <a href="https://pi-tau.github.io/posts/res-nets/#the-residual-block-empowering-deeper-networks" rel="nofollow noreferrer">here</a> if you want more details about the architecture).</p>
<p>If I want to describe a network architecture with fully-connected layers that uses residual connections I would say exactly this.. fcnn with residual connections.</p>
<p>Regarding RNNs, I think that the design of the LSTM cell greatly resembles the <a href="https://arxiv.org/pdf/1505.00387.pdf" rel="nofollow noreferrer">highway networks</a>. In fact I think that the highway networks were inspired by the LSTM (also both are authored by the same guy Jürgen Schmidhuber). For sequential tasks I would say that using gated summation makes more sense than the simple summation that residual connections use. However, I think that there are some ideas to apply residual connections between the layers of a multi-layer rnn.</p>
<p><a href="https://i.sstatic.net/QLAkN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QLAkN.png" alt="multi-layer rnn with residuals" /></a></p>

CC BY-SA 4.0
2023-07-23T19:27:34.130Z
2023-07-23T19:27:34.130Z
pi-tau
<p>When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks or CNNs? Are there applications of ResNets to RNNs or is that pointless because they are conceptually similar (<a href="https://arxiv.org/abs/1604.03640v1" rel="nofollow noreferrer">https://arxiv.org/abs/1604.03640v1</a>)?</p>

CC BY-SA 4.0
2022-12-08T16:40:49.120Z
I think the original implementation was with Deep CNN's, but I think the idea also makes sense for MLPs -- possibly other architectures, but I would need to think more about this.
David
CC BY-SA 4.0
2022-12-11T15:22:55.607Z
@DavidIreland
I looked it up and it has indeed been applied for MLPs as well (see e.g. here: https://arxiv.org/abs/2105.03404). Then probably CNNs and MLPs are the most reasonable application areas. Have you had the chance to think about it more or do you agree?
postnubilaphoebus
<convolutional-neural-networks><recurrent-neural-networks><residual-networks>
What underlying network is typically meant with ResNET?
postnubilaphoebus
# What underlying network is typically meant with ResNET?

(Asked by: username_0 on 2022-12-08)

When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks or CNNs? Are there applications of ResNets to RNNs or is that pointless because they are conceptually similar (<https://arxiv.org/abs/1604.03640v1>)?

**Tags:** convolutional-neural-networks, recurrent-neural-networks, residual-networks

> **Comments:**
> **username_1**: I think the original implementation was with Deep CNN's, but I think the idea also makes sense for MLPs -- possibly other architectures, but I would need to think more about this.  
> **username_0**: @username_1Ireland I looked it up and it has indeed been applied for MLPs as well (see e.g. here: https://arxiv.org/abs/2105.03404). Then probably CNNs and MLPs are the most reasonable application areas. Have you had the chance to think about it more or do you agree?  

---

## 1 answers

### Answer by username_2 on 2023-07-23. Score: 0

I think that the [official paper](https://arxiv.org/pdf/1512.03385.pdf) that introduced residual connections named their network(s) `ResNet`, so when somebody says `ResNet` I would assume that they mean an architecture following the original, i.e. convolutional layers with residual connections in between (see [here](https://username_2.github.io/posts/res-nets/#the-residual-block-empowering-deeper-networks) if you want more details about the architecture).

If I want to describe a network architecture with fully-connected layers that uses residual connections I would say exactly this.. fcnn with residual connections.

Regarding RNNs, I think that the design of the LSTM cell greatly resembles the [highway networks](https://arxiv.org/pdf/1505.00387.pdf). In fact I think that the highway networks were inspired by the LSTM (also both are authored by the same guy Jürgen Schmidhuber). For sequential tasks I would say that using gated summation makes more sense than the simple summation that residual connections use. However, I think that there are some ideas to apply residual connections between the layers of a multi-layer rnn.

[![multi-layer rnn with residuals](https://i.sstatic.net/QLAkN.png)](https://i.sstatic.net/QLAkN.png)


---

