Answers.Body: <p>When using residual connections you want to have your input dimensions match the output dimension so that you can perform the addition operation. In a standard ResNet-style architecture you mostly have layers that keep the dimensions (i.e. they use <code>padding=&quot;same&quot;</code>).</p>
  <p>However, there are a few places where you change the dimensions. Usually, your convolutional layer will reduce the spatial dimensions in half and will double the number of channels, i.e. <span class="math-container">$C \times H \times W \rightarrow 2C \times H//2 \times W//2$</span>. Such a layer can be initialized like this for example:</p>
  <pre><code>nn.Conv2d(in_chan, 2*in_chan, kernel_size=3, stride=2, padding=1)
  </code></pre>
  <p>In this case your skip connection will not be a simple identity function, but you will actually apply a <code>1x1</code> convolution instead. Thus, instead of having <span class="math-container">$F(x) = f(x) + x$</span>, you will have <span class="math-container">$F(x) = f(x) + g(x)$</span>, where g(x) is the <code>1x1</code> convolution. This additional convolutional layer will apply the needed modification of the dimensions of <span class="math-container">$x$</span> so that you can apply the addition operation. For images and more explanations please see: <a href="https://pi-tau.github.io/posts/res-nets/#the-architecture-of-the-resnet" rel="nofollow noreferrer">https://pi-tau.github.io/posts/res-nets/#the-architecture-of-the-resnet</a></p>
  <p>Now, before using this architecture for nlp tasks, I think it is a good idea to know <strong>WHY</strong> we do this dimension reduction. You usually start with a tensor of some size <span class="math-container">$C \times H \times W$</span>. Then you apply some ResBlocks that <strong>do not</strong> modify the dimensions and extract features from this fixed dimensional space. Then you apply <strong>one</strong> special block that modifies the dimensions and again you stack ResBlocks that extract features from this reduced dimensional space. The idea is to squash the spatial dimensions and to expand the channel dimensions so that information that is contained across multiple nearby pixels in the original image is, at the end, concentrated in a single &quot;pixel&quot; which is now embedded in a much larger space.
  (see the link above for even more info)</p>
  <p>Now with NLP sequences we are talking about 1D convolutions, so replicating this behavior would mean gradually reducing your sequence length in order to increase the embedding size. But I think that this design might fail to capture long term relationships which are quite common in text-related tasks. Also if you don't reduce the sequence length and only increase the embedding size, then your computational cost is quite higher in the deeper layers, and you really want near-constant cost across the layers. If you take a look at <a href="https://pi-tau.github.io/posts/transformer/" rel="nofollow noreferrer">the transformer</a>, for example, there they use self-attention to capture long-distance relations, and they also keep the embedding size constant throughout making every encoder block equally expensive.</p>
  <p>Best of luck with your problem :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T19:06:09.347Z
Answers.LastActivityDate: 2023-07-23T19:06:09.347Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>You can just use <code>padding='same'</code>. As noted from the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D" rel="nofollow noreferrer">documentation</a>:</p>
  <blockquote>
  <p>When <code>padding=&quot;same&quot;</code> and <code>strides=1</code>, the output has the same size as the input.</p>
  </blockquote>
  <p>Note that <code>strides</code> is default to 1, and if <code>kernel_size=1</code>, the output also has the same shape as the input.</p>
  <p>I look at two different implementations and can confirm this:</p>
  <ul>
  <li>The implementation of <a href="https://d2l.ai/chapter_convolutional-modern/resnet.html#residual-blocks" rel="nofollow noreferrer">Dive into Deep Learning</a> shows that the Residual block implementation is:</li>
  </ul>
  <pre><code>class Residual(tf.keras.Model):  #@save
      &quot;&quot;&quot;The Residual block of ResNet.&quot;&quot;&quot;
      def __init__(self, num_channels, use_1x1conv=False, strides=1):
          super().__init__()
          self.conv1 = tf.keras.layers.Conv2D(num_channels, padding='same',
                                              kernel_size=3, strides=strides)
          self.conv2 = tf.keras.layers.Conv2D(num_channels, kernel_size=3,
                                              padding='same')
          self.conv3 = None
          if use_1x1conv:
              self.conv3 = tf.keras.layers.Conv2D(num_channels, kernel_size=1,
                                                  strides=strides)
          self.bn1 = tf.keras.layers.BatchNormalization()
          self.bn2 = tf.keras.layers.BatchNormalization()
  
      def call(self, X):
          Y = tf.keras.activations.relu(self.bn1(self.conv1(X)))
          Y = self.bn2(self.conv2(Y))
          if self.conv3 is not None:
              X = self.conv3(X)
          Y += X
          return tf.keras.activations.relu(Y)
  </code></pre>
  <p>which we see <code>conv1</code> and <code>conv2</code> has <code>padding='same', strides=1</code> everywhere.</p>
  <ul>
  <li>The second implementation is from <a href="https://github.com/keras-team/keras/blob/e6784e4302c7b8cd116b74a784f4b78d60e83c26/keras/applications/resnet.py#L279" rel="nofollow noreferrer">Keras official code</a>, which also uses <code>padding='SAME'</code> here.</li>
  </ul>
  <p>Here's the <a href="https://stackoverflow.com/a/62282172/11235205">visualization</a> of how different padding works. In short, <code>'same'</code> automatically calculates the padding dimension based on the kernel size so that the output has the same shape as the input for you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T05:35:50.873Z
Answers.LastActivityDate: 2023-01-11T05:35:50.873Z
Answers.UserDisplayName: Minh-Long Luu
Body: <p>For highway network, it looks like this:</p>
  <p><a href="https://i.sstatic.net/zN1eN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zN1eN.png" alt="enter image description here" /></a></p>
  <p>For residual network, it looks like this:</p>
  <p><a href="https://i.sstatic.net/91aM7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/91aM7.png" alt="enter image description here" /></a></p>
  <p>Pictures are from <a href="https://ai.stackexchange.com/questions/17822/what-is-the-name-of-this-neural-network-architecture-with-layers-that-are-also-c/17845#17845">What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?</a></p>
  <p>My question is, how to handle the size difference between different layers in CNN to make highway network or residual network?
  For example, I am working on a text classification problem. By using the embedding, I have the input size as follows:</p>
  <pre><code>input.shape =[batch_size, embedding_dim, max_length]
  </code></pre>
  <p>I also has a CNN layer as follows:</p>
  <pre><code>Conv1d(in_channels= embedding_dim, out_channels=hidden_dim, kernel_size=n)
  </code></pre>
  <p>So that the size of the output of <code>Conv1d</code> is <code>[batch_size, hidden_dim, max_length-n+1]</code>.</p>
  <p>Here is the question, the input size of the CNN layer is different from the output size. How do handle the size difference so that highway network or residual network can be built?</p>
  <p>Thank you.</p>
  
Tags: <convolutional-neural-networks><deep-neural-networks><residual-networks>
Title: How to handle the size difference of highway network or residual network in cnn?
UserDisplayName: liaoming999
ThreadText: # How to handle the size difference of highway network or residual network in cnn?
  
  (Asked by: username_0 on 2023-01-07)
  
  For highway network, it looks like this:
  
  [![enter image description here](https://i.sstatic.net/zN1eN.png)](https://i.sstatic.net/zN1eN.png)
  
  For residual network, it looks like this:
  
  [![enter image description here](https://i.sstatic.net/91aM7.png)](https://i.sstatic.net/91aM7.png)
  
  Pictures are from [What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?](https://ai.stackexchange.com/questions/17822/what-is-the-name-of-this-neural-network-architecture-with-layers-that-are-also-c/17845#17845)
  
  My question is, how to handle the size difference between different layers in CNN to make highway network or residual network?
  For example, I am working on a text classification problem. By using the embedding, I have the input size as follows:
  
  ```
  input.shape =[batch_size, embedding_dim, max_length]
  
  ```
  
  I also has a CNN layer as follows:
  
  ```
  Conv1d(in_channels= embedding_dim, out_channels=hidden_dim, kernel_size=n)
  
  ```
  
  So that the size of the output of `Conv1d` is `[batch_size, hidden_dim, max_length-n+1]`.
  
  Here is the question, the input size of the CNN layer is different from the output size. How do handle the size difference so that highway network or residual network can be built?
  
  Thank you.
  
  **Tags:** convolutional-neural-networks, deep-neural-networks, residual-networks
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-01-11. Score: 2 | Accepted answer
  
  You can just use `padding='same'`. As noted from the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D):
  
  > When `padding="same"` and `strides=1`, the output has the same size as the input.
  
  Note that `strides` is default to 1, and if `kernel_size=1`, the output also has the same shape as the input.
  
  I look at two different implementations and can confirm this:
  
  * The implementation of [Dive into Deep Learning](https://d2l.ai/chapter_convolutional-modern/resnet.html#residual-blocks) shows that the Residual block implementation is:
  
  ```
  class Residual(tf.keras.Model):  #@save
      """The Residual block of ResNet."""
      def __init__(self, num_channels, use_1x1conv=False, strides=1):
          super().__init__()
          self.conv1 = tf.keras.layers.Conv2D(num_channels, padding='same',
                                              kernel_size=3, strides=strides)
          self.conv2 = tf.keras.layers.Conv2D(num_channels, kernel_size=3,
                                              padding='same')
          self.conv3 = None
          if use_1x1conv:
              self.conv3 = tf.keras.layers.Conv2D(num_channels, kernel_size=1,
                                                  strides=strides)
          self.bn1 = tf.keras.layers.BatchNormalization()
          self.bn2 = tf.keras.layers.BatchNormalization()
  
      def call(self, X):
          Y = tf.keras.activations.relu(self.bn1(self.conv1(X)))
          Y = self.bn2(self.conv2(Y))
          if self.conv3 is not None:
              X = self.conv3(X)
          Y += X
          return tf.keras.activations.relu(Y)
  
  ```
  
  which we see `conv1` and `conv2` has `padding='same', strides=1` everywhere.
  
  * The second implementation is from [Keras official code](https://github.com/keras-team/keras/blob/e6784e4302c7b8cd116b74a784f4b78d60e83c26/keras/applications/resnet.py#L279), which also uses `padding='SAME'` here.
  
  Here's the [visualization](https://stackoverflow.com/a/62282172/11235205) of how different padding works. In short, `'same'` automatically calculates the padding dimension based on the kernel size so that the output has the same shape as the input for you.
  
  
  ---
  
  ### Answer by username_1 on 2023-07-23. Score: 0
  
  When using residual connections you want to have your input dimensions match the output dimension so that you can perform the addition operation. In a standard ResNet-style architecture you mostly have layers that keep the dimensions (i.e. they use `padding="same"`).
  
  However, there are a few places where you change the dimensions. Usually, your convolutional layer will reduce the spatial dimensions in half and will double the number of channels, i.e. $C \times H \times W \rightarrow 2C \times H//2 \times W//2$. Such a layer can be initialized like this for example:
  
  ```
  nn.Conv2d(in_chan, 2*in_chan, kernel_size=3, stride=2, padding=1)
  
  ```
  
  In this case your skip connection will not be a simple identity function, but you will actually apply a `1x1` convolution instead. Thus, instead of having $F(x) = f(x) + x$, you will have $F(x) = f(x) + g(x)$, where g(x) is the `1x1` convolution. This additional convolutional layer will apply the needed modification of the dimensions of $x$ so that you can apply the addition operation. For images and more explanations please see: <https://username_1.github.io/posts/res-nets/#the-architecture-of-the-resnet>
  
  Now, before using this architecture for nlp tasks, I think it is a good idea to know **WHY** we do this dimension reduction. You usually start with a tensor of some size $C \times H \times W$. Then you apply some ResBlocks that **do not** modify the dimensions and extract features from this fixed dimensional space. Then you apply **one** special block that modifies the dimensions and again you stack ResBlocks that extract features from this reduced dimensional space. The idea is to squash the spatial dimensions and to expand the channel dimensions so that information that is contained across multiple nearby pixels in the original image is, at the end, concentrated in a single "pixel" which is now embedded in a much larger space.
  (see the link above for even more info)
  
  Now with NLP sequences we are talking about 1D convolutions, so replicating this behavior would mean gradually reducing your sequence length in order to increase the embedding size. But I think that this design might fail to capture long term relationships which are quite common in text-related tasks. Also if you don't reduce the sequence length and only increase the embedding size, then your computational cost is quite higher in the deeper layers, and you really want near-constant cost across the layers. If you take a look at [the transformer](https://username_1.github.io/posts/transformer/), for example, there they use self-attention to capture long-distance relations, and they also keep the embedding size constant throughout making every encoder block equally expensive.
  
  Best of luck with your problem :)
  
  
  ---
  
  