Answers.Body: <p>Well, I found an answer that satisfies me.</p>
  
  <p>The zero-padded identity is not ideal. Suppose we're mapping from 64 channels to 128 channels. Then the zero-padded identity will map to an output where half of the channels are the same as the inputs, and the other half are all zeros. So that means the main path is learning a residual for half of the output channels, and learning a mapping from the ground up for the other half of the channels.</p>
  
  <p>Now, the alternative is to use option (B) which is the single 1x1 convolution. After reading their <a href="https://arxiv.org/pdf/1603.05027.pdf" rel="nofollow noreferrer">ResNetV2 paper</a> I realised that they really double down on this concept of maintaining a "clean" alternative path all the way up and down the network. So then the question is, what's "cleaner"?: a block of two convolutions of the form 3x3x64 then 3x3x128, or a shortcut with a 1x1x128 convolution? The shortcut is a tempting choice here. And in fact, in the original paper they show empirically that the convolutional shortcut is better than the identity shortcut. Maybe it might be worth running a test to see if the convolutional shortcut is better than no shortcut at all, but until I decide to run such a test, I'll presume the authors would have mentioned it. The other thing worth noting is that there are only a few places in the whole network where the dimensions need to be increased, so maybe the impact is small.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-20T12:43:10.910Z
Answers.LastActivityDate: 2020-03-20T12:43:10.910Z
Answers.UserDisplayName: Alexander Soare
Body: <p>In the <a href="https://arxiv.org/pdf/1512.03385v1.pdf" rel="nofollow noreferrer">original ResNet paper</a> they talk about using plain identity skip connections when the input and output of a block have the same dimensions.</p>
  
  <p><a href="https://i.sstatic.net/pPvVA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pPvVA.png" alt="enter image description here"></a></p>
  
  <p>When the input and output have different dimensions they propose two options:</p>
  
  <p>(A) Use an identity mapping padded with zeros to make up for the extra dimensions</p>
  
  <p>(B) Use a "projection".</p>
  
  <p><a href="https://i.sstatic.net/59wYe.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/59wYe.png" alt="enter image description here"></a></p>
  
  <p>which (after some digging around in <a href="https://towardsdatascience.com/hitchhikers-guide-to-residual-networks-resnet-in-keras-385ec01ec8ff" rel="nofollow noreferrer">other people's code</a>) I see as meaning: do a convolution with a 1x1 kernel with trainable weights.</p>
  
  <p>(B) is confusing to me because it seems to ruin the point of ResNet by making the skip connection trainable. Then the main path is not really learning a "residual" relative to an identity transformation. So at this point, I'm no longer sure how to interpret the intent or expected effect of this type of block. And I would think that one should justify doing it in the first places instead of just not putting a skip connection there at all (which in my mind is the status-quo before this paper).</p>
  
  <p>So can anyone help explain away my confusion here?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-20T13:59:22.883Z
Comments.Text: If it's not, that will make a big difference to the way I think about this. Maybe I'll go double check by looking at more implementations. They keep calling it a "projection" shortcut in the papers, and to me projection means collapsing one dimension of a vector down to zero, so I'm not sure it relates
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-20T14:03:33.727Z
Comments.Text: Here's another implementation in tensorflow. Line 337 defines the `projection_shortcut`. I'm not too familiar with tf but it looks like a vanilla 1x1 conv to me.
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-20T14:04:01.163Z
Comments.Text: What if someone got it wrong then everyone copied them... :|
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-20T14:10:13.490Z
Comments.Text: I think I'll rewind to before you said anything. I was happy then :P
Comments.UserDisplayName: Alexander Soare
Tags: <convolutional-neural-networks><residual-networks>
Title: If the point of the ResNet skip connection is to let the main path learn the residual relative to identity, why are there convolutional skips?
UserDisplayName: Alexander Soare
ThreadText: # If the point of the ResNet skip connection is to let the main path learn the residual relative to identity, why are there convolutional skips?
  
  (Asked by: username_0 on 2020-03-19)
  
  In the [original ResNet paper](https://arxiv.org/pdf/1512.03385v1.pdf) they talk about using plain identity skip connections when the input and output of a block have the same dimensions.
  
  [![enter image description here](https://i.sstatic.net/pPvVA.png)](https://i.sstatic.net/pPvVA.png)
  
  When the input and output have different dimensions they propose two options:
  
  (A) Use an identity mapping padded with zeros to make up for the extra dimensions
  
  (B) Use a "projection".
  
  [![enter image description here](https://i.sstatic.net/59wYe.png)](https://i.sstatic.net/59wYe.png)
  
  which (after some digging around in [other people's code](https://towardsdatascience.com/hitchhikers-guide-to-residual-networks-resnet-in-keras-385ec01ec8ff)) I see as meaning: do a convolution with a 1x1 kernel with trainable weights.
  
  (B) is confusing to me because it seems to ruin the point of ResNet by making the skip connection trainable. Then the main path is not really learning a "residual" relative to an identity transformation. So at this point, I'm no longer sure how to interpret the intent or expected effect of this type of block. And I would think that one should justify doing it in the first places instead of just not putting a skip connection there at all (which in my mind is the status-quo before this paper).
  
  So can anyone help explain away my confusion here?
  
  **Tags:** convolutional-neural-networks, residual-networks
  
  > **Comments:**
  > **username_0**: If it's not, that will make a big difference to the way I think about this. Maybe I'll go double check by looking at more implementations. They keep calling it a "projection" shortcut in the papers, and to me projection means collapsing one dimension of a vector down to zero, so I'm not sure it relates  
  > **username_0**: Here's another implementation in tensorflow. Line 337 defines the `projection_shortcut`. I'm not too familiar with tf but it looks like a vanilla 1x1 conv to me.  
  > **username_0**: What if someone got it wrong then everyone copied them... :|  
  > **username_0**: I think I'll rewind to before you said anything. I was happy then :P  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-03-20. Score: 3
  
  Well, I found an answer that satisfies me.
  
  The zero-padded identity is not ideal. Suppose we're mapping from 64 channels to 128 channels. Then the zero-padded identity will map to an output where half of the channels are the same as the inputs, and the other half are all zeros. So that means the main path is learning a residual for half of the output channels, and learning a mapping from the ground up for the other half of the channels.
  
  Now, the alternative is to use option (B) which is the single 1x1 convolution. After reading their [ResNetV2 paper](https://arxiv.org/pdf/1603.05027.pdf) I realised that they really double down on this concept of maintaining a "clean" alternative path all the way up and down the network. So then the question is, what's "cleaner"?: a block of two convolutions of the form 3x3x64 then 3x3x128, or a shortcut with a 1x1x128 convolution? The shortcut is a tempting choice here. And in fact, in the original paper they show empirically that the convolutional shortcut is better than the identity shortcut. Maybe it might be worth running a test to see if the convolutional shortcut is better than no shortcut at all, but until I decide to run such a test, I'll presume the authors would have mentioned it. The other thing worth noting is that there are only a few places in the whole network where the dimensions need to be increased, so maybe the impact is small.
  
  
  ---
  
  