Answers.Body: <blockquote>
  <p>Of course, it does create more parameters to train, but that seems like a small sacrifice to me.</p>
  </blockquote>
  <p>It is not a &quot;small sacrifice&quot;. For the very deep networks that skip connections are applied to, to get the same benefits when concatenating, you would end up witha significant multiplier on the number of parameters.</p>
  <p>To get the same passthrough effect on gradient signals (and allow later layers to learn modifications to the identity function), each layer's output would need to be copied to all the following layers. This scales poorly.</p>
  <p>Let's take an example of a 10-layer fully-connected network, with 100 neurons per layer in the hidden layers where we want to apply skip connections. In the simple version of this network (ignoring bias to keep the maths simpler), there are 100x100=10,000 parameters for each added layer, making 90,000 parameters overall. If you use addition-based skip connections, the total number of parameters remains the same, at 90,000. If you use concatenation, the layers connect as 100x100, (100+100)x100, (100+100+100)x100 etc, so you end up with 450,000 parameters, five times as many. This is not a &quot;small sacrifice&quot;, this is a scaling problem.</p>
  <p>The technique of concatenating layers into later parts of the network is known and used. As is adding &quot;early output&quot; or head layers to generate gradient signals at different points in the network. These are valid approaches, and can help with vanishing gradient problems in a similar way. However, the additive skip connections in a residual network scale much better into very deep networks.</p>
  <p>Concatenated copies of layers, or additional network heads with the same target functions and loss, are still used, but more sparingly.</p>
  <blockquote>
  <p>This avoids the strange choice of addition, which has a questionable benefit</p>
  </blockquote>
  <p>It may avoid a specific addition mechanism that you seem concerned about. However there are still very similar additions occurring when the concatenated layer feeds forward to the next layer. You could set those weights in a specific way (a copy of the weights of the new layer that it is concatenating with for just two combined layers) when the layer sizes are the same, and the concatenation and addition approaches would very similar.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T08:45:31.263Z
Answers.LastActivityDate: 2021-08-16T10:06:42.657Z
Answers.LastEditDate: 2021-08-16T10:06:42.657Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>There is no obvious reason why concatenating layers will &quot;stabilize&quot; the training of the network. In fact, you are adding more information, which might improve accuracy, but perhaps at certain computational disadvantages.</p>
  <p>ResNets prevent the vanishing gradient problem by &quot;passing information&quot; from previous layers onto the next. Page 6 of <a href="https://www.cs.toronto.edu/%7Ergrosse/courses/csc321_2017/slides/lec16.pdf" rel="nofollow noreferrer">these slides</a> shows calculations that show this to be the case in a specific degenerate edge cases; but as for how it consistently provides such an advantage on more general basis is from what I understand still unclear.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-10T15:46:33.323Z
Answers.Text: tl;dr: read page 6 of those slides
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-10T15:46:14.717Z
Answers.LastActivityDate: 2021-09-10T15:46:14.717Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.Body: <p>One can concatenate with the previous layer outputs as well, and this approach in pursued in <strong>DenseNets</strong>. A nice illustration, that compares difference between ResNets and DenseNets is presented below:
  <a href="https://i.sstatic.net/6Gene.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6Gene.png" alt="enter image description here" /></a></p>
  <p>As pointed in the other answer it will lead to an increase of computation cost, with the same number of channels (given all other properties of architecture are the same).</p>
  <p>Suppose, you had ResNet with the fixed channel size <span class="math-container">$N$</span>. Standard convolution has computational and storage cost proportional to the product of input and output channels or <span class="math-container">$O(N^2)$</span> in the present case.</p>
  <p>If you concatenated features from the previous layer, after each layer number of channels would be doubled. Therefore, the computational cost would grow <span class="math-container">$4$</span> times for each new layer, and the total cost grows exponentially with depth in this approach.</p>
  <p>However, you can make each of the convolution to shrink the number of channels <span class="math-container">$2$</span> times and concatenate only half of the channels from the previous layer. In this way, total computational cost and storage cost is the same in every layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T21:26:23.460Z
Answers.Text: I don't see how it would grow exponentially... You're correct that it could grow if you used the output the concat operation directly in the next skip layer connection. But the idea is that you would instead use the output of the next layer as skip connection input. That being said I see that this growing structure is what Dense net is trying to implement, that is interesting...
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T21:31:45.300Z
Answers.Text: But even if you allowed it to grow the number of parameters would grow linearly, and then plugging into O(N^2) would give you quadratic growth right?
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-17T01:07:49.153Z
Answers.Text: @profPlum it depends on how you craft the layers. Typically, one would double the number of feature maps each next layer, which leads to exponential growth.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T11:57:00.677Z
Answers.LastActivityDate: 2021-08-16T11:57:00.677Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>I know how <em>skip connections</em> work: you add the activations of the previous layer to the activations of a successive layer to stabilize information/gradient flow.</p>
  <p>My question is, why doesn't it just get implemented in the seemingly more sensible way of <strong>concatenating</strong> some previous layer's activations onto a later layer's activations?</p>
  <p>Most regularization methods are implemented somewhat transparently (to avoid possible negative consequences, e.g. BatchNorm having learnable parameters to disable it). While this method instead interferes with regular functioning of the network rather than simply making itself available in case it is useful.</p>
  <p>What is the reasoning behind the choice to do this rather than simply using concatenation?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T20:19:13.237Z
Comments.Text: It's a good question, but why does addition have a "questionable benefit"? If you can explain why you think that, it may help with an answer. As it stands, it is not useful in the question, because it seems to just be an opinion, so cannot be answered.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T20:35:37.950Z
Comments.Text: Because most regularization methods are implemented somewhat transparently (to avoid possible negative consequences, e.g. BatchNorm having learnable parameters to disable it). While this method instead interferes with regular functioning of the network rather than simply making itself available in case it is useful.
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T20:37:29.287Z
Comments.Text: Thanks for the response. It would be helpful to [edit] that into the question.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-14T00:35:37.493Z
Comments.Text: Is your question why adding the output of layers vs concatenating or why the same shape? Either way, the spatial dimensions (in CNN) must have the same shape. As far as I know, skip connections utilize concatenation, whereas ResNet addition (long vs short skip connections)
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-14T10:49:34.530Z
Comments.Text: It seems to me that there are 2 distinct questions in this post (the one in the title and the other in the body of the post). Please, edit your post to leave just one question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-16T21:11:48.673Z
Comments.Text: @nbro well one property is a consequence of the other. My point is that I don't want someone to answer explaining that the reason is because of addition. I want someone to explain the root cause which requires explaining the addition which as far as I know is an arbitrary design choice.
Comments.UserDisplayName: profPlum
Tags: <residual-networks>
Title: Why do skip layer connections require the same layer sizes?
UserDisplayName: profPlum
ThreadText: # Why do skip layer connections require the same layer sizes?
  
  (Asked by: username_0 on 2021-08-13)
  
  I know how *skip connections* work: you add the activations of the previous layer to the activations of a successive layer to stabilize information/gradient flow.
  
  My question is, why doesn't it just get implemented in the seemingly more sensible way of **concatenating** some previous layer's activations onto a later layer's activations?
  
  Most regularization methods are implemented somewhat transparently (to avoid possible negative consequences, e.g. BatchNorm having learnable parameters to disable it). While this method instead interferes with regular functioning of the network rather than simply making itself available in case it is useful.
  
  What is the reasoning behind the choice to do this rather than simply using concatenation?
  
  **Tags:** residual-networks
  
  > **Comments:**
  > **username_1**: It's a good question, but why does addition have a "questionable benefit"? If you can explain why you think that, it may help with an answer. As it stands, it is not useful in the question, because it seems to just be an opinion, so cannot be answered.  
  > **username_0**: Because most regularization methods are implemented somewhat transparently (to avoid possible negative consequences, e.g. BatchNorm having learnable parameters to disable it). While this method instead interferes with regular functioning of the network rather than simply making itself available in case it is useful.  
  > **username_1**: Thanks for the response. It would be helpful to [edit] that into the question.  
  > **username_2**: Is your question why adding the output of layers vs concatenating or why the same shape? Either way, the spatial dimensions (in CNN) must have the same shape. As far as I know, skip connections utilize concatenation, whereas ResNet addition (long vs short skip connections)  
  > **username_3**: It seems to me that there are 2 distinct questions in this post (the one in the title and the other in the body of the post). Please, edit your post to leave just one question.  
  > **username_0**: @username_3 well one property is a consequence of the other. My point is that I don't want someone to answer explaining that the reason is because of addition. I want someone to explain the root cause which requires explaining the addition which as far as I know is an arbitrary design choice.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2021-08-16. Score: 2 | Accepted answer
  
  One can concatenate with the previous layer outputs as well, and this approach in pursued in **DenseNets**. A nice illustration, that compares difference between ResNets and DenseNets is presented below:
  [![enter image description here](https://i.sstatic.net/6Gene.png)](https://i.sstatic.net/6Gene.png)
  
  As pointed in the other answer it will lead to an increase of computation cost, with the same number of channels (given all other properties of architecture are the same).
  
  Suppose, you had ResNet with the fixed channel size $N$. Standard convolution has computational and storage cost proportional to the product of input and output channels or $O(N^2)$ in the present case.
  
  If you concatenated features from the previous layer, after each layer number of channels would be doubled. Therefore, the computational cost would grow $4$ times for each new layer, and the total cost grows exponentially with depth in this approach.
  
  However, you can make each of the convolution to shrink the number of channels $2$ times and concatenate only half of the channels from the previous layer. In this way, total computational cost and storage cost is the same in every layer.
  
  > **Comments:**
  > **username_0**: I don't see how it would grow exponentially... You're correct that it could grow if you used the output the concat operation directly in the next skip layer connection. But the idea is that you would instead use the output of the next layer as skip connection input. That being said I see that this growing structure is what Dense net is trying to implement, that is interesting...  
  > **username_0**: But even if you allowed it to grow the number of parameters would grow linearly, and then plugging into O(N^2) would give you quadratic growth right?  
  > **username_2**: @username_0 it depends on how you craft the layers. Typically, one would double the number of feature maps each next layer, which leads to exponential growth.  
  
  ---
  
  ### Answer by username_1 on 2021-08-16. Score: 0
  
  > Of course, it does create more parameters to train, but that seems like a small sacrifice to me.
  
  It is not a "small sacrifice". For the very deep networks that skip connections are applied to, to get the same benefits when concatenating, you would end up witha significant multiplier on the number of parameters.
  
  To get the same passthrough effect on gradient signals (and allow later layers to learn modifications to the identity function), each layer's output would need to be copied to all the following layers. This scales poorly.
  
  Let's take an example of a 10-layer fully-connected network, with 100 neurons per layer in the hidden layers where we want to apply skip connections. In the simple version of this network (ignoring bias to keep the maths simpler), there are 100x100=10,000 parameters for each added layer, making 90,000 parameters overall. If you use addition-based skip connections, the total number of parameters remains the same, at 90,000. If you use concatenation, the layers connect as 100x100, (100+100)x100, (100+100+100)x100 etc, so you end up with 450,000 parameters, five times as many. This is not a "small sacrifice", this is a scaling problem.
  
  The technique of concatenating layers into later parts of the network is known and used. As is adding "early output" or head layers to generate gradient signals at different points in the network. These are valid approaches, and can help with vanishing gradient problems in a similar way. However, the additive skip connections in a residual network scale much better into very deep networks.
  
  Concatenated copies of layers, or additional network heads with the same target functions and loss, are still used, but more sparingly.
  
  > This avoids the strange choice of addition, which has a questionable benefit
  
  It may avoid a specific addition mechanism that you seem concerned about. However there are still very similar additions occurring when the concatenated layer feeds forward to the next layer. You could set those weights in a specific way (a copy of the weights of the new layer that it is concatenating with for just two combined layers) when the layer sizes are the same, and the concatenation and addition approaches would very similar.
  
  
  ---
  
  ### Answer by username_4 on 2021-09-10. Score: 0
  
  There is no obvious reason why concatenating layers will "stabilize" the training of the network. In fact, you are adding more information, which might improve accuracy, but perhaps at certain computational disadvantages.
  
  ResNets prevent the vanishing gradient problem by "passing information" from previous layers onto the next. Page 6 of [these slides](https://www.cs.toronto.edu/%7Ergrosse/courses/csc321_2017/slides/lec16.pdf) shows calculations that show this to be the case in a specific degenerate edge cases; but as for how it consistently provides such an advantage on more general basis is from what I understand still unclear.
  
  > **Comments:**
  > **username_4**: tl;dr: read page 6 of those slides  
  
  ---
  
  