Answers.Body: <p>I found that this was the old Resnet design and they shortly came up with a newer design that does what I'm talking about: Identity Mappings in Deep Residual Networks</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-06T21:36:11.390Z
Answers.LastActivityDate: 2024-03-06T21:36:11.390Z
Answers.UserDisplayName: JobHunter69
Body: <p>Resnet block:</p>
  <pre><code>    identity = x
  
      out = self.conv1(x)
      out = self.bn1(out)
      out = self.relu(out)
  
      out = self.conv2(out)
      out = self.bn2(out)
  
      if self.downsample is not None:
          identity = self.downsample(x)
  
      out += identity
      out = self.relu(out)
  </code></pre>
  <p>we can see that the skip connection comes out of the activation. In order for the output of layer n to be an input of an arbitrary layer m where m &gt; n, either the identity addition should come after self.relu, or the incoming input should come before the relu in the previous block. Why do they do it like this? The information cannot flow now</p>
  
Tags: <neural-networks><deep-learning><residual-networks>
Title: Information cannot directly flow from past layer to future layer in resnet?
UserDisplayName: JobHunter69
ThreadText: # Information cannot directly flow from past layer to future layer in resnet?
  
  (Asked by: username_0 on 2024-03-06)
  
  Resnet block:
      identity = x
  
      out = self.conv1(x)
      out = self.bn1(out)
      out = self.relu(out)
  
      out = self.conv2(out)
      out = self.bn2(out)
  
      if self.downsample is not None:
          identity = self.downsample(x)
  
      out += identity
      out = self.relu(out)
  
  we can see that the skip connection comes out of the activation. In order for the output of layer n to be an input of an arbitrary layer m where m > n, either the identity addition should come after self.relu, or the incoming input should come before the relu in the previous block. Why do they do it like this? The information cannot flow now
  
  **Tags:** neural-networks, deep-learning, residual-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-03-06. Score: 1
  
  I found that this was the old Resnet design and they shortly came up with a newer design that does what I'm talking about: Identity Mappings in Deep Residual Networks
  
  
  ---
  
  