Answers.Body: <p>I realized that groups in PyTorch works along the same dimension as channels, I thought it worked along the same dimension as the data. I believe the GitHub example does run the same data through each group, it just has an initial convolution to increase the channel length.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-02T04:43:21.267Z
Answers.LastActivityDate: 2024-02-02T04:43:21.267Z
Answers.UserDisplayName: eop3
Body: <p>I have been learning how to create ResNext neural networks, and am confused how input works with cardinality. In <a href="https://datascience.stackexchange.com/questions/47559/cardinality-vs-width-in-the-resnext-architecture">this</a> answer, it seems that it's saying that the data is added together, which I assumed meant some sort of aggregation like element-wise summation, but I've ready other <a href="https://blog.paperspace.com/popular-deep-learning-architectures-densenet-mnasnet-shufflenet/#:%7E:text=In%20a%20nutshell%2C%20the%20repetition%20strategy%20of%20ResNet,output%20where%20each%20block%20follows%20the%20same%20topology." rel="nofollow noreferrer">thing</a>s
  that say that ResNext undergoes split-transform-merge, the same that inception network uses. I also was looking at <a href="https://github.com/prlz77/ResNeXt.pytorch/blob/master/models/model.py" rel="nofollow noreferrer">this</a> Github repo that is a ResNext model in PyTorch, and while I don't completely understand it, it seemed like it just had multiple convolutions working on different sections of the data, instead of running multiple copies of the data through separate convolutions. This is the section I was looking at:</p>
  <pre><code>        width_ratio = out_channels / (widen_factor * 64.)
          D = cardinality * int(base_width * width_ratio)
          self.conv_reduce = nn.Conv2d(in_channels, D, kernel_size=1, stride=1, padding=0, bias=False)
          self.bn_reduce = nn.BatchNorm2d(D)
          self.conv_conv = nn.Conv2d(D, D, kernel_size=3, stride=stride, padding=1, groups=cardinality, bias=False)
          self.bn = nn.BatchNorm2d(D)
          self.conv_expand = nn.Conv2d(D, out_channels, kernel_size=1, stride=1, padding=0, bias=False)
          self.bn_expand = nn.BatchNorm2d(out_channels)
  </code></pre>
  <p>What I would have expected to see (and I am very naive about this), would be concatenating the data to itself cardinality times, and then using cardinality groups.
  So, does ResNext send identical copies of data through separate branches, or split it into sections and run those sections through separate branches?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-31T01:09:42.623Z
Comments.Text: Your second link doesn't work on my end. It seems to be a Bing link -- if it's search result, can you replace it with a link to the original website?
Comments.UserDisplayName: Alexander Wan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-31T01:10:47.133Z
Comments.Text: Also, for the third link it'd be more clear if you added the specific snippet of code that you're referring to in your question.
Comments.UserDisplayName: Alexander Wan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-31T03:38:37.167Z
Comments.Text: My bad, sorry. I just copied the link from the edge search results after I found it, but apparently that doesn't actually copy the link. I fixed it and added some code.
Comments.UserDisplayName: eop3
Tags: <neural-networks><convolutional-neural-networks><pytorch><residual-networks>
Title: Does ResNext split data or copy it?
UserDisplayName: eop3
ThreadText: # Does ResNext split data or copy it?
  
  (Asked by: username_0 on 2024-01-30)
  
  I have been learning how to create ResNext neural networks, and am confused how input works with cardinality. In this answer, it seems that it's saying that the data is added together, which I assumed meant some sort of aggregation like element-wise summation, but I've ready other things
  that say that ResNext undergoes split-transform-merge, the same that inception network uses. I also was looking at this Github repo that is a ResNext model in PyTorch, and while I don't completely understand it, it seemed like it just had multiple convolutions working on different sections of the data, instead of running multiple copies of the data through separate convolutions. This is the section I was looking at:
          width_ratio = out_channels / (widen_factor * 64.)
          D = cardinality * int(base_width * width_ratio)
          self.conv_reduce = nn.Conv2d(in_channels, D, kernel_size=1, stride=1, padding=0, bias=False)
          self.bn_reduce = nn.BatchNorm2d(D)
          self.conv_conv = nn.Conv2d(D, D, kernel_size=3, stride=stride, padding=1, groups=cardinality, bias=False)
          self.bn = nn.BatchNorm2d(D)
          self.conv_expand = nn.Conv2d(D, out_channels, kernel_size=1, stride=1, padding=0, bias=False)
          self.bn_expand = nn.BatchNorm2d(out_channels)
  
  What I would have expected to see (and I am very naive about this), would be concatenating the data to itself cardinality times, and then using cardinality groups.
  So, does ResNext send identical copies of data through separate branches, or split it into sections and run those sections through separate branches?
  
  **Tags:** neural-networks, convolutional-neural-networks, pytorch, residual-networks
  > **Comments:**
  > **username_1**: Your second link doesn't work on my end. It seems to be a Bing link -- if it's search result, can you replace it with a link to the original website?  
  > **username_1**: Also, for the third link it'd be more clear if you added the specific snippet of code that you're referring to in your question.  
  > **username_0**: My bad, sorry. I just copied the link from the edge search results after I found it, but apparently that doesn't actually copy the link. I fixed it and added some code.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-02-02. Score: 1 | Accepted answer
  
  I realized that groups in PyTorch works along the same dimension as channels, I thought it worked along the same dimension as the data. I believe the GitHub example does run the same data through each group, it just has an initial convolution to increase the channel length.
  
  
  ---
  
  