Answers.Body: <p>What happens during the patch merging? Concatenationating is only one part of the whole operations. Below, I quote the code from the original implementation and explain step by step what happens.</p>
  <pre><code>x = x.view(B, H, W, C)
  </code></pre>
  <p>Firstly, x, the input image, needs to be reshaped to regain the height H and width W.</p>
  <pre><code>x0 = x[:, 0::2, 0::2, :]  # B H/2 W/2 C
  x1 = x[:, 1::2, 0::2, :]  # B H/2 W/2 C
  x2 = x[:, 0::2, 1::2, :]  # B H/2 W/2 C
  x3 = x[:, 1::2, 1::2, :]  # B H/2 W/2 C
  </code></pre>
  <p>Then, the input is vectorised. Since Swin merges 4 tokens into 1 patch, one token out of a square of 4 is selected. How does the selection process work? Each vector is specialised in one corner of the square. x0 always consists of the top left corner, x1 of the bottom left corner and so on. So during this prep step, the input is divided into 4 quarters.</p>
  <pre><code>x = torch.cat([x0, x1, x2, x3], -1)  # B H/2 W/2 4*C
  </code></pre>
  <p>The next step is the one you already found. The 4 vectors are concatenated. Since each vector is a quarter of the input, they are half the height (H/2) and half the width (W/2) of the input. 4 quarters exists so the channel size quadruples 4*C.</p>
  <p>By the end of this step, the desired downsampled shape in the height and width dimension are created BUT there are two many channels. Swin doubles it channels during each patch merging operation, not quadruples it. That means Swin keeps 2 tokens to represent a square of 4 tokens.</p>
  <p>How do we get from 4 channels to 2? Via reduction through a linear layer.</p>
  <pre><code>x = self.reduction(x)
  </code></pre>
  <p>This operation reduces the channel size from 4 to 2.</p>
  <p>How is the reduction defined?</p>
  <pre><code>self.reduction = nn.Linear(4 * dim, 2 * dim, bias=False)
  </code></pre>
  <p>As a linear projection from dim 4 to 2.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-08T10:24:47.513Z
Answers.Text: But why doesn't it use a normal convolution to reduce each dimension by half and double the channels?
Answers.UserDisplayName: Cloudy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-11T17:12:46.053Z
Answers.LastActivityDate: 2022-05-11T17:12:46.053Z
Answers.UserDisplayName: emely_pi
Body: <p>In the <a href="https://arxiv.org/abs/2103.14030" rel="nofollow noreferrer">SOTA paper: SWIN-Transformers</a>, the authors have tried their best to explain everything clearly. I have got an idea of how it works except the <code>Patch Merging</code> part. I found some blogs and other things explaining this but still I am not able to comprehend how the shape changes and how come the <strong>SHAPE</strong> of windows are changed at the time when the only thing thy are doing is Concatenating the neighbouring <code>2x2</code> patches. Could someone please explain it in Laymen terms or maybe a video link or something intuitive explanation.</p>
  <ul>
  <li><a href="https://towardsdatascience.com/swin-vision-transformers-hacking-the-human-eye-4223ba9764c3" rel="nofollow noreferrer">This awesome blog explains every detail clearly</a></li>
  <li><a href="https://blog.csdn.net/cangafuture/article/details/116396156" rel="nofollow noreferrer">Implementation in <code>Pytorch</code> from scratch with explanation</a></li>
  <li><a href="https://keras.io/examples/vision/swin_transformers/" rel="nofollow noreferrer"><code>Keras / Tensorflow</code> Implementation</a></li>
  </ul>
  <p>Below are some intuitive images which I am trying to grasp in bits and pieces<a href="https://i.sstatic.net/n39a1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/n39a1.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/3IQMx.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3IQMx.jpg" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><machine-learning><deep-learning><transformer><vision-transformer>
Title: How "Patch Merging" works in SWIN-Transformers?
UserDisplayName: Deshwal
ThreadText: # How "Patch Merging" works in SWIN-Transformers?
  
  (Asked by: username_0 on 2022-03-08)
  
  In the [SOTA paper: SWIN-Transformers](https://arxiv.org/abs/2103.14030), the authors have tried their best to explain everything clearly. I have got an idea of how it works except the `Patch Merging` part. I found some blogs and other things explaining this but still I am not able to comprehend how the shape changes and how come the **SHAPE** of windows are changed at the time when the only thing thy are doing is Concatenating the neighbouring `2x2` patches. Could someone please explain it in Laymen terms or maybe a video link or something intuitive explanation.
  
  * [This awesome blog explains every detail clearly](https://towardsdatascience.com/swin-vision-transformers-hacking-the-human-eye-4223ba9764c3)
  * [Implementation in `Pytorch` from scratch with explanation](https://blog.csdn.net/cangafuture/article/details/116396156)
  * [`Keras / Tensorflow` Implementation](https://keras.io/examples/vision/swin_transformers/)
  
  Below are some intuitive images which I am trying to grasp in bits and pieces[![enter image description here](https://i.sstatic.net/n39a1.png)](https://i.sstatic.net/n39a1.png)
  
  [![enter image description here](https://i.sstatic.net/3IQMx.jpg)](https://i.sstatic.net/3IQMx.jpg)
  
  **Tags:** neural-networks, machine-learning, deep-learning, transformer, vision-transformer
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-11. Score: 1
  
  What happens during the patch merging? Concatenationating is only one part of the whole operations. Below, I quote the code from the original implementation and explain step by step what happens.
  
  ```
  x = x.view(B, H, W, C)
  
  ```
  
  Firstly, x, the input image, needs to be reshaped to regain the height H and width W.
  
  ```
  x0 = x[:, 0::2, 0::2, :]  # B H/2 W/2 C
  x1 = x[:, 1::2, 0::2, :]  # B H/2 W/2 C
  x2 = x[:, 0::2, 1::2, :]  # B H/2 W/2 C
  x3 = x[:, 1::2, 1::2, :]  # B H/2 W/2 C
  
  ```
  
  Then, the input is vectorised. Since Swin merges 4 tokens into 1 patch, one token out of a square of 4 is selected. How does the selection process work? Each vector is specialised in one corner of the square. x0 always consists of the top left corner, x1 of the bottom left corner and so on. So during this prep step, the input is divided into 4 quarters.
  
  ```
  x = torch.cat([x0, x1, x2, x3], -1)  # B H/2 W/2 4*C
  
  ```
  
  The next step is the one you already found. The 4 vectors are concatenated. Since each vector is a quarter of the input, they are half the height (H/2) and half the width (W/2) of the input. 4 quarters exists so the channel size quadruples 4\*C.
  
  By the end of this step, the desired downsampled shape in the height and width dimension are created BUT there are two many channels. Swin doubles it channels during each patch merging operation, not quadruples it. That means Swin keeps 2 tokens to represent a square of 4 tokens.
  
  How do we get from 4 channels to 2? Via reduction through a linear layer.
  
  ```
  x = self.reduction(x)
  
  ```
  
  This operation reduces the channel size from 4 to 2.
  
  How is the reduction defined?
  
  ```
  self.reduction = nn.Linear(4 * dim, 2 * dim, bias=False)
  
  ```
  
  As a linear projection from dim 4 to 2.
  
  > **Comments:**
  > **username_2**: But why doesn't it use a normal convolution to reduce each dimension by half and double the channels?  
  
  ---
  
  