Answers.Body: <p>I suggest to follow the official <a href="https://arxiv.org/pdf/1505.04597.pdf" rel="nofollow noreferrer">U-NET</a> implementation.</p>
  <p>To me, the second option <code>Conv2d -&gt; BatchNorm2d -&gt; ReLU (-&gt; MaxPool2d) -&gt; BatchNorm2d -&gt; Conv2d -&gt; ReLU (-&gt; MaxPool2D)</code> seems more a mistake that an alternative:</p>
  <ul>
  <li>The part <code>-&gt; BatchNorm2d -&gt; ReLU (-&gt; MaxPool2d) -&gt; BatchNorm2d -&gt;</code> doesn't make much sense: the first BN already centers the activation then max-pool takes only the max that doesn't introduce any substantial change in distribution, so the second BN in my opinion is not that much necessary.</li>
  <li>The following part <code>-&gt; Conv2d -&gt; ReLU (-&gt; MaxPool2D)</code>, now, lacks the BN operation which should be put in between <code>conv2d</code> and <code>ReLU</code>.</li>
  </ul>
  <p>So I suggest you to follow the <code>Conv2d -&gt; BN -&gt; ReLU (-&gt; MaxPool)</code> design. At the best you can swap <code>BN</code> with the activation (i.e. <code>conv -&gt; relu -&gt; BN</code>): there is a debate about this regarding BN, some people think is better some other that's equivalent..</p>
  <blockquote>
  <p>Is there a &quot;correct&quot; order for these operations for the encoder/decoder part of the UNet architecture?</p>
  </blockquote>
  <p>The correct order, is the first option you specified.</p>
  <blockquote>
  <p>What is the intuition behind shifting the order of these operations in the model?</p>
  </blockquote>
  <p>At most it can be motivate by re-normalizing again (by means of BN) the activation or even the max-pool's output. But if so, there are two missing BNs: after the second ReLU and max-pool.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-29T17:46:57.837Z
Answers.LastActivityDate: 2023-05-29T17:46:57.837Z
Answers.UserDisplayName: Luca Anzalone
Body: <p><strong>Context</strong></p>
  <p>I'm investigating the UNet architecture for a little while now. After investigating the structure of the official UNet architecture as proposed in the official paper I noticed a recurrent pattern of <strong>Conv2d-&gt;BatchNorm2d-&gt;ReLU(-&gt;MaxPool2d)-&gt;Conv2d-&gt;BatchNorm2d-&gt;ReLU(-&gt;MaxPool2d)</strong> for the encoder part but I have also came across other implementations of a custom UNet where this order is different like <strong>Conv2d-&gt;BatchNorm2d-&gt;ReLU(-&gt;MaxPool2d)-&gt;BatchNorm2d-&gt;Conv2d-&gt;ReLU(-&gt;MaxPool2D)</strong> where the operation in the paranthesis is considered optional.</p>
  <p>I've tried training both models and they both work reasonably well but I still want to understand the following:</p>
  <p><strong>Question</strong></p>
  <ol>
  <li>Is there a &quot;correct&quot; order for these operations for the encoder/decoder part of the UNet architecture?</li>
  <li>What is the intuition behind shifting the order of these operations in the model?</li>
  </ol>
  
Tags: <convolution><batch-normalization><pooling><encoder-decoder>
Title: Is there a correct order of "conv2d", "batchnorm2d", "ReLU/LeakyReLU", "MaxPool2d" for UNet like architectures?
UserDisplayName: timu vlad
ThreadText: # Is there a correct order of "conv2d", "batchnorm2d", "ReLU/LeakyReLU", "MaxPool2d" for UNet like architectures?
  
  (Asked by: username_0 on 2023-05-29)
  
  **Context**
  
  I'm investigating the UNet architecture for a little while now. After investigating the structure of the official UNet architecture as proposed in the official paper I noticed a recurrent pattern of **Conv2d->BatchNorm2d->ReLU(->MaxPool2d)->Conv2d->BatchNorm2d->ReLU(->MaxPool2d)** for the encoder part but I have also came across other implementations of a custom UNet where this order is different like **Conv2d->BatchNorm2d->ReLU(->MaxPool2d)->BatchNorm2d->Conv2d->ReLU(->MaxPool2D)** where the operation in the paranthesis is considered optional.
  
  I've tried training both models and they both work reasonably well but I still want to understand the following:
  
  **Question**
  
  1. Is there a "correct" order for these operations for the encoder/decoder part of the UNet architecture?
  2. What is the intuition behind shifting the order of these operations in the model?
  
  **Tags:** convolution, batch-normalization, pooling, encoder-decoder
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-29. Score: 2 | Accepted answer
  
  I suggest to follow the official [U-NET](https://arxiv.org/pdf/1505.04597.pdf) implementation.
  
  To me, the second option `Conv2d -> BatchNorm2d -> ReLU (-> MaxPool2d) -> BatchNorm2d -> Conv2d -> ReLU (-> MaxPool2D)` seems more a mistake that an alternative:
  
  * The part `-> BatchNorm2d -> ReLU (-> MaxPool2d) -> BatchNorm2d ->` doesn't make much sense: the first BN already centers the activation then max-pool takes only the max that doesn't introduce any substantial change in distribution, so the second BN in my opinion is not that much necessary.
  * The following part `-> Conv2d -> ReLU (-> MaxPool2D)`, now, lacks the BN operation which should be put in between `conv2d` and `ReLU`.
  
  So I suggest you to follow the `Conv2d -> BN -> ReLU (-> MaxPool)` design. At the best you can swap `BN` with the activation (i.e. `conv -> relu -> BN`): there is a debate about this regarding BN, some people think is better some other that's equivalent..
  
  > Is there a "correct" order for these operations for the encoder/decoder part of the UNet architecture?
  
  The correct order, is the first option you specified.
  
  > What is the intuition behind shifting the order of these operations in the model?
  
  At most it can be motivate by re-normalizing again (by means of BN) the activation or even the max-pool's output. But if so, there are two missing BNs: after the second ReLU and max-pool.
  
  
  ---
  
  