Answers.Body: <p>After finding the <a href="https://github.com/adobe/antialiased-cnns" rel="nofollow noreferrer">paper authors' Github</a>, I saw that, although they only have a MobileNet V2 model implemented, they choose the Subsample-after-ReLu option (the first one in the question).
  Although this doesn't fully answer my question, I'll take "the paper authors do it this way" as enough reason to prefer this over the alternative.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-06T23:08:30.037Z
Answers.Text: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction
Answers.UserDisplayName: x89
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-31T12:18:04.080Z
Answers.LastActivityDate: 2019-10-31T12:18:04.080Z
Answers.UserDisplayName: GPhilo
Body: <p>I'm trying to replace the strided convolutions of <a href="https://github.com/keras-team/keras-applications/blob/71acdcd98088501247f4b514b7cbbdf8182a05a4/keras_applications/mobilenet.py" rel="nofollow noreferrer">Keras' MobileNet implementation</a> with the <code>ConvBlurPool</code> operation as defined in the <a href="https://arxiv.org/abs/1904.11486" rel="nofollow noreferrer">Making Convolutional Networks Shift-Invariant Again paper</a>. In the paper, a <code>ConvBlurPool</code> is implemented as follows:</p>
  
  <p><span class="math-container">$$
  Relu \circ Conv_{k,s} \rightarrow Subsample_s \circ Blur_m \circ Relu \circ Conv_{k,1}
  $$</span>
  where <code>k</code> is the convolution's output kernels, <code>s</code> is the stride, <code>m</code> is the blurring kernel size and the subsample+blur is implemented as a strided convolution with a constant kernel.</p>
  
  <p>My issues start when batch normalization enter the picture.
  In MobileNet, a conv block is defined as follows (omitting the zero-padding):</p>
  
  <p><span class="math-container">$$
  Relu \circ BatchNorm \circ Conv_{k,s}
  $$</span></p>
  
  <p>I am leaning towards converting it to:</p>
  
  <p><span class="math-container">$$
  Subsample_s \circ Blur_m \circ Relu \circ BatchNorm \circ Conv_{k,1}
  $$</span></p>
  
  <p>i.e., putting the BN before the activation as it's normally done. This is not equivalent though, because the first BN operates on the downsampled signal.</p>
  
  <p>Another possibility would be:</p>
  
  <p><span class="math-container">$$
  BatchNorm \circ Subsample_s \circ Blur_m \circ Relu \circ Conv_{k,1}
  $$</span></p>
  
  <p>with the BN as last operation. This is also not equivalent, because now the BN comes after the ReLu.</p>
  
  <p>Is there any reason to prefer one option over the other? Are there any other options I'm not considering?</p>
  
Tags: <deep-learning><convolutional-neural-networks>
Title: Positioning of batch normalization layer when converting strided convolution to convolution + blurpool
UserDisplayName: GPhilo
ThreadText: # Positioning of batch normalization layer when converting strided convolution to convolution + blurpool
  
  (Asked by: username_0 on 2019-10-31)
  
  I'm trying to replace the strided convolutions of [Keras' MobileNet implementation](https://github.com/keras-team/keras-applications/blob/71acdcd98088501247f4b514b7cbbdf8182a05a4/keras_applications/mobilenet.py) with the `ConvBlurPool` operation as defined in the [Making Convolutional Networks Shift-Invariant Again paper](https://arxiv.org/abs/1904.11486). In the paper, a `ConvBlurPool` is implemented as follows:
  
  $$
  Relu \circ Conv\_{k,s} \rightarrow Subsample\_s \circ Blur\_m \circ Relu \circ Conv\_{k,1}
  $$
  where `k` is the convolution's output kernels, `s` is the stride, `m` is the blurring kernel size and the subsample+blur is implemented as a strided convolution with a constant kernel.
  
  My issues start when batch normalization enter the picture.
  In MobileNet, a conv block is defined as follows (omitting the zero-padding):
  
  $$
  Relu \circ BatchNorm \circ Conv\_{k,s}
  $$
  
  I am leaning towards converting it to:
  
  $$
  Subsample\_s \circ Blur\_m \circ Relu \circ BatchNorm \circ Conv\_{k,1}
  $$
  
  i.e., putting the BN before the activation as it's normally done. This is not equivalent though, because the first BN operates on the downsampled signal.
  
  Another possibility would be:
  
  $$
  BatchNorm \circ Subsample\_s \circ Blur\_m \circ Relu \circ Conv\_{k,1}
  $$
  
  with the BN as last operation. This is also not equivalent, because now the BN comes after the ReLu.
  
  Is there any reason to prefer one option over the other? Are there any other options I'm not considering?
  
  **Tags:** deep-learning, convolutional-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-10-31. Score: 0 | Accepted answer
  
  After finding the [paper authors' Github](https://github.com/adobe/antialiased-cnns), I saw that, although they only have a MobileNet V2 model implemented, they choose the Subsample-after-ReLu option (the first one in the question).
  Although this doesn't fully answer my question, I'll take "the paper authors do it this way" as enough reason to prefer this over the alternative.
  
  > **Comments:**
  > **username_1**: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction  
  
  ---
  
  