Answers.Body: <p>Your intuition is correct, when you apply many local feature detectors (convolutions) and throw the results into one big pot (any global pooling), only the amount of detected features matters, not their location in the image, i.e. you achieved translation invariance.
  Doing this for deep ConvNets like you describe does not make a lot of sense to me, because applying the global pooling once will squash your feature map into a single feature vector. When you look at the shape before and after the global pooling operation, this would look as follows:</p>
  <p><code>[batch, height, width, channels] --global-pool--&gt; [batch, channels]</code></p>
  <p>Therefore your notion of space disappears completely which would prevent you from applying any subsequent spatial convolutions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-15T15:39:00.703Z
Answers.Text: amazing! this is exactly what I thought. So that's why usually, the global pooling layer is used only at the very last layer (in deep CNNs), and regular small sized poolings are used on the rest of the layers. Only thing is, these deep CNNs are then not exactly translation invariant, as the exact invariance will be lost in the small sized pooling layers. Is it then that we can not achieve exact translation equivariance for deep CNNs? (without augmentation or conveniently transforming data, only by architecture designs)
Answers.UserDisplayName: puradrogasincortar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-15T16:11:38.883Z
Answers.Text: That's exactly right! Regarding the second point: (I'm assuming you mixed up the terms invariance and equivariance a bit in your comment) Firstly, you are right, when local pooling is applied you trade local equivariance for local invariance and the more pooling layers you use the more this applies. However, it is possible to achieve full translational equivariance by using a fully convolutional model without pooling layers (often used in image segmentation tasks). It depends on the task how equivariant or invariant the model should be. I hope this clears it up a bit?
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-15T21:13:40.103Z
Answers.Text: oops yeah I meant invariance instead of equivariance in my last point. I understood everything! I imagine that fully convolutional models make use of padding and strides conveniently to decrease the spatial size at a higher rate; otherwise, it would take an enormous amount of conv layers to obtain a let's say 10 sized output from a high resolution image. Thanks, all clear!
Answers.UserDisplayName: puradrogasincortar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-16T12:22:56.360Z
Answers.Text: That's right! I'm glad it helped :)
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-15T15:27:09.373Z
Answers.LastActivityDate: 2023-03-15T15:27:09.373Z
Answers.UserDisplayName: Chillston
Body: <p>CNNs are naturally translation equivariant, meaning that if we translate the input, then the feature maps are translated the equally.</p>
  <p>With the use of max/avg pooling layers, this translation equivariance leads to approximate translation invariance, in the sense that it gives translation invariance for small translations, but for longer translation, the max/avg values could differ due to the limitations by the size of the pooling and the module of the translation.</p>
  <p>However, if I use the <strong>biggest size possible</strong> for the pooling, i.e. <em>global</em> max/avg pooling, then the CNN will be translation invariant in an exact way, for every translation of the input, no matter the strength (module) of the translation.</p>
  <p>Is this intuition correct? I can visualize this in my head, but I can't really visualize if this effect would hold for deep networks (i.e. after many conv-&gt;global pool layers are stacked).</p>
  
Tags: <convolutional-neural-networks><computer-vision>
Title: Are CNNs exactly translation invariant with global max/average pooling layers?
UserDisplayName: puradrogasincortar
ThreadText: # Are CNNs exactly translation invariant with global max/average pooling layers?
  
  (Asked by: username_0 on 2023-03-15)
  
  CNNs are naturally translation equivariant, meaning that if we translate the input, then the feature maps are translated the equally.
  
  With the use of max/avg pooling layers, this translation equivariance leads to approximate translation invariance, in the sense that it gives translation invariance for small translations, but for longer translation, the max/avg values could differ due to the limitations by the size of the pooling and the module of the translation.
  
  However, if I use the **biggest size possible** for the pooling, i.e. *global* max/avg pooling, then the CNN will be translation invariant in an exact way, for every translation of the input, no matter the strength (module) of the translation.
  
  Is this intuition correct? I can visualize this in my head, but I can't really visualize if this effect would hold for deep networks (i.e. after many conv->global pool layers are stacked).
  
  **Tags:** convolutional-neural-networks, computer-vision
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-15. Score: 2 | Accepted answer
  
  Your intuition is correct, when you apply many local feature detectors (convolutions) and throw the results into one big pot (any global pooling), only the amount of detected features matters, not their location in the image, i.e. you achieved translation invariance.
  Doing this for deep ConvNets like you describe does not make a lot of sense to me, because applying the global pooling once will squash your feature map into a single feature vector. When you look at the shape before and after the global pooling operation, this would look as follows:
  
  `[batch, height, width, channels] --global-pool--> [batch, channels]`
  
  Therefore your notion of space disappears completely which would prevent you from applying any subsequent spatial convolutions.
  
  > **Comments:**
  > **username_0**: amazing! this is exactly what I thought. So that's why usually, the global pooling layer is used only at the very last layer (in deep CNNs), and regular small sized poolings are used on the rest of the layers. Only thing is, these deep CNNs are then not exactly translation invariant, as the exact invariance will be lost in the small sized pooling layers. Is it then that we can not achieve exact translation equivariance for deep CNNs? (without augmentation or conveniently transforming data, only by architecture designs)  
  > **username_1**: That's exactly right! Regarding the second point: (I'm assuming you mixed up the terms invariance and equivariance a bit in your comment) Firstly, you are right, when local pooling is applied you trade local equivariance for local invariance and the more pooling layers you use the more this applies. However, it is possible to achieve full translational equivariance by using a fully convolutional model without pooling layers (often used in image segmentation tasks). It depends on the task how equivariant or invariant the model should be. I hope this clears it up a bit?  
  > **username_0**: oops yeah I meant invariance instead of equivariance in my last point. I understood everything! I imagine that fully convolutional models make use of padding and strides conveniently to decrease the spatial size at a higher rate; otherwise, it would take an enormous amount of conv layers to obtain a let's say 10 sized output from a high resolution image. Thanks, all clear!  
  > **username_1**: That's right! I'm glad it helped :)  
  
  ---
  
  