Answers.Body: <p>Neural networks are not <strong>invariant</strong> to translations, but <strong>equivariant</strong>,</p>
  <h3>Invariance vs Equivariance</h3>
  <p>Suppose we have input <span class="math-container">$x$</span> and the output <span class="math-container">$y=f(x)$</span> of some map between spaces <span class="math-container">$X$</span> and <span class="math-container">$Y$</span>. We apply transformation <span class="math-container">$T$</span> in the input domain. For general map,output will change in some complicated and unpredictable way. However, for certain class of maps, change of the output becomes very tractable.</p>
  <p><strong>Invariance</strong> means that output doesn't change after application of the map <span class="math-container">$T$</span>. Namely:
  <span class="math-container">$$
  f(T(x)) = f(x)
  $$</span></p>
  <p>For CNN example of the map, invariant to translations, is the <strong>GlobalPooling</strong> operation.</p>
  <p><strong>Equivariance</strong> means that symmetry transformation <span class="math-container">$T$</span> on the input domain leads to the symmetry transformation <span class="math-container">$T^{'}$</span> on the output. Here <span class="math-container">$T^{'}$</span> can be the same map <span class="math-container">$T$</span>, identity map - which reduces to invariance, or some other kind of transformation.</p>
  <p>This picture is illustration of translational equivariance.</p>
  <p><a href="https://i.sstatic.net/hxRvr.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hxRvr.png" alt="from https://towardsdatascience.com/translational-invariance-vs-translational-equivariance-f9fbc8fca63a" /></a></p>
  <h3>Equivariance of operations in CNN</h3>
  <ul>
  <li>Convolutions with <code>stride=1</code>:
  <span class="math-container">$$ f(T(x)) = T f(x)
  $$</span>
  Output feature map is shifted in same direction and number of steps.</li>
  <li>Downsampling operations. Convolutions with <code>stride=1</code>, <code>Pooling</code> (non-global):
  <span class="math-container">$$ f(T_{1/s}(x)) = T_{1/s} f(x)
  $$</span>
  They are equivariant to the subgroup of translations, which involves translations with integer number of strides.</li>
  <li><code>GlobalPooling</code> :
  <span class="math-container">$$ f(T(x)) = f(x)
  $$</span>
  These are invariant to arbitrary shifts, this property is useful in classification tasks.</li>
  </ul>
  <h3>Combination of layers</h3>
  <p>Stacking multiple equivariant layers you obtain equivariant architecture a whole.</p>
  <p>For classification layer it makes sense to put <code>GlobalPooling</code> in the end in order to for NN to output the same probabilities for the shifted image.</p>
  <p>For <strong>segmentation</strong> or <strong>detection</strong> problem architecture should be equivariant with the same map <span class="math-container">$T$</span>, in order to translate bounding boxes or segmentation masks by the same amount as the transform on the input.</p>
  <p>Non-global downsampling operations <em>reduce equivariance</em> to the subgroup with shifts integer multiples of stride.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T20:38:20.170Z
Answers.Text: Nice answer! Why though everyone says that is the pooling layers (`MaxPooling`) that help the network to achieve the translational invariance? See for example Figure 9.8 of the ![Deep Learning Book](https://www.deeplearningbook.org/).
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T04:38:46.813Z
Answers.LastActivityDate: 2021-07-09T04:38:46.813Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>FCNs can and typically have downsampling operations. For example, <a href="https://arxiv.org/pdf/1505.04597.pdf" rel="nofollow noreferrer">u-net</a> has downsampling (more precisely, max-pooling) operations. The difference between an FCN and a regular CNN is that the former does not have fully connected layers. See <a href="https://ai.stackexchange.com/a/21824/2444">this answer</a> for more info. </p>
  
  <p>Therefore, FCNs inherit the same properties of CNNs. There's nothing that a CNN (with fully connected layers) can do that an FCN cannot do. In fact, you can even simulate a fully connected layer with a convolution (with a kernel that has the same shape as the input volume).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-13T20:47:21.600Z
Answers.LastActivityDate: 2020-06-13T20:47:21.600Z
Answers.UserDisplayName: nbro
Answers.Body: <p>All convolutional networks (with or without max-pooling) are translation-invariant (AKA spatially invariant) because their filters slide over every position in the image. This means that if a pattern that "matches" a filter is present anywhere in the image, then at least one neuron should activate.</p>
  
  <p>Max-pooling, on the other hand, has nothing to do with spatial invariance. It's simply a regularization technique to help reduce the number of parameters later in the network by downsizing activation layers within the network. This can help combat overfitting, although it's not strictly necessary. Alternatively, neural networks can achieve the same effect by using a convolutional layer with a stride of 2 instead of 1.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T01:04:05.703Z
Answers.Text: what? Max pooling specifically creates translation invariance by only outputting the max value over a subset of the input space. Fully convolutional networks does not have this so I'm curious how/if it retains translation invariant properties. There are some architectures which have a bunch of Convolutional layers with strides to reduce output size with a softmax or SVM at the output. It's not clear why this is translation invariant at all.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T01:57:35.733Z
Answers.Text: Your description of max pooling is correct but that is not a significant form of spatial invariance. The convolutional layers themselves induce spatial invariance through the sliding action of their filters, so max pooling is not always needed. I think [this article](http://cs231n.github.io/convolutional-networks/) explains the difference well. Specifically, see the section called **Getting rid of pooling.**
Answers.UserDisplayName: Brett Daley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T01:59:55.070Z
Answers.Text: The convolution output is shifted if the input is shifted, it's output is not translation invariant. Max pooling specifically acts to produce translation invariance(at least according to the books I have read).
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T02:57:32.950Z
Answers.Text: That is true, but if you shift the input then the output of max pooling can change too. For example, imagine I shift my input image by 1 pixel when I am using 2x2 max pooling with a stride of 2. I am curious if you remember which book says that, so I can check to make sure I understand correctly.
Answers.UserDisplayName: Brett Daley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T03:03:52.673Z
Answers.Text: Does [this Quora answer](https://www.quora.com/Why-and-how-are-convolutional-neural-networks-translation-invariant) help? Looks like the OP is referring to the [Deep Learning Book](https://www.deeplearningbook.org/). It appears that both convolutional layers and max pooling provide (slightly different) forms of translation invariance. That might explain why you can remove max pooling from CNNs and still have translation invariance, like I said in my answer.
Answers.UserDisplayName: Brett Daley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T03:07:45.710Z
Answers.Text: @BrettDaley Please, make sure you distinguish between translation invariance and equivariance and which parts of the CNN actually provide each of these properties.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T20:29:39.237Z
Answers.Text: @nbro Can you explain why pooling adds the translation invariance?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-29T00:54:39.843Z
Answers.Text: @FourierFlux As you go deeper into the network the receptive field increases, meaning that the same pattern can be detected even if it is translated at large amount. So translation invariant can be achieved even without pooling by just using convolutions.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T00:40:12.087Z
Answers.LastActivityDate: 2020-02-21T00:40:12.087Z
Answers.UserDisplayName: Brett Daley
Body: <p>Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?</p>
  
  <p>If not, why do they perform as well as networks which use max-pooling? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-22T08:35:30.830Z
Comments.Text: Convolution is translationally invariant
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-22T21:29:10.083Z
Comments.Text: It's not. The output of a convolutional layer will be shifted.
Comments.UserDisplayName: FourierFlux
Tags: <deep-learning><convolutional-neural-networks><pooling><fully-convolutional-networks><max-pooling>
Title: Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?
UserDisplayName: FourierFlux
ThreadText: # Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?
  
  (Asked by: username_0 on 2020-02-21)
  
  Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?
  
  If not, why do they perform as well as networks which use max-pooling?
  
  **Tags:** deep-learning, convolutional-neural-networks, pooling, fully-convolutional-networks, max-pooling
  
  > **Comments:**
  > **username_1**: Convolution is translationally invariant  
  > **username_0**: It's not. The output of a convolutional layer will be shifted.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2021-07-09. Score: 2
  
  Neural networks are not **invariant** to translations, but **equivariant**,
  
  ### Invariance vs Equivariance
  
  Suppose we have input $x$ and the output $y=f(x)$ of some map between spaces $X$ and $Y$. We apply transformation $T$ in the input domain. For general map,output will change in some complicated and unpredictable way. However, for certain class of maps, change of the output becomes very tractable.
  
  **Invariance** means that output doesn't change after application of the map $T$. Namely:
  $$
  f(T(x)) = f(x)
  $$
  
  For CNN example of the map, invariant to translations, is the **GlobalPooling** operation.
  
  **Equivariance** means that symmetry transformation $T$ on the input domain leads to the symmetry transformation $T^{'}$ on the output. Here $T^{'}$ can be the same map $T$, identity map - which reduces to invariance, or some other kind of transformation.
  
  This picture is illustration of translational equivariance.
  
  [![from https://towardsdatascience.com/translational-invariance-vs-translational-equivariance-f9fbc8fca63a](https://i.sstatic.net/hxRvr.png)](https://i.sstatic.net/hxRvr.png)
  
  ### Equivariance of operations in CNN
  
  * Convolutions with `stride=1`:
    $$ f(T(x)) = T f(x)
    $$
    Output feature map is shifted in same direction and number of steps.
  * Downsampling operations. Convolutions with `stride=1`, `Pooling` (non-global):
    $$ f(T\_{1/s}(x)) = T\_{1/s} f(x)
    $$
    They are equivariant to the subgroup of translations, which involves translations with integer number of strides.
  * `GlobalPooling` :
    $$ f(T(x)) = f(x)
    $$
    These are invariant to arbitrary shifts, this property is useful in classification tasks.
  
  ### Combination of layers
  
  Stacking multiple equivariant layers you obtain equivariant architecture a whole.
  
  For classification layer it makes sense to put `GlobalPooling` in the end in order to for NN to output the same probabilities for the shifted image.
  
  For **segmentation** or **detection** problem architecture should be equivariant with the same map $T$, in order to translate bounding boxes or segmentation masks by the same amount as the transform on the input.
  
  Non-global downsampling operations *reduce equivariance* to the subgroup with shifts integer multiples of stride.
  
  > **Comments:**
  > **username_3**: Nice answer! Why though everyone says that is the pooling layers (`MaxPooling`) that help the network to achieve the translational invariance? See for example Figure 9.8 of the ![Deep Learning Book](https://www.deeplearningbook.org/).  
  
  ---
  
  ### Answer by username_4 on 2020-06-13. Score: 1
  
  FCNs can and typically have downsampling operations. For example, [u-net](https://arxiv.org/pdf/1505.04597.pdf) has downsampling (more precisely, max-pooling) operations. The difference between an FCN and a regular CNN is that the former does not have fully connected layers. See [this answer](https://ai.stackexchange.com/a/21824/2444) for more info.
  
  Therefore, FCNs inherit the same properties of CNNs. There's nothing that a CNN (with fully connected layers) can do that an FCN cannot do. In fact, you can even simulate a fully connected layer with a convolution (with a kernel that has the same shape as the input volume).
  
  
  ---
  
  ### Answer by username_5 on 2020-02-21. Score: 0
  
  All convolutional networks (with or without max-pooling) are translation-invariant (AKA spatially invariant) because their filters slide over every position in the image. This means that if a pattern that "matches" a filter is present anywhere in the image, then at least one neuron should activate.
  
  Max-pooling, on the other hand, has nothing to do with spatial invariance. It's simply a regularization technique to help reduce the number of parameters later in the network by downsizing activation layers within the network. This can help combat overfitting, although it's not strictly necessary. Alternatively, neural networks can achieve the same effect by using a convolutional layer with a stride of 2 instead of 1.
  
  > **Comments:**
  > **username_0**: what? Max pooling specifically creates translation invariance by only outputting the max value over a subset of the input space. Fully convolutional networks does not have this so I'm curious how/if it retains translation invariant properties. There are some architectures which have a bunch of Convolutional layers with strides to reduce output size with a softmax or SVM at the output. It's not clear why this is translation invariant at all.  
  > **username_5**: Your description of max pooling is correct but that is not a significant form of spatial invariance. The convolutional layers themselves induce spatial invariance through the sliding action of their filters, so max pooling is not always needed. I think [this article](http://cs231n.github.io/convolutional-networks/) explains the difference well. Specifically, see the section called **Getting rid of pooling.**  
  > **username_0**: The convolution output is shifted if the input is shifted, it's output is not translation invariant. Max pooling specifically acts to produce translation invariance(at least according to the books I have read).  
  > **username_5**: That is true, but if you shift the input then the output of max pooling can change too. For example, imagine I shift my input image by 1 pixel when I am using 2x2 max pooling with a stride of 2. I am curious if you remember which book says that, so I can check to make sure I understand correctly.  
  > **username_5**: Does [this Quora answer](https://www.quora.com/Why-and-how-are-convolutional-neural-networks-translation-invariant) help? Looks like the OP is referring to the [Deep Learning Book](https://www.deeplearningbook.org/). It appears that both convolutional layers and max pooling provide (slightly different) forms of translation invariance. That might explain why you can remove max pooling from CNNs and still have translation invariance, like I said in my answer.  
  > **username_4**: @username_5 Please, make sure you distinguish between translation invariance and equivariance and which parts of the CNN actually provide each of these properties.  
  > **username_3**: @username_4 Can you explain why pooling adds the translation invariance?  
  > **username_3**: @username_0 As you go deeper into the network the receptive field increases, meaning that the same pattern can be detected even if it is translated at large amount. So translation invariant can be achieved even without pooling by just using convolutions.  
  
  ---
  
  