Answers.Body: <h2>Question 1</h2>
  
  <p>To make it simple</p>
  
  <ul>
  <li><p>You have a transformation <span class="math-container">$T$</span> and an operator <span class="math-container">$C$</span> acting on a given input <span class="math-container">$x$</span></p></li>
  <li><p>Let's say you do this experiment</p>
  
  <ul>
  <li>compute <span class="math-container">$y_{1} = C(T(x))$</span></li>
  <li>compute <span class="math-container">$y_{2} = C(x)$</span></li>
  </ul></li>
  <li><p>You can get three different results:</p>
  
  <ul>
  <li><span class="math-container">$y_{1} = y_{2}$</span> then you can say the operator is invariant with respect to the given transformation</li>
  <li><span class="math-container">$y_{1} = T(y_{2})$</span> then you can say the operator is equivariant to the given transformation as applying it to the input basically reflects its effect completely on the output</li>
  <li>none of the 2</li>
  </ul></li>
  </ul>
  
  <h2>Questions 2</h2>
  
  <h3>Q2.1</h3>
  
  <ul>
  <li>Spatial Pooling is not the only way to perform dimensionality reduction. You can achieve it even simply applying the Conv Kernel with no padding.</li>
  </ul>
  
  <p>For example, let's say your input is a <code>WxHxC</code> tensor and you are applying a kernel which is <code>KxKxC</code> the result will have spatial domain size <code>(W-(K/2))x(H-(K/2))</code> with <code>K/2</code> the integer truncation so if <code>K=5</code> then <code>K/2=2</code>.</p>
  
  <p>Alternatively you can reduce the spatial domain with stride.</p>
  
  <h3>Q2.2</h3>
  
  <ul>
  <li>It seems to me you are talking about a sort of Non Max Suppression Operator rather an emergent behaviour, but certainly with the proper supervision signal you can train a CNN to do this work (even if practically it does not make sense as you can explicitly define it).</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-25T19:18:07.813Z
Answers.Text: I don't think I can use Non-Max Suppression, because what I want is a network that will detect if it's centered around the local maximum or not.
Answers.UserDisplayName: 0x90
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-25T22:31:59.917Z
Answers.Text: The point is you do not need to teach the network (via supervised learning) how to do this (even if you can certainly do this) as you can easily code an algorithm (if you do not want to call this algo NMS it is not a problem for me :) )
Answers.UserDisplayName: Nicola Bernini
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-25T22:35:00.633Z
Answers.Text: so you suggest to have 3x3 CNN with pooling and then use NMS to locate the exact pixel after running the cnn on each pixel.
Answers.UserDisplayName: 0x90
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-25T22:41:48.727Z
Answers.Text: If you just want local max you do not need any complete CNN, it seems you just need a sort of max pooling which, instead of returning the value, returns the pixel with max value position: you can easily code it yourself 
  (and if you can not, just post a question here or on SO and me / somebody else in the community will be happy to help)
Answers.UserDisplayName: Nicola Bernini
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-25T22:43:41.730Z
Answers.Text: my data set is more complicated. It needs to be a max in some subgraph adjacency matrix
Answers.UserDisplayName: 0x90
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-25T22:48:28.550Z
Answers.Text: I think it's easier if you DM me on Twitter with more details @NicolaBernini
Answers.UserDisplayName: Nicola Bernini
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-25T16:58:46.670Z
Answers.LastActivityDate: 2019-04-26T11:48:58.593Z
Answers.LastEditDate: 2019-04-26T11:48:58.593Z
Answers.UserDisplayName: Nicola Bernini
Body: <p>I am trying to understand <a href="https://aboveintelligent.com/ml-cnn-translation-equivariance-and-invariance-da12e8ab7049" rel="nofollow noreferrer">this post</a>, but I get confused by the definitions and the differences. What's definition of equivariant?</p>
  
  <p>If I remove all the pooling layers from a CNN, will it make the network to detect features in pixel resolution? For example, detecting the local maximum of a pixel. For example, can a CNN be designed to return True for the following case?</p>
  
  <p><a href="https://i.sstatic.net/i5t7p.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/i5t7p.png" alt="Enter image description here"></a></p>
  
  <p>And False for the shifted window:</p>
  
  <p><a href="https://i.sstatic.net/2l7Gh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2l7Gh.png" alt="Enter image description here"></a></p>
  
  <p>In the second case it returns false because the 3x3 submatrix isn't centered (yellow dash line) around the local maximum.</p>
  
  <p>Will an architecture that is</p>
  
  <pre><code>from keras.layers import Dense, Conv2D, Flatten
  model = Sequential()
  model.add(Conv2D(128, kernel_size=2, activation=’relu’, padding='same', input_shape=(3,3,1)))
  model.add(Conv2D(64, kernel_size=2, activation=’relu’, padding='same'))
  model.add(Flatten())
  model.add(Dense(10, activation=’softmax’))
  </code></pre>
  
  <p>be able to differentiate between the tiling of the larger grayscale image?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-25T16:02:04.780Z
Comments.Text: Hi! Can you please ask one question per post? Just ask about the difference between invariance and equivariance in this post and ask the other question in another post.
Comments.UserDisplayName: nbro
Tags: <machine-learning><convolutional-neural-networks><keras>
Title: How can I suppress a CNN’s translation invariant or translation equivariant?
UserDisplayName: 0x90
ThreadText: # How can I suppress a CNN’s translation invariant or translation equivariant?
  
  (Asked by: username_0 on 2019-04-25)
  
  I am trying to understand this post, but I get confused by the definitions and the differences. What's definition of equivariant?
  If I remove all the pooling layers from a CNN, will it make the network to detect features in pixel resolution? For example, detecting the local maximum of a pixel. For example, can a CNN be designed to return True for the following case?
  
  And False for the shifted window:
  
  In the second case it returns false because the 3x3 submatrix isn't centered (yellow dash line) around the local maximum.
  Will an architecture that is
  from keras.layers import Dense, Conv2D, Flatten
  model = Sequential()
  model.add(Conv2D(128, kernel_size=2, activation=’relu’, padding='same', input_shape=(3,3,1)))
  model.add(Conv2D(64, kernel_size=2, activation=’relu’, padding='same'))
  model.add(Flatten())
  model.add(Dense(10, activation=’softmax’))
  
  be able to differentiate between the tiling of the larger grayscale image?
  
  **Tags:** machine-learning, convolutional-neural-networks, keras
  > **Comments:**
  > **username_1**: Hi! Can you please ask one question per post? Just ask about the difference between invariance and equivariance in this post and ask the other question in another post.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-04-25. Score: 1
  
  ## Question 1
  
  To make it simple
  
  * You have a transformation $T$ and an operator $C$ acting on a given input $x$
  * Let's say you do this experiment
  
    + compute $y\_{1} = C(T(x))$
    + compute $y\_{2} = C(x)$
  * You can get three different results:
  
    + $y\_{1} = y\_{2}$ then you can say the operator is invariant with respect to the given transformation
    + $y\_{1} = T(y\_{2})$ then you can say the operator is equivariant to the given transformation as applying it to the input basically reflects its effect completely on the output
    + none of the 2
  
  ## Questions 2
  
  ### Q2.1
  
  * Spatial Pooling is not the only way to perform dimensionality reduction. You can achieve it even simply applying the Conv Kernel with no padding.
  
  For example, let's say your input is a `WxHxC` tensor and you are applying a kernel which is `KxKxC` the result will have spatial domain size `(W-(K/2))x(H-(K/2))` with `K/2` the integer truncation so if `K=5` then `K/2=2`.
  
  Alternatively you can reduce the spatial domain with stride.
  
  ### Q2.2
  
  * It seems to me you are talking about a sort of Non Max Suppression Operator rather an emergent behaviour, but certainly with the proper supervision signal you can train a CNN to do this work (even if practically it does not make sense as you can explicitly define it).
  
  > **Comments:**
  > **username_0**: I don't think I can use Non-Max Suppression, because what I want is a network that will detect if it's centered around the local maximum or not.  
  > **username_2**: The point is you do not need to teach the network (via supervised learning) how to do this (even if you can certainly do this) as you can easily code an algorithm (if you do not want to call this algo NMS it is not a problem for me :) )  
  > **username_0**: so you suggest to have 3x3 CNN with pooling and then use NMS to locate the exact pixel after running the cnn on each pixel.  
  > **username_2**: If you just want local max you do not need any complete CNN, it seems you just need a sort of max pooling which, instead of returning the value, returns the pixel with max value position: you can easily code it yourself  (and if you can not, just post a question here or on SO and me / somebody else in the community will be happy to help)  
  > **username_0**: my data set is more complicated. It needs to be a max in some subgraph adjacency matrix  
  > **username_2**: I think it's easier if you DM me on Twitter with more details @NicolaBernini  
  
  ---
  
  