Answers.Body: <p>CNNs work by applying filters over the entire image. The same filter is applied at every pixel in the image. That is, the same weights are used at every pixel.</p>
  <p>Note, when I say &quot;at every pixel&quot; this means across the spatial dimension HxW of the image. You can also have attention in the channel dimension. See for example Squeeze and Excitation: <a href="https://arxiv.org/pdf/1709.01507.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1709.01507.pdf</a></p>
  <p>While this is one of the strengths of CNNs, since it drastically reduces the number of parameters of a network, you can imagine that it may not make sense to treat every part of the image the same regardless of the content. This is what the attention gate is for.</p>
  <p>By performing an element-wise multiplication of the output of a CNN layer with a gate tensor (typically clamped to the 0-1 range with a sigmoid) we can effectively down weigh or ignore features in certain areas of the image.</p>
  <p>CNNs without AG can typically learn the same things but may need more channels and layers. Attention gates allow to treat filters differently depending on the content and hence make it easier to learn with less filters.</p>
  <p>What you use as the attention tensor can vary. In the paper you referenced, the tensor was one of the input tensors of the convolutional layer. In this case, it is called self-attention. However, with attention in general this gate tensor may also come from other information such as another network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T15:33:53.237Z
Answers.Text: "By performing an element-wise multiplication of the output of a CNN layer with a gate tensor (typically clamped to the 0-1 range with a sigmoid) we can effectively down weigh or ignore features in certain areas of the image."
  
  Isn't this similar to having a CNN layer followed by a fully connected one with sigmoid activation? The only difference is that we're performing element wise rather than tensor multiplication
Answers.UserDisplayName: mesllo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-12T18:48:51.947Z
Answers.LastActivityDate: 2020-12-12T18:48:51.947Z
Answers.UserDisplayName: a crazy Minion
Body: <p>Attention models/gates are used to focus/pay attention to the important regions. According to <a href="https://arxiv.org/pdf/1804.03999.pdf" rel="nofollow noreferrer">this paper</a>, the authors describe that a model with Attention Gate (AG) can be trained from scratch. Then the AGs automatically learn to focus on the target.</p>
  <p>What I am having trouble understanding is that, in the context of computer vision, doesn't a filter from the convolutional layers learn the region of interest?</p>
  <p>The authors say that adding Attention Gate reduces complexity when compared with multi-stage CNNs. But the job a trained AG would do is the same as that of a filter in a convolutional layer that would lead to the correct output, right?</p>
  
Tags: <convolutional-neural-networks><computer-vision><attention><filters>
Title: What is the difference between Attention Gate and CNN filters?
UserDisplayName: Prasanjit Rath
ThreadText: # What is the difference between Attention Gate and CNN filters?
  
  (Asked by: username_0 on 2020-12-10)
  
  Attention models/gates are used to focus/pay attention to the important regions. According to [this paper](https://arxiv.org/pdf/1804.03999.pdf), the authors describe that a model with Attention Gate (AG) can be trained from scratch. Then the AGs automatically learn to focus on the target.
  
  What I am having trouble understanding is that, in the context of computer vision, doesn't a filter from the convolutional layers learn the region of interest?
  
  The authors say that adding Attention Gate reduces complexity when compared with multi-stage CNNs. But the job a trained AG would do is the same as that of a filter in a convolutional layer that would lead to the correct output, right?
  
  **Tags:** convolutional-neural-networks, computer-vision, attention, filters
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-12. Score: 3
  
  CNNs work by applying filters over the entire image. The same filter is applied at every pixel in the image. That is, the same weights are used at every pixel.
  
  Note, when I say "at every pixel" this means across the spatial dimension HxW of the image. You can also have attention in the channel dimension. See for example Squeeze and Excitation: <https://arxiv.org/pdf/1709.01507.pdf>
  
  While this is one of the strengths of CNNs, since it drastically reduces the number of parameters of a network, you can imagine that it may not make sense to treat every part of the image the same regardless of the content. This is what the attention gate is for.
  
  By performing an element-wise multiplication of the output of a CNN layer with a gate tensor (typically clamped to the 0-1 range with a sigmoid) we can effectively down weigh or ignore features in certain areas of the image.
  
  CNNs without AG can typically learn the same things but may need more channels and layers. Attention gates allow to treat filters differently depending on the content and hence make it easier to learn with less filters.
  
  What you use as the attention tensor can vary. In the paper you referenced, the tensor was one of the input tensors of the convolutional layer. In this case, it is called self-attention. However, with attention in general this gate tensor may also come from other information such as another network.
  
  > **Comments:**
  > **username_2**: "By performing an element-wise multiplication of the output of a CNN layer with a gate tensor (typically clamped to the 0-1 range with a sigmoid) we can effectively down weigh or ignore features in certain areas of the image."  Isn't this similar to having a CNN layer followed by a fully connected one with sigmoid activation? The only difference is that we're performing element wise rather than tensor multiplication  
  
  ---
  
  