Answers.Body: <p>I assume you're talking about this design: (<a href="https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html" rel="nofollow noreferrer">image source</a>)
  <a href="https://i.sstatic.net/U1ATS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/U1ATS.png" alt="SAGAN attention network" /></a></p>
  <blockquote>
  <p>But wouldn't applying two sets of 1x1 convs to the pixel lead to the chance that the pixel would undergo a different transformation and in the end would have a lower correlation than it should?</p>
  </blockquote>
  <p>Yes, that's the point. We are not trying to measure a pixel's correlation with itself. Rather we are trying to allow it to query different related data. We are giving it freedom to change both the data and the queries.</p>
  <p>It is true that the space for the queries and the keys is the same - but we shouldn't use the same transformation for both, or else each instance of the attention layer is just trying to fetch its own value! Generally the purpose of an attention layer is to query <em>different</em> parts of the input.</p>
  <p>The first half of your question was essentially &quot;why do we have a convolution at all?&quot; and I think this has the same answer: you'd just be able to detect similar pixels, you wouldn't be able to pay attention to noses whenever eyes are detected.</p>
  <p>It is also true that you could probably skip the convolution on the h(x) input. It looks like this one is somewhat redundant because the convolutions on h(x) and v(x) apply in series - which makes it a two-layer convolution, not quite the same as a one-layer convolution, but perhaps only one layer is needed.</p>
  <p>It is possible that if you removed the conv layer on either the keys or the queries (but not both) the model would learn to generate the keys directly as the features, but this would hinder it because it would be unable to output any data in the values and queries that wasn't part of the keys (or vice versa). Seems silly. Don't do that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T15:03:26.160Z
Answers.LastActivityDate: 2021-08-03T15:03:26.160Z
Answers.UserDisplayName: Criticize SE actions means ban
Body: <p>I have two questions about the structure of attention modules:</p>
  <p>Since I work with imagery I will be talking about using convolutions on feature maps in order to obtain attention maps.</p>
  <ol>
  <li><p>If we have a set of feature maps with dimensions [B, C, H, W] (batch, channel, height, width), why do we transform our feature maps before we calculate their affinity/correlation in attention mechanisms? What makes this better than simply taking the cosine distance between the feature vectors (e.g. resizing the maps to [B, C, HW] and [B, HW, C] and multiplying them together). Aren't the feature maps already in an appropriate feature/embedding space that we can just use them directly instead of transforming them first?</p>
  </li>
  <li><p>Most of the time, attention mechanisms will take as input some stack of feature maps (F), and will apply 3 transformations on them to essentially produce a &quot;query&quot;, &quot;key&quot; and &quot;value&quot;. The query and key will be multiplied together to get the affinity/correlation between a given feature vector and all other feature vectors. In computer vision these transformation will typically be performed by the different 1x1 convolutions. My question is, how come we use 3 different 1x1 convolutions? Wouldn't it make more sense to apply the same 1x1 convolution to the input F? My intuition tells me that since we want to transform/project the feature maps F into some embedding/feature space that it would make the most sense if the &quot;query&quot;, &quot;key&quot; and &quot;value&quot; were all obtained by using the same transformation. To illustrate what I mean lets pretend we had a 1x1 feature map and we wanted to see how well the pixel correlates with itself. Obviously it should correlate 100% because it is the same pixel. But wouldn't applying two sets of 1x1 convs to the pixel lead to the chance that the pixel would undergo a different transformation and in the end would have a lower correlation than it should?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-03T09:53:40.260Z
Comments.Text: Hello. Welcome to AI SE. It seems to me that you're asking multiple questions here. When you have multiple questions, even if they are related, it's better to create multiple posts, one for each question (with the appropriate context to understand each post), so that people can focus on a single question at a time. Your question could be closed as "needs more focus" because you're asking many questions. Maybe the questions are very related. I didn't really/carefully read your post.  See [this](https://ai.stackexchange.com/help/on-topic).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-03T14:27:35.203Z
Comments.Text: I'm also having some trouble understanding exactly what you are trying to ask. Can you possibly describe the model architectures you are asking about a bit more precisely?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-03T14:54:45.630Z
Comments.Text: are you asking about [this architecture](https://lilianweng.github.io/lil-log/assets/images/SAGAN.png)?
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <neural-networks><machine-learning><deep-learning><convolutional-neural-networks><attention>
Title: Attention mechanism: Why apply multiple different transformations to obtain query, key, value
UserDisplayName: Steve Ahlswede
ThreadText: # Attention mechanism: Why apply multiple different transformations to obtain query, key, value
  
  (Asked by: username_0 on 2021-08-03)
  
  I have two questions about the structure of attention modules:
  
  Since I work with imagery I will be talking about using convolutions on feature maps in order to obtain attention maps.
  
  1. If we have a set of feature maps with dimensions [B, C, H, W] (batch, channel, height, width), why do we transform our feature maps before we calculate their affinity/correlation in attention mechanisms? What makes this better than simply taking the cosine distance between the feature vectors (e.g. resizing the maps to [B, C, HW] and [B, HW, C] and multiplying them together). Aren't the feature maps already in an appropriate feature/embedding space that we can just use them directly instead of transforming them first?
  2. Most of the time, attention mechanisms will take as input some stack of feature maps (F), and will apply 3 transformations on them to essentially produce a "query", "key" and "value". The query and key will be multiplied together to get the affinity/correlation between a given feature vector and all other feature vectors. In computer vision these transformation will typically be performed by the different 1x1 convolutions. My question is, how come we use 3 different 1x1 convolutions? Wouldn't it make more sense to apply the same 1x1 convolution to the input F? My intuition tells me that since we want to transform/project the feature maps F into some embedding/feature space that it would make the most sense if the "query", "key" and "value" were all obtained by using the same transformation. To illustrate what I mean lets pretend we had a 1x1 feature map and we wanted to see how well the pixel correlates with itself. Obviously it should correlate 100% because it is the same pixel. But wouldn't applying two sets of 1x1 convs to the pixel lead to the chance that the pixel would undergo a different transformation and in the end would have a lower correlation than it should?
  
  **Tags:** neural-networks, machine-learning, deep-learning, convolutional-neural-networks, attention
  > **Comments:**
  > **username_1**: Hello. Welcome to AI SE. It seems to me that you're asking multiple questions here. When you have multiple questions, even if they are related, it's better to create multiple posts, one for each question (with the appropriate context to understand each post), so that people can focus on a single question at a time. Your question could be closed as "needs more focus" because you're asking many questions. Maybe the questions are very related. I didn't really/carefully read your post.  See [this](https://ai.stackexchange.com/help/on-topic).  
  > **username_2**: I'm also having some trouble understanding exactly what you are trying to ask. Can you possibly describe the model architectures you are asking about a bit more precisely?  
  > **username_2**: are you asking about [this architecture](https://lilianweng.github.io/lil-log/assets/images/SAGAN.png)?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-03. Score: 1
  
  I assume you're talking about this design: ([image source](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html))
  [![SAGAN attention network](https://i.sstatic.net/U1ATS.png)](https://i.sstatic.net/U1ATS.png)
  
  > But wouldn't applying two sets of 1x1 convs to the pixel lead to the chance that the pixel would undergo a different transformation and in the end would have a lower correlation than it should?
  
  Yes, that's the point. We are not trying to measure a pixel's correlation with itself. Rather we are trying to allow it to query different related data. We are giving it freedom to change both the data and the queries.
  
  It is true that the space for the queries and the keys is the same - but we shouldn't use the same transformation for both, or else each instance of the attention layer is just trying to fetch its own value! Generally the purpose of an attention layer is to query *different* parts of the input.
  
  The first half of your question was essentially "why do we have a convolution at all?" and I think this has the same answer: you'd just be able to detect similar pixels, you wouldn't be able to pay attention to noses whenever eyes are detected.
  
  It is also true that you could probably skip the convolution on the h(x) input. It looks like this one is somewhat redundant because the convolutions on h(x) and v(x) apply in series - which makes it a two-layer convolution, not quite the same as a one-layer convolution, but perhaps only one layer is needed.
  
  It is possible that if you removed the conv layer on either the keys or the queries (but not both) the model would learn to generate the keys directly as the features, but this would hinder it because it would be unable to output any data in the values and queries that wasn't part of the keys (or vice versa). Seems silly. Don't do that.
  
  
  ---
  
  