Answers.Body: <p>This performs <span class="math-container">$h$</span> different attention lookups in parallel. The amount of computation is kept the same, though, so each one is smaller. They found that multiple smaller lookups were more useful than one big lookup.</p>
  <p>The matrices are not just split, projected and concatenated. They are split, projected, <strong>the attention lookup is performed on each set of pieces</strong> and then the results of the attention function are concatenated.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-31T21:35:54.070Z
Answers.LastActivityDate: 2023-03-31T21:35:54.070Z
Answers.UserDisplayName: Criticize SE actions means ban
Body: <p>In the original transformer paper &quot;Attention is all you need&quot; in section 3.2.2 it is written:</p>
  <p>Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly <strong>project the queries, keys and values h times</strong> with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are <strong>concatenated and once again projected</strong>, resulting in the final values, as depicted in Figure 2.</p>
  <p>I am wondering why you need to project the values h times if you concatenate and project them once again in the end. It seems to me that this just results in two matrices being learned that are multiplied with each other. This should have the same expressiveness as one matrix. So, in my understanding you could just leave away the h projection steps and simply do the final projection. Am I missing something?</p>
  
Tags: <transformer>
Title: Redundancy of Value-Projection-matrix in Multi-headed attention in Transformer model
UserDisplayName: jonaden
ThreadText: # Redundancy of Value-Projection-matrix in Multi-headed attention in Transformer model
  
  (Asked by: username_0 on 2023-03-31)
  
  In the original transformer paper "Attention is all you need" in section 3.2.2 it is written:
  
  Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly **project the queries, keys and values h times** with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are **concatenated and once again projected**, resulting in the final values, as depicted in Figure 2.
  
  I am wondering why you need to project the values h times if you concatenate and project them once again in the end. It seems to me that this just results in two matrices being learned that are multiplied with each other. This should have the same expressiveness as one matrix. So, in my understanding you could just leave away the h projection steps and simply do the final projection. Am I missing something?
  
  **Tags:** transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-31. Score: 1
  
  This performs $h$ different attention lookups in parallel. The amount of computation is kept the same, though, so each one is smaller. They found that multiple smaller lookups were more useful than one big lookup.
  
  The matrices are not just split, projected and concatenated. They are split, projected, **the attention lookup is performed on each set of pieces** and then the results of the attention function are concatenated.
  
  
  ---
  
  