Answers.Body: <p>Your intuition is right. For each token in the sequence, GPT-2 linearly projects token’s embedding into three different Q/K/V spaces using three separate linear layers, each of which has weights and bias. The bias terms allow the model to adjust the Q/K/V vector outputs of the linear projections via weight matrices to become useful for learning more complex patterns.</p>
  <p>However, the biases do not directly affect the attention mechanism or softmax operation after the QKV projections are made, they only affect how the initial QKV projections from their earlier embedding layer are computed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-12T06:03:43.427Z
Answers.LastActivityDate: 2024-11-12T06:03:43.427Z
Answers.UserDisplayName: cinch
Body: <p>This seems illogical according to the interpretation we use for QKV (q - query (what we are looking for), k - key(what do i have), v - value (what are my values), softmax(k(X) @ q(x).T) - in what proportions i want to use V). There is no references to nn.Linear.bias</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-11T23:14:18.710Z
Comments.Text: the networks is already over parametrized, some additional biases won't hurt lol
Comments.UserDisplayName: Alberto
Tags: <neural-networks><transformer><gpt>
Title: Why GPT2 uses biases in QKV attention?
UserDisplayName: Тима 
ThreadText: # Why GPT2 uses biases in QKV attention?
  
  (Asked by: username_0 on 2024-11-11)
  
  This seems illogical according to the interpretation we use for QKV (q - query (what we are looking for), k - key(what do i have), v - value (what are my values), softmax(k(X) @ q(x).T) - in what proportions i want to use V). There is no references to nn.Linear.bias
  
  **Tags:** neural-networks, transformer, gpt
  > **Comments:**
  > **username_1**: the networks is already over parametrized, some additional biases won't hurt lol  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-11-12. Score: 1 | Accepted answer
  
  Your intuition is right. For each token in the sequence, GPT-2 linearly projects token’s embedding into three different Q/K/V spaces using three separate linear layers, each of which has weights and bias. The bias terms allow the model to adjust the Q/K/V vector outputs of the linear projections via weight matrices to become useful for learning more complex patterns.
  
  However, the biases do not directly affect the attention mechanism or softmax operation after the QKV projections are made, they only affect how the initial QKV projections from their earlier embedding layer are computed.
  
  
  ---
  
  