Answers.Body: <p>I suppose the main reason is that they were proposed as a more parallelizable alternative to RNNs that is quicker to compute. Hence, making them just big matrix multiplications achieves this and the lack of intermediate activations is just replaced by the sheer number of transformers that focus their attention on different parts of the input sequence.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T13:22:51.897Z
Answers.LastActivityDate: 2022-06-16T13:22:51.897Z
Answers.UserDisplayName: ImotVoksim
Body: <p>Transformers use just matrices to transform input embeddings, which is halfway to being a connected dense layer (add a bias and activation). So, why don't transformers have dense layers for encoding input into Query Key Value?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T14:11:01.750Z
Comments.Text: The following is about the same question and provides a suitable answer: https://ai.stackexchange.com/questions/40252/why-are-biases-typically-not-used-in-attention-mechanism/40256#40256
Comments.UserDisplayName: Robin van Hoorn
Tags: <transformer><dense-layers>
Title: Why do transformer Key Query Value layers not have biases or activations?
UserDisplayName: Artin Kim
ThreadText: # Why do transformer Key Query Value layers not have biases or activations?
  
  (Asked by: username_0 on 2022-06-15)
  
  Transformers use just matrices to transform input embeddings, which is halfway to being a connected dense layer (add a bias and activation). So, why don't transformers have dense layers for encoding input into Query Key Value?
  
  **Tags:** transformer, dense-layers
  > **Comments:**
  > **username_1**: The following is about the same question and provides a suitable answer: https://ai.stackexchange.com/questions/40252/why-are-biases-typically-not-used-in-attention-mechanism/40256#40256  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-16. Score: -1
  
  I suppose the main reason is that they were proposed as a more parallelizable alternative to RNNs that is quicker to compute. Hence, making them just big matrix multiplications achieves this and the lack of intermediate activations is just replaced by the sheer number of transformers that focus their attention on different parts of the input sequence.
  
  
  ---
  
  