Answers.Body: <p>You can give the entire modified inputs (q,k,v) to each of the heads. But to make it computationally faster, you make the modified inputs' sizes as num_heads * q,k,v and split them. It is the same thing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-21T02:30:25.817Z
Answers.Text: What does the author mean by 768/12? Sorry, I am getting a little confused here.
Answers.UserDisplayName: desert_ranger
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-21T08:35:03.880Z
Answers.Text: Take a look at this image https://github.com/markriedl/transformer-walkthrough/blob/main/transformer.png (large image), see the encoder part first. You will understand the concept.
Answers.UserDisplayName: p0p4k
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-20T08:21:00.780Z
Answers.LastActivityDate: 2022-12-20T08:21:00.780Z
Answers.UserDisplayName: p0p4k
Body: <p>I am reading the book: Natural Language Processing with Transformers. It has the following paragraph</p>
  <blockquote>
  <p>Although head_dim does not have to be smaller than the number of embedding dimensions of the tokens (embed_dim), in practice it is chosen to be a multiple of embed_dim so that the computation across each head is constant. For example, BERT has 12 attention heads, so the dimension of each head is 768/12=64.</p>
  </blockquote>
  <p>While learning transformers, I tried to draw an analogy between CNN filters and multi-headed attention. For instance, increasing the number of filters helps learn different image features, while increasing number of heads help better understand the semantics of a sentence. However, it seems like that the input to the transformer (after being converted to embeddings) is being divided across heads. Perhaps my understanding of multi-head attention is incorrect.</p>
  <p>Basically, I want to know why the author is dividing the inputs across the heads rather than feeding all the inputs to it.</p>
  
Tags: <deep-learning><natural-language-processing><transformer><attention><hyper-parameters>
Title: What's the relationship between number of heads and embedding dimension in Transformers?
UserDisplayName: desert_ranger
ThreadText: # What's the relationship between number of heads and embedding dimension in Transformers?
  
  (Asked by: username_0 on 2022-12-19)
  
  I am reading the book: Natural Language Processing with Transformers. It has the following paragraph
  
  > Although head\_dim does not have to be smaller than the number of embedding dimensions of the tokens (embed\_dim), in practice it is chosen to be a multiple of embed\_dim so that the computation across each head is constant. For example, BERT has 12 attention heads, so the dimension of each head is 768/12=64.
  
  While learning transformers, I tried to draw an analogy between CNN filters and multi-headed attention. For instance, increasing the number of filters helps learn different image features, while increasing number of heads help better understand the semantics of a sentence. However, it seems like that the input to the transformer (after being converted to embeddings) is being divided across heads. Perhaps my understanding of multi-head attention is incorrect.
  
  Basically, I want to know why the author is dividing the inputs across the heads rather than feeding all the inputs to it.
  
  **Tags:** deep-learning, natural-language-processing, transformer, attention, hyper-parameters
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-12-20. Score: 1
  
  You can give the entire modified inputs (q,k,v) to each of the heads. But to make it computationally faster, you make the modified inputs' sizes as num\_heads \* q,k,v and split them. It is the same thing.
  
  > **Comments:**
  > **username_0**: What does the author mean by 768/12? Sorry, I am getting a little confused here.  
  > **username_1**: Take a look at this image https://github.com/markriedl/transformer-walkthrough/blob/main/transformer.png (large image), see the encoder part first. You will understand the concept.  
  
  ---
  
  