Answers.Body: <p>The dense layers do not include interaction between different timesteps. As such, each one is just 4096x4096, and every one of the 2048 4096-dimensional vectors is multiplied by this. (Some of the matrices are rectangular instead of square, but the important part is that there aren't parameters for the full 2048 x whatever input).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T07:58:47.270Z
Answers.Text: Im not sure I understand. At every  timestep, meaning new token being generated, the whole context is considered. And at some point, the whole context must be passed from the attention layer to the dense layers, which includes each variation each of the 32 attention heads generates. Or does it not work that way for LLMs?
Answers.UserDisplayName: user2741831
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T08:02:37.453Z
Answers.Text: For a single attention head in LLaMA it works like this:
  1. Each 4096-dim vector is projected to 3 128-dimensional vectors, the key, query, and value. (All of the reference to parameters is here)
  2. The keys, queries, and values for all positions are used as inputs to the attention function. All the interaction between timesteps happens here, and it doesn't depend on parameters other than the projections into key/query/value.
Answers.UserDisplayName: Davis Yoshida
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T08:08:25.217Z
Answers.Text: By 4096-dim vector, to you mean the embedding size or the context length? Because I'm pretty sure Key Query and Values are supposed to be matrices (representing the input text). But you are saying the get dimensionally reduced to a smaller dimension? At 128x128x32x4096 that would still be 2B Weights just to feed the attention output to the Dense Layers
Answers.UserDisplayName: user2741831
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T09:05:23.047Z
Answers.Text: For a single attention head, K/Q/V are all `sequence_length x head_size` matrices yeah. Each column is a single vector corresponding to a particular timestep. The important thing to understand is that every single linear layer in a transformer is applied to each timestep independently. I'm not sure where you got 128x128x32x4096, but the first dense layer is something like `4096 x 11000`. The output of attention is `sequence_length x 4096` and applying that `4096 x 11000ish` matrix outputs a `sequence_length x 11000ish` matrix (but really you should think of it as a sequence of vectors).
Answers.UserDisplayName: Davis Yoshida
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T09:38:20.297Z
Answers.Text: I assumed the attention heads were calculated in parallel and then fed into a single set of Dense layers. But they are actually stacked on top of each other, so now it makes more sense to me
Answers.UserDisplayName: user2741831
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T06:57:41.553Z
Answers.LastActivityDate: 2023-05-26T06:57:41.553Z
Answers.UserDisplayName: Davis Yoshida
Answers.Body: <p>Let's start over and count the number of parameters, looking at the code in the same repository:</p>
  <ul>
  <li><p>The Attention layer is defined <a href="https://github.com/facebookresearch/llama/blob/a81fb4e211d6656854a5dd24cf8631dc319234d1/llama/model.py#L76" rel="nofollow noreferrer">here in <code>llama/model.py</code></a>, it defines four matrices <code>wq</code>, <code>wk</code>, <code>wv</code> and <code>wo</code>. Each matrix is <code>dim</code> <span class="math-container">$\times$</span> <code>n_heads * head_dim</code>. And <code>head_dim</code> is computed as <code>head_dim = dim // n_heads</code>.</p>
  </li>
  <li><p>The FeedForward bit is defined <a href="https://github.com/facebookresearch/llama/blob/a81fb4e211d6656854a5dd24cf8631dc319234d1/llama/model.py#L153" rel="nofollow noreferrer">here in <code>llama/model.py</code></a>, it defines three matrices  <code>w1</code>, <code>w2</code>, <code>w3</code>.  Each matrix is <code>dim</code> <span class="math-container">$\times$</span> <code>hidden_dim</code>. And <code>hidden_dim</code> is computed as:</p>
  <pre><code>  hidden_dim = 4 * dim    # Call on line 186
    hidden_dim = int(2 * hidden_dim / 3)
    hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of) 
  </code></pre>
  </li>
  </ul>
  <p>Now, <code>dim</code> is the embedding dimension - for 7B model it is 4096 (not 786).<br />
  Number of heads <code>n_heads</code> is 32.
  And <code>multiple_of</code> is some kind of align-to-power-of-2 factor, equals to 256.</p>
  <p>Gathering the whole thing:</p>
  <pre><code>dim = 4096
  n_heads = 32
  multiple_of = 256
  n_blocks = 32
  
  head_dim = dim // n_heads
  n_attention_weights = 4 * dim * (n_heads * head_dim)
  
  hidden_dim = 4 * dim 
  hidden_dim = int(2 * hidden_dim / 3)
  hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)
  n_feed_forward_weights = 3 * dim * hidden_dim
  
  total_weights = n_blocks * (n_attention_weights + n_feed_forward_weights)
  </code></pre>
  <p>I'm getting <code>total_weights = 6 476 005 376</code>.</p>
  <p>There are also input and output embeddings that add <code>2 * vocab_size * dim</code> getting me to 6.9 billon for <code>vocab_size = 50000</code>.</p>
  <p>Note that max context size does not enter the computation at all.</p>
  <p>I'll leave computing the number of weights for larger models as an exercise.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-02T10:46:59.477Z
Answers.Text: If a single K,Q or V matrix has head_dim*head_dim weights, shouldn't the formula for n_attention_weights be n_blocks x (n_heads x (4 x (head_dim x head_dim)))? Also what is wo matrix meant to be?
Answers.UserDisplayName: user2741831
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T12:48:55.230Z
Answers.LastActivityDate: 2023-05-26T12:48:55.230Z
Answers.UserDisplayName: Kostya
Body: <p>I have had a look at LLamas model card, specifically the 7B parameter version:
  <a href="https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md" rel="nofollow noreferrer">https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md</a></p>
  <p>which I assume is an encoder only transformer similar to this:</p>
  <p><img src="https://i.sstatic.net/Kb8Gq.png" alt="" /></p>
  <p>But then I did some math.If the dimension of every Dense layer, including the one connecting to the Attention layer is 4096, the context length is 2048, the number of attention heads is 32 and the embedding size is 786, then the output size of the attention layer is 32 * 786 * 2048 and as such the number of weights to connect it to the dense layer is 32 * 766 * 2048 * 4096, which is 205B parameters, which is obviously far more than 7B. So how is this accomplished? How big is the ouput of the attention layer and how is it connected to the following Dense layers?</p>
  
Tags: <attention><large-language-models>
Title: How is a parameter explosion prevented, when connecting a mutlihead attention layer with the dense layers in LLMs (speciafially, LLama)?
UserDisplayName: user2741831
ThreadText: # How is a parameter explosion prevented, when connecting a mutlihead attention layer with the dense layers in LLMs (speciafially, LLama)?
  
  (Asked by: username_0 on 2023-05-26)
  
  I have had a look at LLamas model card, specifically the 7B parameter version:
  <https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md>
  
  which I assume is an encoder only transformer similar to this:
  
  ![](https://i.sstatic.net/Kb8Gq.png)
  
  But then I did some math.If the dimension of every Dense layer, including the one connecting to the Attention layer is 4096, the context length is 2048, the number of attention heads is 32 and the embedding size is 786, then the output size of the attention layer is 32 \* 786 \* 2048 and as such the number of weights to connect it to the dense layer is 32 \* 766 \* 2048 \* 4096, which is 205B parameters, which is obviously far more than 7B. So how is this accomplished? How big is the ouput of the attention layer and how is it connected to the following Dense layers?
  
  **Tags:** attention, large-language-models
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-05-26. Score: 3 | Accepted answer
  
  Let's start over and count the number of parameters, looking at the code in the same repository:
  
  The Attention layer is defined here in llama/model.py, it defines four matrices wq, wk, wv and wo. Each matrix is dim $\times$ n_heads * head_dim. And head_dim is computed as head_dim = dim // n_heads.
  
  The FeedForward bit is defined here in llama/model.py, it defines three matrices  w1, w2, w3.  Each matrix is dim $\times$ hidden_dim. And hidden_dim is computed as:
    hidden_dim = 4 * dim    # Call on line 186
    hidden_dim = int(2 * hidden_dim / 3)
    hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of) 
  
  
  
  Now, dim is the embedding dimension - for 7B model it is 4096 (not 786).
  Number of heads n_heads is 32.
  And multiple_of is some kind of align-to-power-of-2 factor, equals to 256.
  Gathering the whole thing:
  dim = 4096
  n_heads = 32
  multiple_of = 256
  n_blocks = 32
  
  head_dim = dim // n_heads
  n_attention_weights = 4 * dim * (n_heads * head_dim)
  
  hidden_dim = 4 * dim 
  hidden_dim = int(2 * hidden_dim / 3)
  hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)
  n_feed_forward_weights = 3 * dim * hidden_dim
  
  total_weights = n_blocks * (n_attention_weights + n_feed_forward_weights)
  
  I'm getting total_weights = 6 476 005 376.
  There are also input and output embeddings that add 2 * vocab_size * dim getting me to 6.9 billon for vocab_size = 50000.
  Note that max context size does not enter the computation at all.
  I'll leave computing the number of weights for larger models as an exercise.
  
  > **Comments:**
  > **username_0**: If a single K,Q or V matrix has head_dim*head_dim weights, shouldn't the formula for n_attention_weights be n_blocks x (n_heads x (4 x (head_dim x head_dim)))? Also what is wo matrix meant to be?  
  
  ---
  
  ### Answer by username_1 on 2023-05-26. Score: 2
  
  The dense layers do not include interaction between different timesteps. As such, each one is just 4096x4096, and every one of the 2048 4096-dimensional vectors is multiplied by this. (Some of the matrices are rectangular instead of square, but the important part is that there aren't parameters for the full 2048 x whatever input).
  
  > **Comments:**
  > **username_0**: Im not sure I understand. At every  timestep, meaning new token being generated, the whole context is considered. And at some point, the whole context must be passed from the attention layer to the dense layers, which includes each variation each of the 32 attention heads generates. Or does it not work that way for LLMs?  
  > **username_1**: For a single attention head in LLaMA it works like this: 1. Each 4096-dim vector is projected to 3 128-dimensional vectors, the key, query, and value. (All of the reference to parameters is here) 2. The keys, queries, and values for all positions are used as inputs to the attention function. All the interaction between timesteps happens here, and it doesn't depend on parameters other than the projections into key/query/value.  
  > **username_0**: By 4096-dim vector, to you mean the embedding size or the context length? Because I'm pretty sure Key Query and Values are supposed to be matrices (representing the input text). But you are saying the get dimensionally reduced to a smaller dimension? At 128x128x32x4096 that would still be 2B Weights just to feed the attention output to the Dense Layers  
  > **username_1**: For a single attention head, K/Q/V are all `sequence_length x head_size` matrices yeah. Each column is a single vector corresponding to a particular timestep. The important thing to understand is that every single linear layer in a transformer is applied to each timestep independently. I'm not sure where you got 128x128x32x4096, but the first dense layer is something like `4096 x 11000`. The output of attention is `sequence_length x 4096` and applying that `4096 x 11000ish` matrix outputs a `sequence_length x 11000ish` matrix (but really you should think of it as a sequence of vectors).  
  > **username_0**: I assumed the attention heads were calculated in parallel and then fed into a single set of Dense layers. But they are actually stacked on top of each other, so now it makes more sense to me  
  
  ---
  
  