Answers.Body: <blockquote>
  <p>Shouldn't the variable size of this output matrix cause some issues down the line through the feed forward parts of the transformer?</p>
  </blockquote>
  <p>Variable lengths wouldn't affect the feed-forward layers of the transformer because, for each layer, the same feed-forward network is applied to all positions in the sequence. Increasing/decreasing the length would just mean that you'd have to run the FF network more/less times per sequence.</p>
  <blockquote>
  <p>Also even if mathematically everything works out, how does the algorithm learn how to work with variable input sizes</p>
  </blockquote>
  <p>During pre-training, models may see shorter sequences (e.g., <a href="https://arxiv.org/pdf/2107.02027" rel="nofollow noreferrer">see the distribution of sequence lengths for the pretraining of BERT in Figure 1</a>). But, even if, during pre-training, models are only presented with long-sequences, I would guess that performance on shorter sequences would still be OK because, in many cases, predicting the next-token/a masked token only requires information from near-by tokens anyway.</p>
  <blockquote>
  <p>and does it need to explicitly told how many tokens it is handling at a time?</p>
  </blockquote>
  <p>It's not really explicitly told, but it could probably infer the sequence length from the position embeddings.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-26T23:05:46.097Z
Answers.LastActivityDate: 2024-10-26T23:05:46.097Z
Answers.UserDisplayName: Alexander Wan
Answers.Body: <p>A single attention head's output has N rows and (H/A) columns, where N = number of input tokens, H = hidden size, and A = number of attention heads. This is because Q, K, and V are matrices of size N x (H/A). The outputs from all attention heads are then concatenated together to get a matrix of size N x H, and transformed via a learned linear layer with H*H + 1 parameters (including bias) parameters to get an output of size N x H, the output of a multi-headed attention layer (MHAL). In BERT base, N (including [CLS], [SEP] and any [PAD] tokens that are ignored) = 512, H = 768 A = 12, and H/A = 64.</p>
  <p>The feed-forward layer that comes after an MHAL and layer normalization will then get a matrix of size N x H. The feed-forward layer will then process each row (token) of this matrix independently, to produce an output of size N x H.</p>
  <p>You can think of matrix multiplications in BERT as computations per token. In theory, N can be arbitrarily large. But BERT base implementation limited this to 512, as a tradeoff between computational speed and context size.</p>
  <p>The illustrations <a href="https://medium.com/analytics-vidhya/understanding-bert-architecture-3f35a264b187" rel="nofollow noreferrer">here</a> and equations <a href="https://johnthickstun.com/docs/transformers.pdf" rel="nofollow noreferrer">here</a> can help.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-06T23:02:41.907Z
Answers.LastActivityDate: 2024-12-06T23:02:41.907Z
Answers.UserDisplayName: Fijoy Vadakkumpadan
Answers.Body: <p>Transformer-based models have fixed size input; this is what we called the context length. You are confusing it with RNN, which can take variable-size input.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T13:14:26.107Z
Answers.Text: As far as I'm aware modern transformers such as Llama do not need to pad to max context length.
  
  Yes there is a maximum but they can work with smaller inputs without any problem.
Answers.UserDisplayName: ILikeMusic
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-26T08:16:13.177Z
Answers.Text: @ILikeMusic, you are right.
Answers.UserDisplayName: vishal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T06:31:47.510Z
Answers.LastActivityDate: 2024-10-24T06:31:47.510Z
Answers.UserDisplayName: vishal
Body: <p>My question is with regards to a single input of arbitrary length, I understand that if you batch you do things like padding. But from everything I've read if batch size equals 1 then the length of the input doesn't matter as long as it is below the maximum size, no padding needed.</p>
  <p>When looking at output of a single self attention head, while the weights themselfs are independent of the input length. The output of this algorithm</p>
  <p><span class="math-container">$$\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^T}{\sqrt{d_k}})V$$</span></p>
  <p>is a matrix with dimensions <span class="math-container">$L,d_V$</span> with <span class="math-container">$L$</span> being the input token length and <span class="math-container">$d_V$</span> one dimension of the value matrix. Shouldn't the variable size of this output matrix cause some issues down the line through the feed forward parts of the transformer? Also even if mathematically everything works out, how does the algorithm learn how to work with variable input sizes, and does it need to explicitly told how many tokens it is handling at a time?</p>
  <p>If it makes a difference I'd be most interested in how this works for newer Llama models should there be any differences compared to older ones like BERT etc.</p>
  <p>Thank you for your time answering this question.</p>
  
Tags: <neural-networks><transformer><math>
Title: Transformer architecture and length of input
UserDisplayName: ILikeMusic
ThreadText: # Transformer architecture and length of input
  
  (Asked by: username_0 on 2024-10-23)
  
  My question is with regards to a single input of arbitrary length, I understand that if you batch you do things like padding. But from everything I've read if batch size equals 1 then the length of the input doesn't matter as long as it is below the maximum size, no padding needed.
  
  When looking at output of a single self attention head, while the weights themselfs are independent of the input length. The output of this algorithm
  
  $$\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^T}{\sqrt{d\_k}})V$$
  
  is a matrix with dimensions $L,d\_V$ with $L$ being the input token length and $d\_V$ one dimension of the value matrix. Shouldn't the variable size of this output matrix cause some issues down the line through the feed forward parts of the transformer? Also even if mathematically everything works out, how does the algorithm learn how to work with variable input sizes, and does it need to explicitly told how many tokens it is handling at a time?
  
  If it makes a difference I'd be most interested in how this works for newer Llama models should there be any differences compared to older ones like BERT etc.
  
  Thank you for your time answering this question.
  
  **Tags:** neural-networks, transformer, math
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2024-10-26. Score: 2
  
  > Shouldn't the variable size of this output matrix cause some issues down the line through the feed forward parts of the transformer?
  
  Variable lengths wouldn't affect the feed-forward layers of the transformer because, for each layer, the same feed-forward network is applied to all positions in the sequence. Increasing/decreasing the length would just mean that you'd have to run the FF network more/less times per sequence.
  
  > Also even if mathematically everything works out, how does the algorithm learn how to work with variable input sizes
  
  During pre-training, models may see shorter sequences (e.g., [see the distribution of sequence lengths for the pretraining of BERT in Figure 1](https://arxiv.org/pdf/2107.02027)). But, even if, during pre-training, models are only presented with long-sequences, I would guess that performance on shorter sequences would still be OK because, in many cases, predicting the next-token/a masked token only requires information from near-by tokens anyway.
  
  > and does it need to explicitly told how many tokens it is handling at a time?
  
  It's not really explicitly told, but it could probably infer the sequence length from the position embeddings.
  
  
  ---
  
  ### Answer by username_2 on 2024-12-06. Score: 0
  
  A single attention head's output has N rows and (H/A) columns, where N = number of input tokens, H = hidden size, and A = number of attention heads. This is because Q, K, and V are matrices of size N x (H/A). The outputs from all attention heads are then concatenated together to get a matrix of size N x H, and transformed via a learned linear layer with H\*H + 1 parameters (including bias) parameters to get an output of size N x H, the output of a multi-headed attention layer (MHAL). In BERT base, N (including [CLS], [SEP] and any [PAD] tokens that are ignored) = 512, H = 768 A = 12, and H/A = 64.
  
  The feed-forward layer that comes after an MHAL and layer normalization will then get a matrix of size N x H. The feed-forward layer will then process each row (token) of this matrix independently, to produce an output of size N x H.
  
  You can think of matrix multiplications in BERT as computations per token. In theory, N can be arbitrarily large. But BERT base implementation limited this to 512, as a tradeoff between computational speed and context size.
  
  The illustrations [here](https://medium.com/analytics-vidhya/understanding-bert-architecture-3f35a264b187) and equations [here](https://johnthickstun.com/docs/transformers.pdf) can help.
  
  
  ---
  
  ### Answer by username_3 on 2024-10-24. Score: 0
  
  Transformer-based models have fixed size input; this is what we called the context length. You are confusing it with RNN, which can take variable-size input.
  
  > **Comments:**
  > **username_0**: As far as I'm aware modern transformers such as Llama do not need to pad to max context length.  Yes there is a maximum but they can work with smaller inputs without any problem.  
  > **username_3**: @ILikeMusic, you are right.  
  
  ---
  
  