Answers.Body: <ol>
  <li>You can implement the Transformer architecture such that you can indeed change the block size. It can be made smaller and larger. It should be noted, however, that there is no guarantee on performance if you change the max-sequence length. I have no clue how much performance you are going to lose by changing it.</li>
  <li>Not too much in the transformer architecture is based on the actual max-sequence length, and all of it can be implemented such that it can be changed. The only things in my own implementation which are based on the block size are the positional embedding (before the actual blocks) and the masking buffer. The positional embedding is done in the original paper using sine/cosine embeddings which can be generalized to any length. The masking buffer can simply be generated on the spot if you would want that. The rest is not based on the actual max-sequence length and is simply done using the size of the batch and matrix multiplications.</li>
  </ol>
  <ol start="3">
  <li>The transformer is <em>not</em> a recurrent neural network. It does not need a hidden/carry state from the previous computation to continue with the next computation. It simply needs to context (often called prompt) and does all the calculations based on that context. So the whole context is fed into the network, and out comes the prediction for the next token.</li>
  <li>It does process all tokens at the same time.</li>
  </ol>
  <p>If you have further questions, im happy to answer them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-22T17:12:59.503Z
Answers.Text: I din't understand few things, 1.)  like how a single transformer is able to manage multiple variable input lengths, like after attention mechanism in original transformer paper there is Fully Connected Neural Network right, how is variable length is getting compatible with this FNN?  2.) " The output of each step is fed to the bottom decoder in the next time step, and the decoders bubble up their decoding results just like the encoders did. " This is text from jalammar about transformers, but he mentioned that the transformer process one step at a time
Answers.UserDisplayName: Arjun Reddy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-22T08:39:01.100Z
Answers.LastActivityDate: 2023-04-23T07:25:42.513Z
Answers.LastEditDate: 2023-04-23T07:25:42.513Z
Answers.UserDisplayName: Robin van Hoorn
Answers.Body: <p>I am explicitly elaborating on question 3. I feel the other answers are complete with regard to the other 3 questions posed.</p>
  <p>During Inference, the decoder is auto-regressive. I.E. it has to process the next token one at a time.</p>
  <p>During Training, the decoder is trained using teacher forcing. Since we already have the ground truth label, we can just pass the golden (provided) tokens {x0,x1....xn}. This permits efficient training.</p>
  <p><a href="https://stackoverflow.com/a/58021905/20522929">https://stackoverflow.com/a/58021905/20522929</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-22T20:12:11.860Z
Answers.Text: So, in the training as you mentioned, decoder processes one token at a time right? Then how is decoder is processing all tokens at the same time
Answers.UserDisplayName: Arjun Reddy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-22T22:17:24.330Z
Answers.Text: Teacher forcing is a technique that processes all of the ground truth tokens at once. Hence why the decoder input ids are the labels shifted to the right.
Answers.UserDisplayName: Thomas K
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-23T00:25:45.887Z
Answers.Text: I mean "right" in the sense Am I correct? or Am I right?, can you explain how teacher forcing process all tokens at once, because teacher forcing actually works one-token at a time like If the model ground truth is "He is good", Then first we give "He" to model as input and then pass the "is"(As per our ground truth)   in next time step, so that means the transformer has to process one token at a time, Am I correct?
Answers.UserDisplayName: Arjun Reddy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-23T16:18:56.470Z
Answers.Text: By right, I mean left/right. The labels would be "I have a cat </s>" and the decoder inputs would be "<s>I have a cat". You are correct, teacher forcing works one token at a time. But we can run all steps in parallel during training for the decoder because we already have all the known tokens.
Answers.UserDisplayName: Thomas K
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-22T18:29:58.800Z
Answers.LastActivityDate: 2023-04-22T22:16:48.557Z
Answers.LastEditDate: 2023-04-22T22:16:48.557Z
Answers.UserDisplayName: Thomas K
Body: <p>Assume the transformer is trained on 512 max length sentences:</p>
  <ol>
  <li><p>can we fine-tune it on 256 max-length sentences?</p>
  </li>
  <li><p>If we can fine-tune it, how is it even possible because the input shapes are different, and how is the change happening in high-level overview from weights matrices to end layer</p>
  </li>
  <li><p>Does every transformer decoder processes one token at the same time, or does it process all tokens at a time if it doesn't, how the decoder process all tokens at the same time?</p>
  </li>
  <li><p>I have doubt that the process of output is different for Sentiment Analysis and Text generation in Transformer Decoder architecture because in text generation, the decoder process one token at a time, while the sentiment analysis need not be one-token, it can be all tokens at the same time, right? So, how is this difference in both examples, the decoder is able to capture by its architecture, <strong>Does all decoder process one token at the same time or all tokens at same time?, If not how is same architecture is able to capture both examples as mentioned above?</strong></p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-22T15:36:50.247Z
Comments.Text: Please, one question per post, even if the questions are in the same context.
Comments.UserDisplayName: nbro
Tags: <transformer><fine-tuning>
Title: 4 Questions on Transformers
UserDisplayName: Arjun Reddy
ThreadText: # 4 Questions on Transformers
  
  (Asked by: username_0 on 2023-04-21)
  
  Assume the transformer is trained on 512 max length sentences:
  
  1. can we fine-tune it on 256 max-length sentences?
  2. If we can fine-tune it, how is it even possible because the input shapes are different, and how is the change happening in high-level overview from weights matrices to end layer
  3. Does every transformer decoder processes one token at the same time, or does it process all tokens at a time if it doesn't, how the decoder process all tokens at the same time?
  4. I have doubt that the process of output is different for Sentiment Analysis and Text generation in Transformer Decoder architecture because in text generation, the decoder process one token at a time, while the sentiment analysis need not be one-token, it can be all tokens at the same time, right? So, how is this difference in both examples, the decoder is able to capture by its architecture, **Does all decoder process one token at the same time or all tokens at same time?, If not how is same architecture is able to capture both examples as mentioned above?**
  
  **Tags:** transformer, fine-tuning
  
  > **Comments:**
  > **username_1**: Please, one question per post, even if the questions are in the same context.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-04-22. Score: 1
  
  1. You can implement the Transformer architecture such that you can indeed change the block size. It can be made smaller and larger. It should be noted, however, that there is no guarantee on performance if you change the max-sequence length. I have no clue how much performance you are going to lose by changing it.
  2. Not too much in the transformer architecture is based on the actual max-sequence length, and all of it can be implemented such that it can be changed. The only things in my own implementation which are based on the block size are the positional embedding (before the actual blocks) and the masking buffer. The positional embedding is done in the original paper using sine/cosine embeddings which can be generalized to any length. The masking buffer can simply be generated on the spot if you would want that. The rest is not based on the actual max-sequence length and is simply done using the size of the batch and matrix multiplications.
  
  3. The transformer is *not* a recurrent neural network. It does not need a hidden/carry state from the previous computation to continue with the next computation. It simply needs to context (often called prompt) and does all the calculations based on that context. So the whole context is fed into the network, and out comes the prediction for the next token.
  4. It does process all tokens at the same time.
  
  If you have further questions, im happy to answer them.
  
  > **Comments:**
  > **username_0**: I din't understand few things, 1.)  like how a single transformer is able to manage multiple variable input lengths, like after attention mechanism in original transformer paper there is Fully Connected Neural Network right, how is variable length is getting compatible with this FNN?  2.) " The output of each step is fed to the bottom decoder in the next time step, and the decoders bubble up their decoding results just like the encoders did. " This is text from jalammar about transformers, but he mentioned that the transformer process one step at a time  
  
  ---
  
  ### Answer by username_3 on 2023-04-22. Score: 0
  
  I am explicitly elaborating on question 3. I feel the other answers are complete with regard to the other 3 questions posed.
  
  During Inference, the decoder is auto-regressive. I.E. it has to process the next token one at a time.
  
  During Training, the decoder is trained using teacher forcing. Since we already have the ground truth label, we can just pass the golden (provided) tokens {x0,x1....xn}. This permits efficient training.
  
  <https://stackoverflow.com/a/58021905/20522929>
  
  > **Comments:**
  > **username_0**: So, in the training as you mentioned, decoder processes one token at a time right? Then how is decoder is processing all tokens at the same time  
  > **username_3**: Teacher forcing is a technique that processes all of the ground truth tokens at once. Hence why the decoder input ids are the labels shifted to the right.  
  > **username_0**: I mean "right" in the sense Am I correct? or Am I right?, can you explain how teacher forcing process all tokens at once, because teacher forcing actually works one-token at a time like If the model ground truth is "He is good", Then first we give "He" to model as input and then pass the "is"(As per our ground truth)   in next time step, so that means the transformer has to process one token at a time, Am I correct?  
  > **username_3**: By right, I mean left/right. The labels would be "I have a cat </s>" and the decoder inputs would be "<s>I have a cat". You are correct, teacher forcing works one token at a time. But we can run all steps in parallel during training for the decoder because we already have all the known tokens.  
  
  ---
  
  