Answers.Body: <p>Decoder only text LLMs are autoregressive. Like any autoregressive function if you do not left pad there will be no prior token to your intial token for efficient gradient descent.
  In most text generation models, the process of token generation is AutoRegressive, which means that the model generates one token at a time, conditioned on the previous tokens it has generated 2.</p>
  <p>In the equation you provided, X = (A, b, c)  X = F(X(-1)), it seems like X is a vector of length 3, and F is a function that takes the previous value of X as input and returns the next value of X. This is a recursive definition of X, where the current value of X depends on the previous value of X. This is similar to the AutoRegressive process used in text generation models, where the current token depends on the previous tokens generated by the model.</p>
  <p>MarvsAI</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-12T06:15:17.623Z
Answers.LastActivityDate: 2023-11-12T06:15:17.623Z
Answers.UserDisplayName: Sam Marvasti
Answers.Body: <p>I got an answer to this question, probably a correct explanation.</p>
  <p>In decoder-only model architectures, the output of the model is a continuation of the model input.</p>
  <p>For example, input: I love apple [pad] [pad]. The output of the model will contain the input and add additional output information.</p>
  <p>For example, output: I love apple [pad] [pad],because it is delicious.</p>
  <p>This would result in [pad] being stuck in the middle of the text. It is very bad for the model to process text. If we use left-padding, the output of this model will be</p>
  <p>output : [pad] [pad] i love apple,because it is delicious.</p>
  <p>Such complete semantic information is continuous.</p>
  <p>ref:<a href="https://github.com/huggingface/transformers/issues/18388#issuecomment-1204369688" rel="noreferrer">https://github.com/huggingface/transformers/issues/18388#issuecomment-1204369688</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-13T02:10:04.007Z
Answers.Text: I don't think this a complete answer. We could simply take the output probabilities correponding to the input "apple" and overwrite the first pad token with the new token. I think the real answer is just that it's slightly more convenient not to have to worry about this by leftpadding everything
Answers.UserDisplayName: johannesack
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-30T05:23:30.257Z
Answers.LastActivityDate: 2023-07-30T05:23:30.257Z
Answers.UserDisplayName: 尹雅博
Body: <p>While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?</p>
  <p>Many papers use Left Padding, but is right padding wrong since transformers gives the following warning if using right padding &quot; A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer.&quot;</p>
  <p>The attention mask will anyways ignore the padding tokens.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T10:08:58.440Z
Comments.Text: Next time, please, put your question in the title. Thanks.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-29T11:13:36.060Z
Comments.Text: I'm concerned about this as well, and the generate() method in the transformers library explicitly suggests that decoder-only models should use the left padding method. Is there some correlation? I would also like to know the reason for this left padding.
  [![enter image description here](https://i.sstatic.net/Qr63L.png)](https://i.sstatic.net/Qr63L.png)
Comments.UserDisplayName: 尹雅博
Tags: <transformer><pytorch><padding>
Title: While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?
UserDisplayName: basujindal
ThreadText: # While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?
  
  (Asked by: username_0 on 2023-07-26)
  
  While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?
  
  Many papers use Left Padding, but is right padding wrong since transformers gives the following warning if using right padding " A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding\_side='left' when initializing the tokenizer."
  
  The attention mask will anyways ignore the padding tokens.
  
  **Tags:** transformer, pytorch, padding
  
  > **Comments:**
  > **username_1**: Next time, please, put your question in the title. Thanks.  
  > **username_2**: I'm concerned about this as well, and the generate() method in the transformers library explicitly suggests that decoder-only models should use the left padding method. Is there some correlation? I would also like to know the reason for this left padding. [![enter image description here](https://i.sstatic.net/Qr63L.png)](https://i.sstatic.net/Qr63L.png)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-07-30. Score: 9 | Accepted answer
  
  I got an answer to this question, probably a correct explanation.
  
  In decoder-only model architectures, the output of the model is a continuation of the model input.
  
  For example, input: I love apple [pad] [pad]. The output of the model will contain the input and add additional output information.
  
  For example, output: I love apple [pad] [pad],because it is delicious.
  
  This would result in [pad] being stuck in the middle of the text. It is very bad for the model to process text. If we use left-padding, the output of this model will be
  
  output : [pad] [pad] i love apple,because it is delicious.
  
  Such complete semantic information is continuous.
  
  ref:<https://github.com/huggingface/transformers/issues/18388#issuecomment-1204369688>
  
  > **Comments:**
  > **username_4**: I don't think this a complete answer. We could simply take the output probabilities correponding to the input "apple" and overwrite the first pad token with the new token. I think the real answer is just that it's slightly more convenient not to have to worry about this by leftpadding everything  
  
  ---
  
  ### Answer by username_3 on 2023-11-12. Score: 1
  
  Decoder only text LLMs are autoregressive. Like any autoregressive function if you do not left pad there will be no prior token to your intial token for efficient gradient descent.
  In most text generation models, the process of token generation is AutoRegressive, which means that the model generates one token at a time, conditioned on the previous tokens it has generated 2.
  
  In the equation you provided, X = (A, b, c) X = F(X(-1)), it seems like X is a vector of length 3, and F is a function that takes the previous value of X as input and returns the next value of X. This is a recursive definition of X, where the current value of X depends on the previous value of X. This is similar to the AutoRegressive process used in text generation models, where the current token depends on the previous tokens generated by the model.
  
  MarvsAI
  
  
  ---
  
  