Answers.Body: <p>If you are training a <strong>decoder-only</strong> model, you don't need any manual augmentation. Recall that a decoder-only model only allows tokens to attend to previous tokens during self-attention. Additionally, decoder-only models will often predict the next token at the position of the next-to-last token. In other words, the model's output representation at position <span class="math-container">$i$</span> (say, <span class="math-container">$h_i$</span>) is used to predict the token <span class="math-container">$t_{i+1}$</span>, with <span class="math-container">$h_i$</span> only depending on tokens <span class="math-container">$t_1, ..., t_{i-1}$</span>.</p>
  <p>So, with a single forward pass, <strong>each</strong> output hidden state <span class="math-container">$h_i$</span> (and subsequent token prediction) constitutes a valid autoregressive prediction (i.e., it predicts the next token based on only the previous values).</p>
  <p>Contrast that with training an <strong>encoder-only</strong> MLM model, like <a href="https://arxiv.org/abs/1810.04805" rel="nofollow noreferrer">BERT</a>, where (part of) the pretraining objective is to predict a masked-out word. This masking can be in the form of a <code>[MASK]</code> token or a randomly chosen token. You have to do <em>some</em> kind of masking as otherwise the model will just learn to repeat the input token (although in the BERT paper, they do leave the token unchanged 10% of the time). To produce these masks, you need to do some preprocessing beforehand, although you can mask multiple tokens (and thus calculate losses for them) in a single forward pass.</p>
  <p><strong>Encoder-decoder models</strong> are often also trained a similar way. E.g., for <a href="https://arxiv.org/abs/1910.10683" rel="nofollow noreferrer">T5</a>, you mask out, then predict consecutive spans of tokens, but your model can predict multiple token spans per forward pass.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-03T06:25:48.383Z
Answers.LastActivityDate: 2023-09-03T22:09:37.193Z
Answers.LastEditDate: 2023-09-03T22:09:37.193Z
Answers.UserDisplayName: Alexander Wan
Body: <p>If my dataset consists of the sentence &quot;I have an apple&quot; do I need to feed my model the separate examples &quot;I&quot; -&gt; &quot;have&quot;, &quot;I have&quot; -&gt; &quot;an&quot;, and &quot;I have an&quot; -&gt; &quot;apple&quot; or is that essentially what the masking does, and I only have to feed in the entire sentence at once?</p>
  
Tags: <transformer>
Title: Do I need to manually segment/augment my data for transformer training?
UserDisplayName: User
ThreadText: # Do I need to manually segment/augment my data for transformer training?
  
  (Asked by: username_0 on 2023-09-02)
  
  If my dataset consists of the sentence "I have an apple" do I need to feed my model the separate examples "I" -> "have", "I have" -> "an", and "I have an" -> "apple" or is that essentially what the masking does, and I only have to feed in the entire sentence at once?
  
  **Tags:** transformer
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-03. Score: 1
  
  If you are training a **decoder-only** model, you don't need any manual augmentation. Recall that a decoder-only model only allows tokens to attend to previous tokens during self-attention. Additionally, decoder-only models will often predict the next token at the position of the next-to-last token. In other words, the model's output representation at position $i$ (say, $h\_i$) is used to predict the token $t\_{i+1}$, with $h\_i$ only depending on tokens $t\_1, ..., t\_{i-1}$.
  
  So, with a single forward pass, **each** output hidden state $h\_i$ (and subsequent token prediction) constitutes a valid autoregressive prediction (i.e., it predicts the next token based on only the previous values).
  
  Contrast that with training an **encoder-only** MLM model, like [BERT](https://arxiv.org/abs/1810.04805), where (part of) the pretraining objective is to predict a masked-out word. This masking can be in the form of a `[MASK]` token or a randomly chosen token. You have to do *some* kind of masking as otherwise the model will just learn to repeat the input token (although in the BERT paper, they do leave the token unchanged 10% of the time). To produce these masks, you need to do some preprocessing beforehand, although you can mask multiple tokens (and thus calculate losses for them) in a single forward pass.
  
  **Encoder-decoder models** are often also trained a similar way. E.g., for [T5](https://arxiv.org/abs/1910.10683), you mask out, then predict consecutive spans of tokens, but your model can predict multiple token spans per forward pass.
  
  
  ---
  
  