Answers.Body: <p>Docoder-only transformers are also trained on text pairs. You first pre-train on general text data. After that you fine tune with a standard two language dataset containing example translations.</p>
  <p>The trick is that you concatenate the source and the target sentence, you put a special separator token, and you treat the task as language modelling, i.e. you ask the model to predict the next word one-by-one.</p>
  <p>Input: [&quot;Good&quot;, &quot;morning&quot;, &quot;everyone&quot;, &quot;&lt;SEP&gt;&quot;, &quot;Guten&quot;, &quot;Morgen&quot;, &quot;zusammen&quot;, &quot;&lt;END&gt;&quot;.]<br />
  Decoder output: [<span class="math-container">$z_1, z_2, z_3, z_4, z_5, z_6, z_7, z_8$</span>]<br />
  You then compute the loss by considering only the outputs from <span class="math-container">$z_5$</span> to <span class="math-container">$z_8$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T15:28:52.437Z
Answers.LastActivityDate: 2023-07-29T15:28:52.437Z
Answers.UserDisplayName: pi-tau
Body: <p>In my understanding encoder-decoder transformers for translation are trained with sentence or text pairs. How can it be explained in simple (high-level) terms that decoder-only transformers (e.g. GPT) are so <a href="https://arxiv.org/pdf/2302.09210.pdf" rel="nofollow noreferrer">good at machine translation</a>, even though they are not trained on sentence or text pairs but only on unrelated multilingual training data? Why can decoder-only transformers do without it? Or did I get something wrong?</p>
  <p>Are the documents in the training data containing accidentally sentence pairs near each other possibly enough?</p>
  
Tags: <transformer><machine-translation><encoder-decoder>
Title: Why can decoder-only transformers be so good at machine translation?
UserDisplayName: Hans-Peter Stricker
ThreadText: # Why can decoder-only transformers be so good at machine translation?
  
  (Asked by: username_0 on 2023-06-07)
  
  In my understanding encoder-decoder transformers for translation are trained with sentence or text pairs. How can it be explained in simple (high-level) terms that decoder-only transformers (e.g. GPT) are so [good at machine translation](https://arxiv.org/pdf/2302.09210.pdf), even though they are not trained on sentence or text pairs but only on unrelated multilingual training data? Why can decoder-only transformers do without it? Or did I get something wrong?
  
  Are the documents in the training data containing accidentally sentence pairs near each other possibly enough?
  
  **Tags:** transformer, machine-translation, encoder-decoder
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-29. Score: 1
  
  Docoder-only transformers are also trained on text pairs. You first pre-train on general text data. After that you fine tune with a standard two language dataset containing example translations.
  
  The trick is that you concatenate the source and the target sentence, you put a special separator token, and you treat the task as language modelling, i.e. you ask the model to predict the next word one-by-one.
  
  Input: ["Good", "morning", "everyone", "<SEP>", "Guten", "Morgen", "zusammen", "<END>".]  
  Decoder output: [$z\_1, z\_2, z\_3, z\_4, z\_5, z\_6, z\_7, z\_8$]  
  You then compute the loss by considering only the outputs from $z\_5$ to $z\_8$.
  
  
  ---
  
  