Answers.Body: <p>Transfer learning in machine translation is a relatively common technique in machine translation. Mostly, it means fine-tuning pre-trained self-supervised sequence-to-sequence models, such as <a href="https://aclanthology.org/2020.tacl-1.47" rel="nofollow noreferrer">mBART</a>. It is also often used for low-resource languages to transfer from a related high-resource language, as e.g. most of the participants in the <a href="https://aclanthology.org/2021.wmt-1.72" rel="nofollow noreferrer">WMT21 low-resource competition</a> did.</p>
  <p>What you suggest is very close to a <a href="https://aclanthology.org/W18-6325/" rel="nofollow noreferrer">2018 paper by Kocmi and Bojar</a>. You might use their setup as a starting point. The main challenges addressed in the paper are:</p>
  <ul>
  <li>Carefully setting the learning rate schedule to avoid catastrophic forgetting.</li>
  <li>Do something about the vocabulary mismatch (Finnish uses a different vocabulary than English).</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T14:26:45.313Z
Answers.LastActivityDate: 2023-01-11T14:26:45.313Z
Answers.UserDisplayName: Jindřich
Body: <p>I am working on the Transformer example demonstrated on TensorFlow's website. <a href="https://www.tensorflow.org/text/tutorials/transformer" rel="nofollow noreferrer">https://www.tensorflow.org/text/tutorials/transformer</a></p>
  <p>In this example, Machine Translation model is trained to translate from Portuguese to English. The transformer is coded from scratch and other popular libraries like huggingface are not used.</p>
  <p>Let's say I have another dataset which includes pairs of sentences of Portuguese and Finnish and let's say this dataset is fairly small. Since it is a small dataset, I want to use my model trained on Portuguese to English as a PreTrained model for creating the translation model for Portuguese to Finnish.</p>
  <p>My question is, what are the key points to consider when using such a PreTrained model and changing ONLY its decoder output structure?</p>
  
Tags: <transformer><machine-translation><pretrained-models><seq2seq>
Title: Fine Tuning Transformer Model for Machine Translation
UserDisplayName: boyaronur
ThreadText: # Fine Tuning Transformer Model for Machine Translation
  
  (Asked by: username_0 on 2022-12-16)
  
  I am working on the Transformer example demonstrated on TensorFlow's website. <https://www.tensorflow.org/text/tutorials/transformer>
  
  In this example, Machine Translation model is trained to translate from Portuguese to English. The transformer is coded from scratch and other popular libraries like huggingface are not used.
  
  Let's say I have another dataset which includes pairs of sentences of Portuguese and Finnish and let's say this dataset is fairly small. Since it is a small dataset, I want to use my model trained on Portuguese to English as a PreTrained model for creating the translation model for Portuguese to Finnish.
  
  My question is, what are the key points to consider when using such a PreTrained model and changing ONLY its decoder output structure?
  
  **Tags:** transformer, machine-translation, pretrained-models, seq2seq
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-11. Score: 0
  
  Transfer learning in machine translation is a relatively common technique in machine translation. Mostly, it means fine-tuning pre-trained self-supervised sequence-to-sequence models, such as [mBART](https://aclanthology.org/2020.tacl-1.47). It is also often used for low-resource languages to transfer from a related high-resource language, as e.g. most of the participants in the [WMT21 low-resource competition](https://aclanthology.org/2021.wmt-1.72) did.
  
  What you suggest is very close to a [2018 paper by Kocmi and Bojar](https://aclanthology.org/W18-6325/). You might use their setup as a starting point. The main challenges addressed in the paper are:
  
  * Carefully setting the learning rate schedule to avoid catastrophic forgetting.
  * Do something about the vocabulary mismatch (Finnish uses a different vocabulary than English).
  
  
  ---
  
  