Answers.Body: <p>The short answer: Very unlikely.</p>
  <p>The extended answer: If you fine-tune a model, it becomes specialized for the type of data you fine-tune it on but you trade in some of its generalization capabilities. In your particular example, that means that your model becomes very attuned to user feedback and performs very highly on user feedback in the languages it has seen during training. It looses both its capabilities for more general texts and other languages from pre-training. It will perform lower on general texts and significantly lower on user feedback in different languages from your training set.</p>
  <p>To address this problem, you need something called domain adaptation or transfer learning. Domain adaptation means you train your model on a domain dataset, e.g. your user feedback dataset in a small range of languages, and then adapt/transfer it to your actual target dataset. See an example below:</p>
  <p><a href="https://i.sstatic.net/QEhdr.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QEhdr.png" alt="Domain Adaptation in Vision" /></a></p>
  <p>There are different approaches for this. Some adapt the data, others prefer to transfer the model. There is a lot of research in this area. This is a good <a href="https://www.ruder.io/transfer-learning/" rel="nofollow noreferrer">article</a> that covers the basics and might provide you with a more specific solution you can try out.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-06T05:42:34.373Z
Answers.LastActivityDate: 2023-07-06T05:42:34.373Z
Answers.UserDisplayName: emely_pi
Answers.Body: <p>The attention doesn't discriminate between what token it is producing as long as it is following the protocol/heuristic defined by the finetuning dataset,</p>
  <p>So essentially, a finetuning dataset that contains</p>
  <p><code>Hi, How are you</code> ---&gt; <code>I am fine, thank you.</code></p>
  <p>should be able to generalize it to French as well,</p>
  <p><code>Salut, Comment allez-vous</code> ---&gt; <code>Je vais bien, merci.</code></p>
  <p>As long as the transformer understands both English and French.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-06T06:33:45.753Z
Answers.Text: While this could be true, it is very unlikely that there is the same kind of performance on the untrained languages.
Answers.UserDisplayName: N. Kiefer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-06T05:33:32.490Z
Answers.LastActivityDate: 2023-07-06T05:33:32.490Z
Answers.UserDisplayName: Chinmay
Body: <p>Example: <a href="https://huggingface.co/google/umt5-base" rel="nofollow noreferrer">https://huggingface.co/google/umt5-base</a></p>
  <blockquote>
  <p>Note: UMT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.</p>
  </blockquote>
  <p>The model was pre-trained on a whole lot of languages. Let's suppose I devise a fine-tuning dataset to classify sentences (like user feedback). But due to resource constraints, only a few languages will be included. In my fine-tuned task, will the model be able to generalize to languages it was pre-trained, but not fine-tuned on?</p>
  
Tags: <natural-language-processing><transformer><fine-tuning>
Title: Does fine-tuning a multilingual transformer model allow it to generalize to languages unseen in the fine-tuning dataset?
UserDisplayName: Michał B.
ThreadText: # Does fine-tuning a multilingual transformer model allow it to generalize to languages unseen in the fine-tuning dataset?
  
  (Asked by: username_0 on 2023-07-06)
  
  Example: <https://huggingface.co/google/umt5-base>
  
  > Note: UMT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.
  
  The model was pre-trained on a whole lot of languages. Let's suppose I devise a fine-tuning dataset to classify sentences (like user feedback). But due to resource constraints, only a few languages will be included. In my fine-tuned task, will the model be able to generalize to languages it was pre-trained, but not fine-tuned on?
  
  **Tags:** natural-language-processing, transformer, fine-tuning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-07-06. Score: 2 | Accepted answer
  
  The short answer: Very unlikely.
  
  The extended answer: If you fine-tune a model, it becomes specialized for the type of data you fine-tune it on but you trade in some of its generalization capabilities. In your particular example, that means that your model becomes very attuned to user feedback and performs very highly on user feedback in the languages it has seen during training. It looses both its capabilities for more general texts and other languages from pre-training. It will perform lower on general texts and significantly lower on user feedback in different languages from your training set.
  
  To address this problem, you need something called domain adaptation or transfer learning. Domain adaptation means you train your model on a domain dataset, e.g. your user feedback dataset in a small range of languages, and then adapt/transfer it to your actual target dataset. See an example below:
  
  [![Domain Adaptation in Vision](https://i.sstatic.net/QEhdr.png)](https://i.sstatic.net/QEhdr.png)
  
  There are different approaches for this. Some adapt the data, others prefer to transfer the model. There is a lot of research in this area. This is a good [article](https://www.ruder.io/transfer-learning/) that covers the basics and might provide you with a more specific solution you can try out.
  
  
  ---
  
  ### Answer by username_2 on 2023-07-06. Score: 1
  
  The attention doesn't discriminate between what token it is producing as long as it is following the protocol/heuristic defined by the finetuning dataset,
  
  So essentially, a finetuning dataset that contains
  
  `Hi, How are you` ---> `I am fine, thank you.`
  
  should be able to generalize it to French as well,
  
  `Salut, Comment allez-vous` ---> `Je vais bien, merci.`
  
  As long as the transformer understands both English and French.
  
  > **Comments:**
  > **username_3**: While this could be true, it is very unlikely that there is the same kind of performance on the untrained languages.  
  
  ---
  
  