From English to Foreign Languages: Transferring Pre-trained Language Models
Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain a foreign BERT-base model within a day and a foreign BERT-large within two days. Furthermore, evaluating our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and dependency parsing.
In this work, the authors propose a way to transfer a pre-trained English BERT model to a new language within a short amount of time. The key insight is to map English embeddings to the foreign language and have separate embeddings for both English and the foreign language. The resulting bilingual LM is evaluated for zero-shot transfer learning on two tasks: XNLI and dependency parsing. Pros: - The authors provide good details into their hyperparameter settings and about how the obtain the foreign language word embeddings. - By leveraging existing pre-trained models, they’re able to do pre-training for their bilingual LM within 2 days. Cons: I find that a key comparison point in this paper is missing, which is Bilingual BERT trained on just the two languages that are being considered for their RAMEN system. This is not a fair comparison while mBERT which is trained on 100+ languages is not. All comparisons are not fair since a simple baseline of just training mBERT on two languages with monolingual data and with a shared WPM is not evaluated here. The proposed system has an unfair advantage over mBERT since it’s initialized from BERT/RoBERTA and fine-tuned only on two languages. Hence most of the parameters are used for just the two languages while mBERT uses the parameters for 104 languages. Given this unfair comparison, I’m not sure if we can draw a meaningful conclusion from all the experiments. Rating justification: Given the lack a fair comparison between the bilingual and multilingual BERT models, I don*t think the conclusions are insightful.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
From English to Foreign Languages: Transferring Pre-trained Language Models
Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain a foreign BERT-base model within a day and a foreign BERT-large within two days. Furthermore, evaluating our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and dependency parsing.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



In this work, the authors propose a way to transfer a pre-trained English BERT model to a new language within a short amount of time. The key insight is to map English embeddings to the foreign language and have separate embeddings for both English and the foreign language. The resulting bilingual LM is evaluated for zero-shot transfer learning on two tasks: XNLI and dependency parsing. Pros: - The authors provide good details into their hyperparameter settings and about how the obtain the foreign language word embeddings. - By leveraging existing pre-trained models, they’re able to do pre-training for their bilingual LM within 2 days. Cons: I find that a key comparison point in this paper is missing, which is Bilingual BERT trained on just the two languages that are being considered for their RAMEN system. This is not a fair comparison while mBERT which is trained on 100+ languages is not. All comparisons are not fair since a simple baseline of just training mBERT on two languages with monolingual data and with a shared WPM is not evaluated here. The proposed system has an unfair advantage over mBERT since it’s initialized from BERT/RoBERTA and fine-tuned only on two languages. Hence most of the parameters are used for just the two languages while mBERT uses the parameters for 104 languages. Given this unfair comparison, I’m not sure if we can draw a meaningful conclusion from all the experiments. Rating justification: Given the lack a fair comparison between the bilingual and multilingual BERT models, I don*t think the conclusions are insightful.
assistant
In this work, the authors propose a way to transfer a pre-trained English BERT model to a new language within a short amount of time. The key insight is to map English embeddings to the foreign language and have separate embeddings for both English and the foreign language. The resulting bilingual LM is evaluated for zero-shot transfer learning on two tasks: XNLI and dependency parsing. Pros: - The authors provide good details into their hyperparameter settings and about how the obtain the foreign language word embeddings. - By leveraging existing pre-trained models, they’re able to do pre-training for their bilingual LM within 2 days. Cons: I find that a key comparison point in this paper is missing, which is Bilingual BERT trained on just the two languages that are being considered for their RAMEN system. This is not a fair comparison while mBERT which is trained on 100+ languages is not. All comparisons are not fair since a simple baseline of just training mBERT on two languages with monolingual data and with a shared WPM is not evaluated here. The proposed system has an unfair advantage over mBERT since it’s initialized from BERT/RoBERTA and fine-tuned only on two languages. Hence most of the parameters are used for just the two languages while mBERT uses the parameters for 104 languages. Given this unfair comparison, I’m not sure if we can draw a meaningful conclusion from all the experiments. Rating justification: Given the lack a fair comparison between the bilingual and multilingual BERT models, I don*t think the conclusions are insightful.