Transfer learning with a unified Transformer framework (T5) that converts all language problems into a text-to-text format was recently proposed as a simple and effective transfer learning approach. Although a multilingual version of the T5 model (mT5) was also introduced, it is not clear how well it can fare on non-English tasks involving diverse data. To investigate this question, we apply mT5 on a language with a wide variety of dialectsArabic. For evaluation, we introduce a novel benchmark for AR abic language GEN eration (ARGEN), covering seven important tasks. For model comparison, we pre-train three powerful Arabic T5-style models and evaluate them on ARGEN. Although pre-trained with  49% less data, our new models perform significantly better than mT5 on all ARGEN tasks (in 52 out of 59 test sets) and set several new SOTAs. Our models also establish new SOTA on the recently-proposed, large Arabic language understanding evaluation benchmark ARLUE (Abdul-Mageed et al., 2021). Our models are publicly available. We also link to individual ARGEN datasets through our public repository. 1 1 Introduction Due to their remarkable ability to transfer knowledge from unlabeled data to downstream tasks, pre-trained Transformer-based language models have emerged as important components of modern natural language processing (NLP) systems. In particular, the unified framework that converts all text-based language problems into a text-to-text format presented through the T5 model (Raffel et al., 2019) is attractive. In addition to its simplicity, this approach is effective since it allows knowledge transfer from high-resource to low-resource tasks 1 https://github.com/UBC-NLP/araT5 (cid:63) All authors contributed equally. without the need for changing model architecture. Unlike models such as BERT (Devlin et al., 2019), which are based on encoders only, the T5 model is an encoder-decoder that can naturally be employed for natural language generation. Although the T5 model, originally pre-trained for English, was recently extended to the multilingual setting as mT5 (Xue et al., 2020), it is not clear how suited it is to individual languages (and varieties of these languages). In addition, systematic issues have been discovered in multilingual corpora on which language models have been trained (Kreutzer et al., 2021). In absence of comparisons with monolingual pre-trained language models that serve different non-English contexts, it remains unknown how multilingual models really fare against language-specific models. In this work, we offer the first comparison of the mT5 model to similar encoder-decoder models dedicated to Arabic. We choose Arabic as our context due to its large set of diverse varieties as well as its wide use on social media. Our work aims at uncovering the extent to which mT5 can serve Arabic's different varieties. Our work also meets an existing need for pre-trained Transformer-based sequence-to-sequence models. In other words, while several BERT-based models have been pre-trained for Arabic (Antoun et al., 2020; Abdul-Mageed et al., 628 2021; Inoue et al., 2021), no such attempts have been made to create sequence-to-sequence models that we know of. Another motivation for our work is absence of an evaluation benchmark for Arabic language generation tasks. Apart from machine translation where researchers are starting to propose benchmarks such as AraBench (Sajjad et al., 2020), there are no benchmarks that can be used to methodically measure Arabic natural language generation performance. Our main contributions are as follows: (1) We introduce three powerful variants of the text-to-text transformer (T5) model dedicated to Modern Standard Arabic (MSA) and a diverse set of Arabic dialects. We include in our vocabulary 11 languages other than Arabic (e.g., English, French, German, Russian), which also allows us to evaluate our models under zero-shot pre-training conditions involving these languages. (2) We propose a novel unified benchmark for AR abic natural language GE eneration ( ARGEN ) composed of seven tasks: machine translation, code-switched text translation, summarization, news title generation, question generation, paraphrasing, and transliteration. ARGEN is collected from a total of 19 datasets, including 9 new datasets proposed in this work. (3) To show the utility of our new models, we evaluate them on ARGEN under both full and zero-shot pre-training conditions. Our models set new SOTA on the majority of datasets in all seven tasks. (4) Although the main focus of our work is language generation , we also show the effectiveness of our models on Arabic language understanding by fine-tuning our new models on a large, recently proposed Arabic language understanding benchmark. Again, our models establish new SOTA on the majority of language understanding tasks. The rest of the paper is organized as follows: Section 2 describes our Arabic pre-tained models. In Section 3, we introduce ARGEN, our new natural language generation benchmark. We evaluate our models on ARGEN in Section 4. Section 5 is an analysis and discussion of our results. In Section 6, we provide an overview of related work. We conclude in Section 7. We now introduce our new pre-trained models. MSA Data. We use 70 GB of MSA text ( 7 . 1 B tokens) from the following sources: AraNews (Nagoudi et al., 2020), El-Khair El-Khair (2016), Gigaword, 2 , OSCAR (Surez et al., 2019), OSIAN (Zeroual et al., 2019), Wikipedia Arabic, and Hindawi Books. 3 Twitter Data. We randomly sample 1 . 5 B Arabic tweets ( 178 GB) from a large in-house dataset of  10 B tweets. We use string matching to only include tweets with at least 3 Arabic words, regardless whether the tweet has non-Arabic string or not. Our combined MSA and Twitter data make up 29 B tokens, and hence is  49% less than Arabic tokens on which mT5 is pre-trained ( 57 B Arabic tokens). More information about our pre-training data is in Table 1. MSA Vs. Dialect Distribution. In order to analyze MSA-dialect distribution in our Twitter data, we run the binary (MSA-dialect) classifier introduced in Abdul-Mageed et al. (2020b) on a random sample of 100 M tweets. We find the data to involve 28 . 39 % predicted dialect tweets and 71 . 61 % predicted MSA. We also acquire country-level dialect labels using an in-house strong classifier on the dialectal portion of the data (i.e.,  28 . 39 millions tweets), finding dialectal tweets to be truly geographically diverse as shown in Figure 2. Figure 2: Country-level distribution in the dialectal portion of our data. Naturally-Occurring Code-Switching. Using 1 M random tweets from our data, we perform an analysis of code-switching. For this, we employ simple string matching to identify Arabic and run the CLD3 language ID tool 4 on the non-Arabic string sequences. We find the data to have 4 . 14 % non-Arabic. These turn out to be almost always natural code-switching involving many foreign languages (e.g., English, French, Korean, etc.). We remove diacritics and replace URLs and user mentions with <URL> and <USER> . We also clean the data by removing HTML tags, elongation, and the hash signs. Further, we reduce repetitive characters, emojis, and emoticons to one. To create our language model vocabulary, we use Sentence-Piece (Kudo, 2018) to encode text as WordPiece tokens (Sennrich et al., 2016) with 110 K Word-Pieces. To allow for further pre-training (and/or fine-tuning) on additional languages, we extract our vocabulary as follows: 70 M MSA sentences, 200 M Arabic twitter data, 15 M sentences from Wikipedia English, and 5 M sentences from the Wikipedia of 10 other languages (Bulgarian, French, German, Greek, Italian, Portuguese, Russian, Spanish, Turkish, Czech). 5 In  3.1.2, we describe parallel data from four of these languages on which we fine-tune our models for X  Arabic MT. Our respective results (reported in Table 4.2) demonstrate the utility of including foreign vocabulary in our models. Model Architecture. We leverage our unlabeled MSA and Twitter data described in  2.1 to pretrain three models: AraT5 MSA on MSA data, AraT5 TW on twitter data, and AraT5 on both MSA and twitter data using the T5 Base encoder-decoder architecture (Raffel et al., 2019). Each of the encoder and decoder components is similar in size and configuration to BERT Base (Devlin et al., 2019), with 12 layers each with 12 attention heads, and 768 hidden units. In total, this results in a model with  220 million parameters. 6 Objective. Raffel et al. (2019) pre-train T5 Base using a self-5 The MSA and twitter data are extracted from our training data presented in Section 2.1. 6 The output dimensionality is d ff = 3 , 072 and inner dimensionality of d kv = 64 . supervised (denoising) objective. The main idea is to feed the model with masked (corrupted) versions of the original sentence, and train it to reconstruct the original sequence. Inspired by BERT's objective (Devlin et al., 2019), the denoising objective (Raffel et al., 2019) works by randomly sampling and dropping out 15 % of tokens in the input sequence. All consecutive spans of dropped-out tokens are then replaced by a single sentinel token. Pre-Training. For all three of our pre-trained models, we use a learning rate of 0 . 01 , a batch size of 128 sequences, and a maximum sequence length of 512 , except for AraT5 TW where the maximum sequence is 128 . 7 We pre-train each model for 1 M steps. Pre-training of each model took  80 days on one Google Cloud TPU with 8 cores (v 3 . 8 ) from TensorFlow Research Cloud (TFRC). 8 We now introduce our language generation and understating benchmarks. In order to evaluate our pre-trained language models, we introduce our new benchmark for Arabic language generation evaluation ARGEN . It includes 19 different datasets with 59 test splits and covers seven tasks: machine translation (MT), code-switched translation (CST), text summarization (TS), news title generation (NGT), question generation (QG), transliteration (TR), and paraphrasing (PPH). As such, ARGEN has wide-coverage both in terms of the number of tasks and datasets. It is also linguistically diverse as it covers both MSA and various Arabic dialects, in addition to Arabizi (romanized Arabic in the TS task) and code-switching (in the CST task). We now describe each component of ARGEN. To design the MT component of ARGEN, ARGENMT , we consolidate 7 unique datasets with 46 different test splits. The datasets come from both MSA and Arabic dialects, and range between 600 138 K sentences (details in Table C.2 in Ap-pendix). We introduce each dataset briefly here. man-7 We choose the same maximum sequence used in MARBERT (Abdul-Mageed et al., 2021), the most powerful model trained on Arabic twitter to date (Farha and Magdy, 2021). 8 https://www.tensorflow.org/tfrc. ually translated UN documents covering the six official UN languages (i.e., Arabic, Chinese, English, French, Russian, and Spanish). The corpus consists of development and test sets only, each of which comprise 4 , 000 sentences that are one-to-one alignments across all official languages. (2) IWSLT Corpus. Several Arabic-to-English parallel datasets were released during IWSLT evaluation campaigns (Federico et al., 2012; Cettolo et al., 2013, 2014, 2016). The datasets are mainly extracted from transcriptions of TED talks between 2010 and 2016, and the QCRI Educational Domain Corpus (QED 2016) (Abdelali et al., 2014). AraBench Datasets. Sajjad et al. (2020) introduce AraBench, an evaluation suite for MSA and dialectal Arabic to English MT consisting of five publicly available datasets: (3) ADPT: Arabic-Dialect/English Parallel Text (Zbib et al., 2012), (4) MADAR: Multi-Arabic Dialect Applications and Resources dataset (Bouamor et al., 2018), (5) QAraC: Qatari-English speech corpus (Elmahdy et al., 2014), and (6) Bible: The English Bible translated into MSA, Moroccan, and Tunisian Arabic dialects. 9 For all these datasets, we use the same splits as Sajjad et al. (2020) in our experiments. To investigate ability of our models to generate Arabic starting from foreign languages in our vocabulary, we create an X  Arabic benchmark of four languages (English, French, German, and Russian) by extracting parallel data from OPUS (Tiedemann, 2012). For each language, we pick 1 M sentences for training and 5 K sentences for each of development and test splits. This gives us our seventh ARGENMT dataset, which we call (7) OPUS-X-Ara . There is rising interest in translating code-switched data (Nagoudi et al., 2021). Our purpose here is to translate Arabic text involving code-switching from a foreign language into (i) that foreign language as well as into (ii) MSA. Hence we create ARGENCST , our code-switched translation benchmark component, using four sub-test sets. Two of these are natural and two are synthetic , as follows: Natural Code-Switched Data. We create two human written (natural) code-switched parallel 9 The United Bible Societies https://www.bible.com. datasets: (1) ALG-CST. This is collected from Algerian Twitter and consists of code-switched Arabic-French posts. We translate these manually into monolingual French. (2) JOR-CST. This is collected from Jordanian Twitter and consists of code-switched Arabic-English posts, which we manually translate into monolingual English. Each of ALG-CST and JOR-CST comprises 300 tweets (total= 600 ). Human translation is performed by one native speaker from each dialect with seminative English/French fluency. Synthetic Code-Switched Data. We use the multilingual sequence-to-sequence model mBART (Liu et al., 2020) to create synthetic code-switched data following Jawahar et al. (2021). We exploit the UN multi-parallel data (Ziemski et al., 2016) using the Arabic-English and Arabic-French test splits ( 4 , 000 sentences each, described in  3.1) to generate our two code-switched test sets (3) MSA-EN and (4) MSA-FR . In each case, we use mBART to translate  30% random Arabic n-grams into the target language (i.e., English or French). component, ARGENTS , we use the following: Essex Arabic Summaries Corpus (EASC). EASC (El-Haj et al., 2010) contains 153 Arabic Wikipedia and newspaper articles, each with 5 human-generated extractive summaries (total= 765 summaries). The summaries are crowdsourced via Mechanical Turk. 10 WikiLingua. An abstractive summarization dataset in 18 languages, including Arabic (Faisal Ladhak and McKeown, 2020). It contains articles and their summaries from WikiHow. 11 The Arabic part includes summaries for 29 . 2 K articles, which we split into 80% Train ( 23 . 4 K), 10% Dev ( 2 . 9 K), and 10% Test ( 2 . 9 K). The purpose of the news title generation (NTG) task is to produce proper news article titles (Liang et al., 2020). We introduce NTG as a new task for Arabic language generation. Given an article, a title generation model needs to output a short grammatical sequence of words suited to the article content. For this, we introduce ARGENNTG , a novel NTG dataset exploiting 120 K articles along 10 http://www.mturk.com/ 11 http://www.wikihow.com 631 with their titles extracted from AraNews (Nagoudi et al., 2020). 12 We only include titles with at least three words in this dataset. We split ARGENNTG data into 80% Train ( 93 . 3 K), 10% Dev ( 11 . 7 K), and 10% Test ( 11 . 7 K). Details about ARGENNTG are in Table C.1 (Appendix). A sample of a news article from our Test split and example titles generated by our models are in Table D.5 (Appendix). In the question generation (QG) task, a question is produced for a passage (Gehrmann et al., 2021). Given the absence of an Arabic QG dataset, we create a new Arabic QG dataset ( ARGENQG ) using a publicly available Arabic question answering (QA) resource. We follow Kriangchaivech and Wangperawong (2019) who train a model to generate simple questions relevant to passages and answers extracted from SQuAD (Rajpurkar et al., 2016). In our case, we build ARGENQG by extracting 96 K (passage, answer, and question) triplets from (1) The Arabic QA dataset ARCD (Mozan-nar et al., 2019), and (2) three multi-lingual QA datasets: XTREME benchmark (Hu et al., 2020), MLQA (Lewis et al., 2019), XQuAD (Artetxe et al., 2020), and TyDi QA (Artetxe et al., 2020). The main goal of this task is to produce for a given Arabic sentence a paraphrase with the same meaning. In order to build our paraphrasing benchmark component ( ARGENPPH ), we use the following three datasets: AraPara. We introduce AraPara, a new multi-domain Arabic paraphrasing dataset we create using English-Arabic parallel OPUS data (Tiede-mann, 2012). AraPara covers several domains such as news, religion, politics, movies, and technology. To create a high quality machine generated paraphrase dataset, we follow four careful steps involving human validation (more details are offered in Appendix C.1). AraPara consists of 122 K paraphrase pairs. We only use AraPara for model development, and hence we split it into 116 K Train and 6 K Dev. Arabic SemEval Paraphrasing (ASEP). We also create a new Arabic paraphrasing dataset using three existing Arabic semantic similarity datasets released during SemEval 2017 (Cer et al., 2017). 12 We ensure no overlap exists between ARGENTG and the AraNews data we use to pre-train our language models (described in  2.3). These are MSR-Paraphrase ( 510 pairs), MSR-Video ( 368 pairs), and SMTeuroparl ( 203 pairs). The pairs are labeled with a similarity score on a scale from 0 to 5 . For our purpose, we only keep sentence pairs with a semantic similarity score  3 . 5 which gives us 603 pairs. We merge and shuffle all three ASEP datasets for our use. Arabic Paraphrasing Benchmark (APB). APB is created by Alian et al. (2019). It consists of 1 , 010 Arabic sentence pairs that are collected from different Arabic books. Paraphrasing was performed manually using six transformation procedures (i.e., addition, deletion, expansion, permutation, reduction, and replacement). Transliteration involves mapping a text written with orthographic symbols in a given script into another (Beesley, 1998). We use the BOLT Egyptian Arabic SMS/Chat and Transliteration dataset (Song et al., 2014), 13 a collection of naturally-occurring chat and short messages (SMS) from Egyptian native speakers. The messages (sources) were natively written in either romanized Arabizi or Egyptian Arabic orthography. The target is the Egyptian transliteration of these message. 14 For experiments, we use the same split proposed by Shazal et al. (2020) ( 58 . 9 K for Train and 5 . 4 K for Dev and Test each). We refer to this dataset as ARGENTR . Baselines and Procedure. For all tasks, we compare our models to models fine-tuned with mT5 using the same training data. In addition, for MT, we compare to a vanilla sequence-to-sequence (S2S) Transformer (Vaswani et al., 2017) trained from scratch as implemented in Fairseq (Ott et al., 2019). For all models and baselines, across all tasks, we identify the best model on the respective Dev data and blind-test it on Test data. As a rule, we report on both Dev and Test sets. All our Dev results are in Section C.2 in the Appendix. We train two S2S Transformers models on 2 M (S2S 2M ) and 10 M (S2S 10M ) MSA-English parallel sentences extracted from OPUS. We take these 13 https://catalog.ldc.upenn.edu/LDC2017T07 14 Some transliteration sequences involve code mixing between Egyptian Arabic and English. two models as our baseline I . We also fine-tune our three models as well as mT5 on the same OPUS 2 M MSA-English parallel sentences used for baseline I. Fine-tuned mT5 is our second baseline baseline II . Arabic  English. Results of ARGENMT are reported in Table 2. Results show that our models achieve best BLEU score in 37 out of the 42 tests splits. AraT5 MSA acquires best results in 32 of these test splits, outperforming all the baselines (S2S 2M ), (S2S 10M ), and mT5 with + 5 . 25 , + 4 . 99 , and + 0 . 45 BLEU points. These results are striking since our language models are pre-trained on Arabic data only (although they include English vocabulary and marginal amounts of code-switching; see  2.1). In other words, even under this arguably zero-shot setting, 15 the models perform very well. In addition, our AraT5 model outperforms even the S2S model trained with 5 X more data. For completeness, we also provide the current SOTA on each of our datasets. We do not compare our results to SOTA since these are acquired by models fine-tuned on much larger datasets than ours. For example, Sajjad et al. (2020) exploit  42 M par-ralel sentences to train their models. To limit GPU needs during our experiments, especially given the time-consuming fine-tuning process typical of T5 models, we do not fine-tune the models on the full amounts of available parallel data. However, in the future we plan to compare our models under the full data setting. X  Arabic. Our language models are not pre-trained on foreign data, but we include vocabulary from 11 foreign languages. Our X  Arabic experiments here are hence zero-shot (from the perspective of pre-training). Table 4.2 shows the results of AraT5 MSA and mT5 on OPUS-X-Ara. 16 We observe that our model outperforms mT5 in the four X  Arabic sub-tasks with an average of + 1 . 12 and + 0 . 86 BLEU points on Dev and Test, respectively. For this task, we test on the two natural code-switched translation (CST) test sets that we manually created, ALG-FR  FR and JOR-EN  EN. We also evaluate on our two synthetic CST datasets, MSA-EN and MSA-FR, one time with EN/FR as target (e.g., MSA-EN  EN) and another with MSA as target (e.g., MSA-EN  MSA). We fine-tune 15 At best, this can be viewed as few-shot pre-training. 16 To limit GPU time, we fine-tune only AraT5 MSA model on the X  Arabic direction since it performed best on Arabic  English section above. our three pre-trained models as well as mT5 on the OPUS-X-Ara segments involving English and French (each with 1 M parallel sentences, described in  3.1.2), in both directions. Since these MT models are only fine-tuned on parallel monolingual data, we refer to these experiments as zero-shot . We test these models on both our natural and synthetic code-switched data (described in  3.2). We report results in Table 3. Our models achieve best results in one out of the two natural test sets (with + 4 . 36 BLEU points on ALG-FR) and all four synthetic test sets (e.g., + 4 . 55 BLEU points on MSA-EN  MSA). These results clearly show our models' remarkable language generation ability especially in the Arabic direction. For the two ARGENST datasets, we fine-tune and identify the best model on the Train and Dev splits of WikiLingua (Faisal Ladhak and McKeown, 2020) and test on all EASC and the Test of WikiLingua. We report different ROUGE scores (Lin, 2004) in Table 5. As the Table shows, AraT5 Tw acquires best results on WikiLingua data, while mT5 outperforms us on EASC (we hypothesize since EASC is older data that is likely part of the mC4 on which mT5 was pre-trained). On both datasets, we establish new SOTA (both with our pre-trained models and mT5). For both tasks, we fine-tune all our models on the Train splits of ARGENNTG and ARGENQG , respectively. As Table 6 shows, all our models outperform mT5 on each of the two tasks. AraT5 MSA excels with 20 . 61 % BLEU on ARGENNTG and AraT5 is at 16 . 99 % on ARGENQG . For the paraphrasing task, we fine-tune and validate on our new AraPra dataset and blind-test on both APB and ASEP datasets (described in 3.6). As Table 6 shows, AraT5 MSA is best on APB ( 17 . 52 BLEU) and ASEP ( 19 . 38 BLEU). For transliteration , we fine-tune our models on the Train split of ARGENTR . As Table 6 shows, each of AraT5 MSA and AraT5 outperform mT5. Notably, AraT5 MSA is at 65 . 88 BLEU, outperforming previous SOTA (Shazal et al., 2020) by 7 . 1 points. We also evaluate our new pre-trained models on the recently proposed Arabic language understanding and evaluation benchmark, ARLUE (Abdul-Mageed et al., 2021) that involves six cluster tasks (i.e., sentiment analysis, social meaning, topic classification, dialect identification, named entity recognition, and question answering). Our models establish new SOTA on the benchmark with an ARLUE score of 77 . 52 vs. the previous SOTA of 76 . 53 , reported by ARLUE authors. We provide results of this set of experiments in Appendix B. 5 Analysis and Discussion 5.1 Multilingual vs. Dedicated Models. Our results confirm the utility of dedicated language models as compared to multilingual models such as mT5 ( 101+ languages). Our AraT5 model outperforms mT5, even though it is pre-trained with 49% less data (see  2.1). One reason might be that massively multilingual models are more prone to suffering from capacity issues. Data quality is another challenge for multilingual models. As pointed out earlier, Kreutzer et al. (2021) find systematic issues with data representing several languages (including Arabic) in the mC4 dataset on which mT5 is pre-trained. We perform a data quality study confirming the findings of Kreutzer et al. (2021). We also find Arabic mC4 data to be less geographically diverse than our Twitter pretraining data (described in  2.1). Our mC4 data study is in Appendix A. Code-Switching. We also study code-switching in both our Twitter dataset and the Arabic part of mC4. We find that while our Twitter data involves natural code-switching (  4% of sequences), code-switching in Arabic mC4 is very rare. This explains the strong performance of our AraT5 Tw model on the natural code-switched translation data on French. We conjecture that mT5 good performance on English code-switched data is due to it being pre-trained on very large amounts of English rather than natural code-switching. We were inquisitive how MT models fine-tuning our pre-trained language models compare to mT5 under different length conditions. For this, we (1) merge all MSA and dialectal Test datasets in our Arabic  English experiments to form a single dataset that we then (2) split into three bins/Test sets based on sentence length as shown in Table D.1. As the Table shows, our AraT5 MSA outperform mT5 in all but one condition (where our model acquires marginally less performance). We also performed similar evaluation on the merged Dev sets of all MSA and dialectal Arabic MT datasets in the Arabic  English direction. We do not show related results here, but we note our AraT5 MSA outperforms mT5 on all conditions. We also perform qualitative analyses of the outputs of several of our models, including as to length of MT source data (Appendix D). In particular, our analyses are for the following tasks: machine translation, code-switched translation, paraphrasing, transliteration, and news title generation. MT Model. Table D.2 (Appendix) shows three examples of Arabic  English MT models. Sentence (1) is in MSA source , sentence (2) is in Levantine Arabic source, and sentence (3) is in Egyptian source. In all three examples, one or more of our models generate(s) more fluent translations than mT5. This includes ability of our models to translate dialectal sentences where mT5 seems to struggle (e.g., mT5 is not able to translate the equivalents of drive" from Egyptian Arabic). Code-Switched Translation Model. Table 7 shows two code-switched examples from ARGENCS . Sentence (1) is Algerian dialect at source translated into French, while sentence (2) is Jordanian dialect translated into English. In both cases, our models not only handle the dialects but also their use in code-switched contexts better than mT5. Paraphrasing, Transliteration, and Title Generation. Each of Tables D.3, D.4, and D.5 (Ap-pendix D) shows two output samples from our paraphrasing, transliteration, and title generation models, respectively. In each case, the samples are high-quality, informative, and fluent. Our paraphrase samples also tightly capture the meaning of the source sentences. Multilingual LMs. mBERT is the multilingual version of BERT (Devlin et al., 2019), which is an encoder model with bidirectional representations from Transformers trained with a denoising objective. mBERT is trained on Wikipedia for 104 languages, including Arabic. XLM-R (Conneau et al., 2020) is also a Transformer-based multilingual masked language model pre-trained on more than 2 TB of CommonCrawl (CC) data in 100 languages, including Arabic ( 2 . 9 B tokens). XLM-R model uses the same masking objective as BERT, but not the next sentence prediction. mT5 (Xue et al., 2020) is the multilingual version of T ext-t oT ext T ransfer T ransformer model (T5) (Raffel et al., 2019). T5 is an encoder-decoder Transformer similar in configuration and size to a BERT Base . It is trained on mC4, which is  26 . 76 TB for 101 languages generated from 71 CC dumps. Arabic LMs. AraBERT (Antoun et al., 2020) is an Arabic pre-trained language model based on the BERT Base architecture with 24GB of MSA data. ARBERT and MARBERT (Abdul-Mageed et al., 2021) are two BERT-based models, with the first focused on MSA ( 61 GB) and the second on both MSA and dialects ( 128 GB). MARBERT achieves SOTA on most Arabic NLU tasks. QARiB (Abde-lali et al., 2021) is similarly a BERT-based model covering both MSA and dialects. CamelBERT (In-oue et al., 2021) is also a BERT-based model pre-trained with MSA, dialectal, and classical Arabic. We introduced three powerful Arabic-specific text-to-text Transformer models trained on large MSA and/or Arabic dialectal data. We also introduced ARGEN, a unified benchmark for Arabic Natural Language generation evaluation composed of seven tasks collected from a total of 19 datasets. Our models outperform mT5 on all ARGEN tasks ( 52 out of 59 test sets, i.e., 88 . 14% ). This is true even for MT involving four foreign languages from which the models have seen marginal or no pretraining data (i.e., zeroand few-shot pre-training). Our models also set new SOTA on the large Arabic language understanding evaluation benchmark ARLUE. Our models involve vocabulary from 11 languages other than Arabic, and hence can easily be further pre-trained/fine-tuned in these languages. Our models are publicly available, and ARGEN datasets are accessible from our repository. We gratefully acknowledge support from the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435-2018-0576; 895-2020-1004), Canadian Foundation for Innovation (CFI; 37771), Compute Canada (CC), 17 , UBC ARC-Sockeye, 18 and Advanced Micro Devices, Inc. (AMD). We thank the Google TFRC program for providing us with free TPU access. 19 Any opinions, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NSERC, SSHRC, CFI, CC, ARC-Sockeye, AMD, or Google. We thank Bashar Talafha for help with code-switching data preparation. Energy efficiency. Our models, similar to many deep learning language models, take significant pre-training time and are not energy efficient. We acknowledge this important issue and believe work on creating energy efficient models should receive scholarly attention. Data. Our pre-training datasets are collected from the public domain and cover diverse communities. As we have demonstrated, our resulting models are better equipped to power applications involving several varieties of Arabic as well as code-switched language use involving Arabic. From this perspective, we hope they add to ongoing efforts in the community to design models that are fairer and more representative. ARGEN Benchmark Release. We design ARGEN using both existing datasets and new datasets that we create for this work. In our accompanying GitHub repository, we link to all existing publicly available components of the benchmark with standard splits from source as well as components that can be acquired from data organizations. In addition, we released all the new datasets we have developed. While we have prioritized standardizing evaluation on as many unified and consolidated datasets and tasks as possible, we also report performance on individual test sets so as to enable the community to replicate our work even on particular parts or tasks of ARGEN if they so wish. AraT5 Models Release. All our pre-trained models are publicly available for non-malicious use. We acknowledge our models may still be misused in real world. However, we hope the models will be deployed in domains such as education, disaster management, health, recreation, travel, etc. in socially beneficial ways. These meaningful potential use cases are behind our decision to release the models.
abstain abstain objective objective method objective abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain objective method objective method method objective abstain objective method objective objective objective objective objective objective method objective method result method abstain objective abstain method abstain method abstain method method abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain method method method method method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method other abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain method abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain method abstain method abstain abstain method method abstain method abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method abstain abstain method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain method abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other other other other other other method method result abstain objective method abstain other other other other abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain