Pre-trained language models (LMs) are currently integral to many natural language processing systems. Although multilingual LMs were also introduced to serve many languages, these have limitations such as being costly at inference time and the size and diversity of non-English data involved in their pre-training. We remedy these issues for a collection of diverse Arabic varieties by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT. To evaluate our models, we also introduce ARLUE, a new benchmark for multi-dialectal Arabic language understanding evaluation. ARLUE is built using 42 datasets targeting six different task clusters, allowing us to offer a series of standardized experiments under rich conditions. When fine-tuned on ARLUE, our models collectively achieve new state-of-the-art results across the majority of tasks ( 37 out of 48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score ( 77 . 40 ) across all six task clusters, outperforming all other models including XLM-R Large (  3 . 4  larger size). Our models are publicly available at https://github.com/UBC-NLP/marbert and ARLUE will be released through the same repository. Language models (LMs) exploiting self-supervised learning such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019a) have recently emerged as powerful transfer learning tools that help improve a very wide range of natural language processing (NLP) tasks. Multilingual LMs such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (XLM-R) (Conneau et al., 2020) have also been introduced, but are usually outperformed by monolingual models pre-trained with larger vocabulary and bigger language-specific datasets (Virtanen et al., 2019; Antoun et al., 2020; Dadas et al., 2020;  All authors contributed equally. de Vries et al., 2019; Le et al., 2020; Martin et al., 2020; Nguyen and Tuan Nguyen, 2020). Since LMs are costly to pre-train, it is important to keep in mind the end goals they will serve once developed. For example, (i) in addition to their utility on standard' data, it is useful to endow them with ability to excel on wider real world settings such as in social media. Some existing LMs do not meet this need since they were trained on datasets that do not sufficiently capture the nuances of social media language (e.g., frequent use of abbreviations, emoticons, and hashtags; playful character repetitions; neologisms and informal language). It is also desirable to build models able to (ii) serve diverse communities (e.g., speakers of dialects of a given language), rather than focusing only on mainstream varieties. In addition, once created, models should be (iii) usable in energy efficient scenarios. This means that, for example, medium-to-large models with competitive performance should be preferred to large-to-mega models. A related issue is (iv) how LMs are evaluated. Progress in NLP hinges on our ability to carry out meaningful comparisons across tasks, on carefully designed benchmarks. Although several benchmarks have been introduced to evaluate LMs, the majority of these are either exclusively in English (e.g., DecaNLP (McCann et al., 2018), GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019)) or use machine translation in their training splits (e.g., XTREME (Hu et al., 2020)). Again, useful as these benchmarks are, this circumvents our ability to measure progress in real-world settings (e.g., training and evaluation on native vs. translated data) for both cross-lingual NLP and in monolingual, non-English environments. Context. Our objective is to showcase a scenario where we build LMs that meet all four needs listed above. That is, we describe novel LMs that (i) excel across domains, including social media, (ii) can serve diverse communities, and (iii) perform well compared to larger (more energy hungry) models (iv) on a novel, standardized benchmark. We choose Arabic as the context for our work since it is a widely spoken language (  400 M native speakers), with a large number of diverse dialects differing among themselves and from the standard variety, Modern Standard Arabic (MSA). Arabic is also covered by the popular mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020), which provides us a setup for meaningful comparisons. That is, not only are we able to empirically measure monolingual vs. multilingual performance under robust conditions using our new benchmark, ARLUE, but we can also demonstrate how our base-sized models outperform (or at least are on par with) larger models (i.e., XLM-R Large , which is  3 . 4  larger than our models). In the context of our work, we also show how the currently best-performing model dedicated to Arabic, AraBERT (Antoun et al., 2020), suffers from a number of issues. These include (a) not making use of easily accessible data across domains and, more seriously, (b) limited ability to handle Arabic dialects and (c) narrow evaluation. We rectify all these limitations. Our contributions. With our stated goals in mind, we introduce ARBERT and MARBERT , two Arabic-focused LMs exploiting large-to-massive diverse datasets. For evaluation, we also introduce a novel AR abic natural L anguage U nderstanding E valuation benchmark ( ARLUE ). ARLUE is composed of 42 different datasets, making it by far the largest and most diverse Arabic NLP benchmark we know of. We arrange ARLUE into six coherent cluster tasks and methodically evaluate on each independent dataset as well as each cluster task, ultimately reporting a single ARLUE score. Our models establish new state-of-the-art (SOTA) on the majority of tasks, across all cluster tasks. Our goal is for ARLUE to serve the critical need for measuring progress on Arabic, and facilitate evaluation of multilingual and Arabic LMs. To summarize, we offer the following contributions: 1. We develop ARBERT and MARBERT , two novel Arabic-specific Transformer LMs pre-trained on very large and diverse datasets to facilitate transfer learning on MSA as well as Arabic dialects. on 42 datasets across six different Arabic language understanding cluster tasks, thereby facilitating measurement of progress on Arabic and multilingual NLP. 3. We fine-tune our new powerful models on ARLUE and provide an extensive set of comparisons to available models. Our models achieve new SOTA on all task clusters in 37 out of 48 individual datasets and a SOTA ARLUE score . The rest of the paper is organized as follows: In Section 2, we provide an overview of Arabic LMs. Section 3 describes our Arabic pre-tained models. We evaluate our models on downstream tasks in Section 4, and present our benchmark ARLUE and evaluation on it in Section 5. Section 6 is an overview of related work. We conclude in Section 7. We now introduce existing Arabic LMs. The term Arabic refers to a collection of languages, language varieties, and dialects. The standard variety of Arabic is MSA, and there exists a large number of dialects that are usually defined at the level of the region or country (Abdul-Mageed et al., 2020a, 2021a,b). A number of Arabic LMs has been developed. The most notable among these is AraBERT (Antoun et al., 2020), which is trained with the same architecture as BERT (De-vlin et al., 2019) and uses the BERT Base configuration. AraBERT is trained on 23 GB of Arabic text, making  70 M sentences and 3 B words, from Arabic Wikipedia, the Open Source International dataset (OSIAN) (Zeroual et al., 2019) ( 3 . 5 M news articles from 24 Arab countries), and 1 . 5 B words Corpus from El-Khair (2016) ( 5 M articles extracted from 10 news sources). Antoun et al. (2020) evaluate AraBERT on three Arabic downstream tasks. These are (1) sentiment analysis from six different datasets: HARD (Elnagar et al., 2018), ASTD (Nabil et al., 2015), ArsenTD-Lev (Baly et al., 2019), LABR (Aly and Atiya, 2013), and ArSaS (Elmadany et al., 2018). (2) NER, with the ANERcorp (Benajiba and Rosso, 2007), and (3) Arabic QA, on Arabic-SQuAD and ARCD (Mozannar et al., 2019) datasets. Another Arabic LM that was also introduced is ArabicBERT (Safaya et al., 2020), which is similarly based on BERT architecture. ArabicBERT was pre-trained on two datasets only, Arabic Wikipedia and Arabic OSACAR (Suarez et al., 2019). Since both of these datasets are already included in AraBERT, and Arabic OSACAR 1 has significant duplicates, we compare to AraBERT only. GigaBERT (Lan et al., 2020), an Arabic and English LM designed with code-switching data in mind, was also introduced. 2 3 Our Models 3.1 ARBERT 3.1.1 Training Data We train ARBERT on 61 GB of MSA text ( 6 . 5 B tokens) from the following sources:  Books (Hindawi) . We collect and preprocess 1 , 800 Arabic books from the public Arabic bookstore Hindawi. 3  El-Khair . This is a 5 M news articles dataset from 10 major news sources covering eight Arab countries from El-Khair (2016). Gigaword . We use Arabic Gigaword 5 th Edition from the Linguistic Data Consortium (LDC). 4 The dataset is a comprehensive archive of newswire text from multiple Arabic news sources. OSCAR . This is the MSA and Egyptian Arabic portion of the Open Super-large Crawled Almanach coRpus (Suarez et al., 2019), 5 a huge multilingual subset from Common Crawl 6 obtained using language identification and filtering. OSIAN . The Open Source International Arabic News Corpus (OSIAN) (Zeroual et al., 2019) consists of 3 . 5 million articles from 31 news sources in 24 Arab countries. Wikipedia Arabic . We download and use the December 2019 dump of Arabic Wikipedia. We use WikiExtractor 7 to extract articles and remove markup from the dump. 1 https://oscar-corpus.com. 2 Since GigaBERT is very recent, we could not compare to it. However, we note that our pre-training datasets are much larger (i.e., 15 . 6 B tokens for MARBERT vs. 4 . 3 B Arabic tokens for GigaBERT) and more diverse. 3 https://www.hindawi.org/books/. 4 https://catalog.ldc.upenn.edu/LDC2011T11. 5 https://oscar-corpus.com/. 6 https://commoncrawl.org. 7 https://github.com/attardi/wikiextractor. We provide relevant size and token count statistics about the datasets in Table 1. 3.1.2 Training Procedure Pre-processing. To prepare the raw data for pretraining, we perform light pre-processing. This helps retain a faithful representation of the naturally occurring text. We only remove diacritics and replace URLs, user mentions, and hashtags that may exist in any of the collections with the generic string tokens URL , USER , and HASHTAG , respectively. We do not perform any further preprocessing of the data before splitting the text off to wordPieces (Schuster and Nakajima, 2012). Multilingual models such as mBERT and XLM-R have 5 K (out of 110 K) and 14 K (out of 250 K) Arabic WordPieces, respectively, in their vocabularies. AraBERT employs a vocabulary of 60 K (out of 64 K). 8 For ARBERT, we use a larger vocabulary of 100 K WordPieces. For tokenization, we use the WordPiece tokenizer (Wu et al., 2016) provided by Devlin et al. (2019). Pre-training. For ARBERT, we follow Devlin et al. (2019)'s pre-training setup. To generate each training input sequence, we use the whole word masking, where 15% of the N input tokens are selected for replacement. These tokens are replaced 80% of the time with the [MASK] token, 10% with a random token, and 10% with the original token. We use the original implementation of BERT in the TensorFlow framework. 9 As mentioned, we use the same network architecture as BERT Base : 12 layers, 768 hidden units, 12 heads, for a total of  163 M parameters. We use a batch size of 256 sequences and a maximum sequence length of 128 tokens ( 256 sequences  128 tokens = 32 , 768 tokens/batch) for 8 M steps, which is approximately 42 epochs over the 6 . 5 B tokens. For all our models, we use a learning rate of 1 e  4 . We pre-train the model on one Google Cloud TPU with eight cores (v 2 . 8 ) from TensorFlow Research Cloud (TFRC). 10 Training took  16 days, for 42 epochs over all the tokens. Table 2 shows a comparison of ARBERT with mBERT, XLM-R, AraBERT, and MARBERT (see Section 3.2) in terms of data sources and size, vocabulary size, and model parameters. As we pointed out in Sections 1 and 2, Arabic has a large number of diverse dialects. Most of these dialects are under-studied due to rarity of resources. Multilingual models such as mBERT and XLM-R are trained on mostly MSA data, which is also the case for AraBERT and ARBERT. As such, these models are not best suited for downstream tasks involving dialectal Arabic. To treat this issue, we use a large Twitter dataset to pre-train a new model, MARBERT, from scratch as we describe next. To pre-train MARBERT, we randomly sample 1 B Arabic tweets from a large in-house dataset of about 6 B tweets. We only include tweets with at least three Arabic words, based on character string matching, regardless whether the tweet has non-Arabic string or not. That is, we do not remove non-Arabic so long as the tweet meets the three Arabic word criterion. The dataset makes up 128 GB of text ( 15 . 6 B tokens). Pre-training. We use the same network architecture as BERT Base , but without the next sentence prediction (NSP) objective since tweets are short. 11 We use the same vocabulary size ( 100 K wordPieces) as ARBERT, and MARBERT also has  160 M parameters. We train MARBERT for 17 M steps (  36 epochs) with a batch size of 256 and a maximum sequence length of 128 . Training took  40 days on one Google Cloud TPU (eight cores). We now present a comparison between our models and popular multilingual models as well as AraBERT. Our models compare to mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020) (base and large), and AraBERT (Antoun et al., 2020) in terms of training data size, vocabulary size, and overall model capacity as we summarize in Table 2. In terms of the actual Arabic variety involved, Devlin et al. (2019) train mBERT with Wikipedia Arabic data, which is MSA. XLM-R (Conneau et al., 2020) is trained on Common Crawl data, which likely involves a small amount of Arabic dialects. AraBERT is trained on MSA data only. ARBERT is trained on a large collection of MSA datasets. Unlike all other models, our MARBERT model is trained on Twitter data, which involves both MSA and diverse dialects. We now describe our fine-tuning setup. We evaluate our models by fine-tuning them on a wide range of tasks, which we thematically organize into six clusters: (1) sentiment analysis (SA), (2) social meaning (SM) (i.e., age and gender, dangerous and hateful speech, emotion, irony, and sar-casm), (3) topic classification (TC), (4) dialect identification (DI), (5) named entity recognition (NER), and (6) question answering (QA). For all classification tasks reported in this paper, we compare our models to four other models: mBERT, XLM-R Base , XLM-R Large , and AraBERT. We note that XLM-R Large is  3 . 4  larger than any of our own models (  550 M parameters vs.  160 M). We offer two main types of evaluation: on (i) individual tasks , which allows us to compare to other works on each individual dataset ( 48 classification tasks on 42 datasets), and (ii) ARLUE clusters (six task clusters). For all reported experiments, we follow the same light pre-processing we use for pre-training. For all individual tasks and ARLUE task clusters, we fine-tune on the respective training splits for 25 epochs, identifying the best epoch on development data, and reporting on both development and test data. 12 We typically use the exact data splits provided by original authors of each dataset. Whenever no clear 12 A minority of datasets came with no development split from source, and so we identify and report the best epoch only on test data for these. This allows us to compare all the models under the same conditions ( 25 epochs) and report a fair comparison to the respective original works. For all ARLUE cluster tasks, we identify the best epoch exclusively on our development sets (shown in Table 10). splits are available, or in cases where expensive cross-validation was used in source, we divide the data following a standard 80% training, 10% development, and 10% test split. For all experiments, whether on individual tasks or ARLUE task clusters, we use the Adam optimizer ( ? ) with input sequence length of 256 , a batch size of 32 , and a learning rate of 2 e  6 . These values were identified in initial experiments based on development data of a few tasks. 13 We now introduce individual tasks. Datasets. We fine-tune the language models on all publicly available SA datasets we could find in addition to those we acquired directly from authors. In total, we have the following 17 MSA and DA datasets: AJGT (Alo-mari et al., 2017), AraNET Sent (Abdul-Mageed et al., 2020b), AraSenTi-Tweet (Al-Twairesh et al., 2017), ArSarcasm Sent (Farha and Magdy, 2020), ArSAS (Elmadany et al., 2018), ArSenD-Lev (Baly et al., 2019), ASTD (Nabil et al., 2015), AWATIF (Abdul-Mageed and Diab, 2012), BBNS & SYTS (Salameh et al., 2015), CAMel Sent (Obeid et al., 2020), HARD (Elnagar et al., 2018), LABR (Aly and Atiya, 2013), Twitter Abdullah (Ab-dulla et al., 2013), Twitter Saad , 14 and SemEval-2017 (Rosenthal et al., 2017). Details about the datasets and their splits are in Section A.1. Baselines. We compare to the STOA listed in Table 3 and Table 4 captions. For all datasets with no baseline in Table 3, we consider AraBERT our baseline. Details about SA baselines are in Section A.2. Results. To facilitate comparison to previous works with the appropriate evaluation metrics, we 13 NER and QA are expetions, where we use sequence lengths of 128 and 384 , respectively; a batch sizes of 16 for both; and a learning rate of 2 e  6 and 3 e  5 , respectively. Table 4 : SA results (II) in Acc. SOTA by Antoun et al. (2020). split our results into two tables: We show results in F 1PN in Table 3 and F 1 in Table 4. We typically bold the best result on each dataset. Our models achieve best results in 13 out of the 17 classification tasks reported in the two tables combined , while XLM-R (which is a much larger model) outperforms our models in the 4 remaining tasks. We also note that XLM-R acquires better results than AraBERT in the majority of tasks, a trend that continues for the rest of tasks. Results also clearly show that MARBERT is more powerful than than ARBERT. This is due to MARBERT's larger and more diverse pre-training data, especially that many of the SA datasets involve dialects and come from social media. We collectively refer to a host of tasks as social meaning . These are age and gender detection; dangerous, hateful, and offensive speech detection; emotion detection; irony detection; and sarcasm detection. We now describe datasets we use for each of these tasks. Datasets. For both age and gender, we use Task(classes) SOTA mBERT XLM-RBXLM-RL AraBERT ARBERT MARBERT Age(3) 51 . Table 5 : Results on social meaning tasks. F 1 score is the evaluation metric. (cid:63) Hassan et al. (2020), (cid:63)(cid:63) Djandji et al. (2020),  Zhang and Abdul-Mageed (2019a),  ? ,  Farha and Magdy (2020),  Abdul-Mageed et al. (2020b). Arap-Tweet (Zaghouani and Charfi, 2018). We use AraDan (Alshehri et al., 2020) for dangerous speech. For offensive language and hate speech, we use the dataset released in the shared task (sub-tasks A and B) of offensive speech by Mubarak et al. (2020). We also use AraNET Emo (Abdul-Mageed et al., 2020b), IDAT@FIRE2019 (Ghanem et al., 2019), and ArSarcasm (Farha and Magdy, 2020) for emotion, irony and sarcasm, respectively. More information about these datasets and their splits is in Appendix B.1. Baselines. Baselines for social meaning tasks are the SOTA listed in Table 5 caption. Details about each baseline is in Appendix B.2. Results. As Table 5 shows, our models acquire best results on all eight tasks. Of these, MARBERT achieves best performance on seven tasks, while ARBERT is marginally better than MARBERT on one task (irony@FIRE2019). The sizeable gains MARBERT achieves reflects its superiority on social media tasks. On average, our models are 9 . 83 F 1 better than all previous SOTA. Classifying documents by topic is a classical task that still has practical utility. We use four TC datasets, as follows: Datasets. We fine-tune on Arabic News Text (ANT) (Chouigui et al., 2017) under three pre-taining settings ( title only , text only , and title+text .), Khaleej (Abbas et al., 2011), and OSAC (Saad and Ashour, 2010). Details about these datasets and the classes therein are in Appendix C.1. Baselines. Since, to the best of our knowledge, there are no published results exploiting deep learning on TC, we consider AraBERT a strong baseline. Results. As Table 6 shows, ARBERT acquires best results on both OSAC and Khaleej, and the title-only setting of ANT. AraBERT slightly outperforms our models on the text-only and title+text Dataset(classes) mBERT XLM-RBXLM-RL AraBERT ARBERT MARBERT ANTText(5) 84 . Arabic dialect identification can be performed at different levels of granularity, including binary (i.e., MSA-DA), regional (e.g., Gulf , Levantine ), country level (e.g., Algeria , Morocco ), and recently province level (e.g., the Egyptian province of Cairo , the Saudi province of Al-Madinah ) (Abdul-Mageed et al., 2020a, 2021b). Datasets. We fine-tune our models on the following datasets: Arabic Online Commentary (AOC) (Zaidan and Callison-Burch, 2014), ArSarcasm Dia (Farha and Magdy, 2020), 15 MADAR (sub-task 2) (Bouamor et al., 2019), NADI-2020 (Abdul-Mageed et al., 2020a), and QADI (Abdelali et al., 2020). Details about these datasets are in Table D.1. Baselines. Our baselines are marked in Table 7 caption. Details about the baselines are in Table D.2. Results. As Table 7 shows, our models outperform all SOTA as well as the baseline AraBERT across all classification levels with sizeable margins. These results reflect the powerful and diverse dialectal representation of MARBERT, enabling it to serve wider communities . Although ARBERT is developed mainly for MSA, it also outperforms all other models. We fine-tune the models on five NER datasets. Datasets. We use ACE03NW and ACE03BN (Mitchell et al., 2004), ACE04NW (Mitchell et al., 2004), ANERcorp (Benajiba and Rosso, 2007), and TW-NER (Darwish, 2013). Table E.1 shows the 15 ArSarcasm Dia carries regional dialect labels. Baseline. We compare our results with SOTA presented by Khalifa and Shaalan (2019) and follow them in focusing on person (PER), location (LOC) and organization (ORG) named entity labels while setting other labels to the unnamed entity (O). Details about Khalifa and Shaalan (2019) SOTA models are in Appendix E.2. Results. As Table 8 shows, our models outperform SOTA on two out of the five NER datasets. We note that even though SOTA (Khalifa and Shaalan, 2019) employ a complex combination of CNNs and character-level LSTMs, which may explain their better results on two datasets, MARBERT still achieves highest performance on the social media dataset (TW-NER) . Datasets. We use ARCD (Mozannar et al., 2019) and the three human translated Arabic test sections of the XTREME benchmark (Hu et al., 2020): MLQA (Lewis et al., 2020), XQuAD (Artetxe et al., 2020), and TyDi QA (Artetxe et al., 2020). Details about these datasets are in Table F.1. Baselines. We compare to Antoun et al. (2020) and consider their system a baseline on ARCD. We follow the same splits they used where we fine-tune on Arabic SQuAD (Mozannar et al., 2019) and 50% of ARCD and test on the remaining 50% of ARCD (ARCD-test). For all other experiments, we fine-tune on the Arabic machine translated SQuAD (AR-XTREME) from the XTREME multilingual benchmark (Hu et al., 2020) and test on the human translated test sets listed above. Our baselines in these is Hu et al. (2020)'s mBERT Base model on gold (human) data. Results. As is standard, we report QA results in terms of both Exact Match (EM) and F 1 . We find that results with ARBERT and MARBERT on QA are not competitive, a clear discrepancy from what we have observed thus far on other tasks. We hypothesize this is because the two models are pre-trained with a sequence length of only 128 , which does not allow them to sufficiently capture both a question and its likely answer within the same sequence window during the pre-training. 16 To rectify this, we further pre-train the stronger model, MARBERT, on the same MSA data as ARBERT in addition to AraNews dataset (Nagoudi et al., 2020) ( 8 . 6 GB), but with a bigger sequence length of 512 tokens for 40 epochs. We call this further pre-trained model MARBERT-v2 , noting it has 29 B tokens. As Table 9 shows, MARBERT-v2 acquires best performance on all but one test set , where XLM-R Large marginally outperforms us (only in F 1 ). We concatenate the corresponding splits of the individual datasets to form ARLUE , which is a conglomerate of task clusters. That is, we concatenate all training data from each group of tasks into a single TRAIN, all development into a single DEV, and all test into a single TEST. One exception is the social meaning tasks whose data we keep independent (see ARLUESM below). Table 10 shows a summary of the ARLUE datasets. 17 We now briefly describe how we merge individual datasets into ARLUE. ARLUE Senti . To construct ARLUE Senti , we collapse the labels very negative into negative , very positive into positive , and objective into neutral , and remove the mixed class. This gives us the 3 classes negative , positive , and neutral for ARLUE Senti . Details are in Table A.1. ARLUESM . We refer to the different social meaning datasets collectively as ARLUESM . We do not merge these datasets to preserve the conceptual coherence specific to each of the tasks. Details about individual datasets in ARLUESM are in B.1. ARLUE Topic. We straightforwardly merge the TC datasets to form ARLUE Topic , without modifying any class labels. Details of ARLUE Topic data are in Table C.1. ARLUE Dia. We construct three ARLUE Dia categories. Namely, we concatenate the AOC and AraSarcasm Dia MSA-DA classes to form ARLUE Dia-B (binary) and the region level classes from the same two datasets to acquire ARLUE Dia-R (4-classes, region ). We then merge the country 16 In addition, MARBERT is not trained on Wikipedia data from where some questions come. 17 Again, ARLUESM datasets are kept independent, but to provide a summary of all ARLUE datasets we collate the numbers in Table 10. Table 10 : ARLUE categories across the different data splits. (cid:63) Refer to Table B.1 for details about individual social meaning datasets in ARLUESM . Statistics are at the token level. Number of question-answer pairs. (21-classes, country ). Details are in Table D.1. ARLUENER & ARLUEQA . We straightforwardly concatenate all corresponding splits from the different NER and QA datasets to form ARLUENER and ARLUEQA , respectively. Details of each of these task clusters data are in Tables E.1 and F.1, respectively. We present results on each task cluster independently using the relevant metric for both the development split (Table 11) and test split (Table 12). Inspired by McCann et al. (2018) and Wang et al. (2018) who score NLP systems based on their performance on multiple datasets, we introduce an ARLUE score . The ARLUE score is simply the macro-average of the different scores across all task clusters, weighting each task equally. Following Wang et al. (2018), for tasks with multiple metrics (e.g., accuracy and F 1 ), we use an unweighted average of the metrics as the score for the task when computing the overall macro-average. As Table 12 shows, our MARBERT-v2 model achieves the highest ARLUE score ( 77 . 40 ) , followed by XLM-RL ( 76 . 55 ) and ARBERT ( 76 . 07 ). We also note that in spite of its superiority on social data, MARBERT ranks top 4 . This is due to MARBERT suffering on the QA tasks (due to its short input sequence length), and to a lesser extent on NER and TC. English and Multilingual LMs. Pre-trained LMs exploiting a self-supervised objective with masking such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b) have revolutionized NLP. Multilingual versions of these models such as mBERT and XLM-RoBERTa (Conneau et al., 2020) were also pre-trained. Other models with different objectives and/or architectures such as ALBERT (Lan et al., 2019), T5 (Raffel et al., 2020) and its multilingual version, mT5 (Xue et al., 2021), and GPT3 (Brown et al., 2020) were also introduced. More information about BERT-inspired LMs can be found in Rogers et al. (2020). Non-English LMs. Several models dedicated to individual languages other than English have been developed. These include AraBERT (An-toun et al., 2020) and ArabicBERT (Safaya et al., 2020) for Arabic, Bertje for Dutch (de Vries et al., 2019), CamemBERT (Martin et al., 2020) and FlauBERT (Le et al., 2020) for French, PhoBERT for Vietnamese (Nguyen and Tuan Nguyen, 2020), and the models presented by Virtanen et al. (2019) for Finnish, Dadas et al. (2020) for Polish, and Malmsten et al. (2020) for Swedish. Pyysalo et al. (2020) also create monolingual LMs for 42 languages exploiting Wikipedia data. Our models contributed to this growing work of dedicated LMs, and has the advantage of covering a wide range of dialects. Our MARBERT and MARBERT-v2 models are also trained with a massive scale social media dataset, endowing them with a remarkable ability for real-world downstream tasks. NLP Benchmarks. In recent years, several NLP benchmarks were designed for comparative evaluation of pre-trained LMs. For English, McCann et al. (2018) introduced NLP Decathlon (DecaNLP) which combines 10 common NLP datasets/tasks. Wang et al. (2018) proposed GLUE, a popular benchmark for evaluating nine NLP tasks. Wang et al. (2019) also presented SuperGLUE, a more challenging benchmark than GLUE covering seven tasks. In the cross-lingual setting, Hu et al. (2020) Dataset mBERT XLM-RBXLM-RL AraBERT ARBERT MARBERT MARBERT (v2) ARLUE Senti (cid:63) 79 . Table 11 : Performance of our models on the DEV splits of ARLUE. (cid:63) Metric for ARLUE Senti is F 1PN . ARLUESM results is the average score across the social meaning tasks described in Table B.2. Metric for ARLUEQA is Exact Match (EM) / F 1 . Table 12 : Performance of our models on the TEST splits of ARLUE (Acc / F 1 ). (cid:63) Metric for ARLUE Senti is Acc/ F 1PN . ARLUESM results is the average score across the social meaning tasks described in Table 5. Metric for ARLUEQA is Exact Match (EM) / F 1 . provide a Cross-lingual TRansfer Evaluation of Multilingual Encoders (XTREME) benchmark for the evaluation of cross-lingual transfer learning covering nine tasks for 40 languages ( 12 language families). ARLUE complements these benchmarking efforts, and is focused on Arabic and its dialects. ARLUE is also diverse (involves 42 datasets) and challenging (our best ARLUE score is at 77 . 40 ). We presented our efforts to develop two powerful Transformer-based language models for Arabic. Our models are trained on large-to-massive datasets that cover different domains and text genres, including social media. By pre-training MARBERT and MARBERT-v2 on dialectal Arabic, we aim at enabling downstream NLP technologies that serve wider and more diverse communities. Our best models perform better than (or on par with) XLM-R Large (  3 . 4  larger than our models), and hence are more energy efficient at inference time. Our models are also significantly better than AraBERT, the currently best-performing Arabic pre-trained LM. We also introduced AraLU, a large and diverse benchmark for Arabic NLU composed of 42 datasets thematically organized into six main task clusters. ARLUE fills a critical gap in Arabic and multilingual NLP, and promises to help propel innovation and facilitate meaningful comparisons in the field. Our models are publicly available. We also plan to publicly release our ARLUE benchmark. In the future, we plan to explore self-training our language models as a way to improve performance following Khalifa et al. (2021). We also plan to investigate developing more energy efficient models. We gratefully acknowledges support from the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, Canadian Foundation for Innovation, Compute Canada and UBC ARC-Sockeye ( https://doi.org/10.14288/SOCKEYE ). We also thank the Google TFRC program for providing us with free TPU access. Although our language models are pre-trained using datasets that were public at the time of collection, parts of these datasets might become private or get removed (e.g., tweets that are deleted by users). For this reason, we will not release or redistribute any of the pre-training datasets. Data coverage is another important consideration: Our datasets have wide coverage, and one of our contributions is offering models that can serve more diverse communities in better ways than existing models. However, our models may still carry biases that we have not tested for and hence we recommend they be used with caution. Finally, our models deliver better performance than larger-sized models and as such are more energy conserving. However, smaller models that can achieve simply good enough' results should also be desirable. This is part of our own future research, and the community at large is invited to develop novel methods that are more environment friendly.
abstain abstain abstain objective method objective result other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain objective abstain abstain abstain abstain objective method objective result abstain abstain abstain abstain abstain objective objective objective abstain method objective objective objective objective objective objective objective method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method other method abstain other other other other other method abstain method abstain method method abstain abstain method method abstain method method abstain method method method abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain method abstain method method method method method abstain method method abstain abstain abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain other other other other other other other other other objective method other other other other other other abstain other other other other abstain other other other other other other abstain objective method objective method method abstain abstain abstain abstain abstain objective other other method abstain abstain method abstain abstain abstain