Pretrained multilingual models are able to perform cross-lingual transfer in a zero-shot setting, even for languages unseen during pretraining. However, prior work evaluating performance on unseen languages has largely been limited to low-level, syntactic tasks, and it remains unclear if zero-shot learning of high-level, semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, an extension of XNLI (Conneau et al., 2018) to 10 Indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. Additionally, we explore model adaptation via continued pretraining and provide an analysis of the dataset by considering hypothesis-only models. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.48%. Continued pretraining offers improvements, with an average accuracy of 43.85%. Surprisingly, training on poorly translated data by far outperforms all other methods with an accuracy of 49.12%. Pretrained multilingual models such as XLM (Lample and Conneau, 2019), multilingual BERT (mBERT; Devlin et al., 2019), and XLM-R (Con-neau et al., 2020) achieve strong cross-lingual transfer results for many languages and natural language processing (NLP) tasks. However, there exists a discrepancy in terms of zero-shot performance between languages present in the pretraining data and those that are not: performance is generally highest for well-represented languages and decreases with less representation. Yet, even for unseen languages, performance is generally above chance, and model adaptation approaches have been shown to yield Language ISO Family Dev Test Aymara aym Aymaran 743 750 Ashninka cni Arawak 658 750 Bribri bzd Chibchan 743 750 Guaran gn Tupi-Guaran 743 750 Nahuatl nah Uto-Aztecan 376 738 Otom oto Oto-Manguean 222 748 Quechua quy Quechuan 743 750 Rarmuri tar Uto-Aztecan 743 750 Shipibo-Konibo shp Panoan 743 750 Wixarika hch Uto-Aztecan 743 750 Table 1: The languages in AmericasNLI, along with their ISO codes, language families, and dataset sizes. further improvements (Muller et al., 2020; Pfeiffer et al., 2020a,b; Wang et al., 2020). Importantly, however, there are currently no datasets for high-level, semantic tasks which focus solely on low-resource languages. As these languages are most likely to be unseen to commonly used pretrained models, practically all work evaluating unseen language performance and language adaptation methods has been limited to low-level, syntactic tasks such as part-of-speech tagging, dependency parsing, and named-entity recognition (Muller et al., 2020; Wang et al., 2020). This largely limits our ability to draw more general conclusions with regards to the zero-shot learning abilities of pretrained multilingual models for unseen languages. In this work, we introduce AmericasNLI, an extension of XNLI (Conneau et al., 2018)  a natural language inference (NLI; cf. 2.3) dataset covering 15 high-resource languages  to 10 Indigenous languages spoken in the Americas: Ashninka, Aymara, Bribri, Guaran, Nahuatl, Otom, Quechua, Rarmuri, Shipibo-Konibo, and Wixarika. All of them are truly low-resource languages: they have little to no digitally available labeled or unlabeled 6279 data, and they are not typically studied by the mainstream NLP community. The goal of this work is two-fold: First, we hope to increase the visibility of these languages by providing a portion of the resources necessary for NLP research. Second, we aim to allow for a more comprehensive study of multilingual model performance on unseen languages, where improvements will help extend the reach of NLP techniques to a larger set of languages. We are specifically interested in the following research questions: (1) Do pretrained multilingual models still perform above random chance for a high-level, semantic task in an unseen language? (2) Do methods aimed at adapting models to unseen languages  previously exclusively evaluated on low-level, syntactic tasks  also increase performance on NLI? (3) Are translation-based approaches effective for truly low-resource languages, where translation quality is typically very poor? 1 We experiment with XLM-R, both with and without model adaptation via continued pretraining on monolingual corpora in the target language. Our results show that the performance of XLM-R out-of-the-box is moderately above chance, and model adaptation leads to improvements of up to 5.86 percentage points. Training on machine-translated training data, however, results in an even larger performance gain of 11.13 percentage points over the corresponding XLM-R model without adaptation. We further perform an analysis via experiments with hypothesis-only models, to examine potential artifacts which may have been inherited from XNLI and find that performance is above chance for most models, but still below that for using the full example. AmericasNLI is publicly available 2 and we hope that it will serve as a benchmark for measuring the zero-shot natural language understanding abilities of multilingual models for unseen languages. Additionally, we hope that our dataset will motivate the development of novel pretraining and model adaptation techniques which are suitable for truly low-resource languages. Prior to the widespread use of pretrained transformer models, cross-lingual transfer was mainly achieved through word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017), either by aligning monolingual embeddings into the same embedding space (Lample et al., 2018b,a; Grave et al., 2018) or by training multilingual embeddings (Ammar et al., 2016; Artetxe and Schwenk, 2019). Pretrained multilingual models represent the extension of multilingual embeddings to pretrained transformer models. These models follow the standard pretraining finetuning paradigm: they are first trained on unlabeled monolingual corpora from various languages (the pretraining languages ) and later finetuned on target-task data in a  usually high-resource  source language. Having been exposed to a variety of languages through this training setup, cross-lingual transfer results for these models are competitive with the state of the art for many languages and tasks. Commonly used models are mBERT (Devlin et al., 2019), which is pretrained on the Wikipedias of 104 languages with masked language modeling (MLM) and next sentence prediction (NSP), and XLM, which is trained on 15 languages and introduces the translation language modeling objective, which is based on MLM, but uses pairs of parallel sentences. XLM-R has improved performance over XLM, and trains on data from 100 different languages with only the MLM objective. Common to all models is a large shared subword vocabulary created using either BPE (Sen-nrich et al., 2016) or SentencePiece (Kudo and Richardson, 2018) tokenization. Just as in the monolingual setting, where benchmarks such as GLUE (Wang et al., 2018) and Su-perGLUE (Wang et al., 2019) provide a look into the performance of models across various tasks, multilingual benchmarks (Hu et al., 2020; Liang et al., 2020) cover a wide variety of tasks involving sentence structure, classification, retrieval, and question answering. Additional work has been done examining what mechanisms allow multilingual models to transfer across languages (Pires et al., 2019; Wu and Dredze, 2019). Wu and Dredze (2020) examine transfer performance dependent on a language's representation in the pretraining data. For languages with low representation, multiple methods have been proposed to improve performance, in-6280 cluding extending the vocabulary, transliterating the target text, and continuing pretraining before finetuning (Lauscher et al., 2020; Chau et al., 2020; Muller et al., 2020; Pfeiffer et al., 2020a,b; Wang et al., 2020). In this work, we focus on continued pretraining to analyze the performance of model adaptation for a high-level, semantic task. Given two sentences, the premise and the hypothesis , the task of NLI consists of determining whether the hypothesis logically entails, contradicts, or is neutral to the premise. The most widely used datasets for NLI in English are SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). XNLI (Conneau et al., 2018) is the multilingual expansion of MNLI to 15 languages, providing manually translated evaluation sets and machine-translated training sets. While datasets for NLI or the similar task of recognizing textual entailment exist for other languages (Bos et al., 2009; Alab-bas, 2013; Eichler et al., 2014; Amirkhani et al., 2020), their lack of similarity prevents a generalized study of cross-lingual zero-shot performance. This is in contrast to XNLI, where examples for all 15 languages are parallel. To preserve this property of XNLI, when creating AmericasNLI, we choose to translate Spanish XNLI as opposed to creating examples directly in the target language. However, NLI datasets are not without issue: Gururangan et al. (2018) show that artifacts from the creation of MNLI allow for models to classify examples depending on only the hypothesis, showing that models may not be reasoning as expected. Motivated by this, we provide further analysis of AmericasNLI in Section 6 by comparing the performance of hypothesis-only models to models trained on full examples. AmericasNLI is the translation of a subset of XNLI (Conneau et al., 2018). As translators between Spanish and the target languages are more frequently available than those for English, we translate from the Spanish version. Additionally, some translators reported that code-switching is often used to describe certain topics, and, while many words without an exact equivalence in the target language are worked in through translation or interpretation, others are kept in Spanish. To minimize the amount of Spanish vocabulary in the translated examples, we choose sentences from genres that we judged to be relatively easy to translate into the target languages: face-to-face, letters, and telephone. We choose up to 750 examples from each of the development and test set, with exact counts for each language in Table 1. 3.2 Languages We now discuss the languages in AmericasNLI. For additional background on previous NLP research on Indigenous languages of the Americas, we refer the reader to Mager et al. (2018). A summary of this information can be found in Table C.1. Aymara Aymara is a polysynthetic Amerindian language spoken in Bolivia, Chile, and Peru by over two million people (Homola, 2012). Aymara follows an SOV word order and has multiple dialects, including Northern and Southern Aymara, spoken on the southern Peruvian shore of Lake Titicaca as well as around La Paz and, respectively, in the eastern half of the Iquique province in northern Chile, the Bolivian department of Oruro, in northern Potosi, and southwest Cochabamba. AmericasNLI examples are translated into the Central Aymara variant, specifically Aymara La Paz. Ashninka Ashninka is an Amazonian language from the Arawak family, spoken by 73,567 people 3 in Central and Eastern Peru, in a geographical region located between the eastern foothills of the Andes and the western fringe of the Amazon basin (Mihas, 2017). Ashninka is an agglutinating and polysynthetic language with a VSO word order. Bribri Bribri is a Chibchan language spoken by 7,000 people in Southern Costa Rica (INEC, 2011). It has three dialects, and while it is still spoken by children, it is currently a vulnerable language (Moseley, 2010; Snchez Avendao, 2013). Bribri is a tonal language with SOV word order. There are several orthographies which use different diacritics for the same phenomena, however even for researchers who use the same orthography, the Uni-code encoding of similar diacritics differs amongst authors. Furthermore, the dialects of Bribri differ in their exact vocabularies, and there are phonological processes, like the deletion of unstressed vowels, which also change the tokens found in texts. As 3 https://bdpi.cultura.gob.pe/pueblos/ ashaninka 6281 Language Premise Hypothesis en And he said, Mama, I'm home. Bribri has only been a written language for about 40 years, existing materials have a large degree of idiosyncratic variation. These variations are standardized in AmericasNLI, which is written in the Amubri variant. Guaran Guaran is spoken by between 6 to 10 million people in South America and roughly 3 million people use it as their main language, including more than 10 native nations in Paraguay, Brazil, Argentina, and Bolivia, along with Paraguayan, Argentinian, and Brazilian peoples. According to the Paraguayan Census, in 2002 there were around 1.35 million monolingual speakers, which has since increased to around 1.5 million people (Dos Santos, 2017; Meli, 1992). 4 Although the use of Guaran as spoken language is much older, the first written record dates to 1591 (Catechism) followed by the first dictionary in 1639 and linguistic descriptions in 1640. The official grammar of Guaran was approved in 2018. Guaran is an agglutinative language, with ample use of prefixes and suffixes. Nahuatl Nahuatl belongs to the Nahuan subdivision of the Uto-Aztecan language family. There are 30 recognized variants of Nahuatl spoken by over 1.5 million speakers across Mexico, where Nahuatl is recognized as an official language (SEGOB, 2020b). Nahuatl is polysynthetic and agglutinative, and many sentences have an SVO word order or, for contrast and focus, a VSO order, and for emphasis, an SOV order (MacSwan, 1998). The 4 https://www.ine.gov.py/news/ 25-de-agosto-dia-del-Idioma-Guarani.php translations in AmericasNLI belong to the Central Nahuatl (Nhuatl de la Huasteca) dialect. As there is a lack of consensus regarding the orthographic standard, the orthography is normalized to a version similar to Classical Nahuatl. Otom Otom belongs to the Oto-Pamean language family and has nine linguistic variants with different regional self-denominations. Otom is a tonal language following an SVO order, and there are around 307,928 speakers spread across 7 Mexican states. In the state of Tlaxcala, the yuhmu or uhmu variant is spoken by fewer than 100 speakers, and we use this variant for the Otom examples in AmericasNLI. Quechua Quechua, or Runasimi , is an Indigenous language family spoken primarily in the Peruvian Andes. It is the most widely spoken pre-Columbian language family of the Americas, with around 8-10 million speakers. Approximately 25% (7.7 million) of Peruvians speak a Quechuan language, and it is the co-official language in many regions of Peru. There are multiple subdivisions of Quechua , and AmericasNLI examples are translated into the standard version of Southern Quechua, Quechua Chanka, also known as Quechua Ayacucho, which is spoken in different regions of Peru and can be understood in different areas of other countries, such as Bolivia or Argentina. In AmericasNLI, the apostrophe and pentavocalism from other regions are not used. to the Taracahitan subgroup of the Uto-Aztecan language family (Goddard, 1996), and is polysynthetic and agglutinative. Rarmuri is an official language of Mexico, spoken mainly in the Sierra Madre Occidental region by a total of 89,503 speakers (SEGOB, 2020c). AmericasNLI examples are translated into the Highlands variant (IN-ALI, 2009), and translation orthography and word boundaries are similar to Caballero (2008). Shipibo-Konibo Shipibo-Konibo is a Panoan language spoken by around 35,000 native speakers in the Amazon region of Peru. Shipibo-Konibo uses an SOV word order (Faust, 1973) and postpositions (Vasquez et al., 2018). The translations in AmericasNLI make use of the official alphabet and standard writing supported by the Ministry of Education in Peru. Wixarika The Wixarika, or Huichol , language, meaning the language of the doctors and healers (Lumholtz, 2011), is a language in the Cora-chol subgroup of the Uto-Aztecan language family (Campbell, 2000). Wixarika is a national language of Mexico with four variants , spoken by a total of around 47,625 speakers (SEGOB, 2020a). Wixarika is a polysynthetic language and follows an SOV word order. Translations in AmericasNLI are in Northern Wixarika and use an orthography common among native speakers (Mager-Hois, 2017). In this section, we detail the experimental setup we use to evaluate the performance of various approaches on AmericasNLI. Pretrained Model We use XLM-R (Conneau et al., 2020) as the pretrained multilingual model in our experiments. The architecture of XLM-R is based on RoBERTa (Liu et al., 2019), and it is trained using MLM on web-crawled data in 100 languages. It uses a shared vocabulary consisting of 250k subwords, created using SentencePiece (Kudo and Richardson, 2018) tokenization. We use the Base version of XLM-R for our experiments. Adaptation Methods To adapt XLM-R to the various target languages, we continue training with the MLM objective on monolingual text in the target language before finetuning. To keep a fair comparison with other approaches, we only use target data which was also used to train the translation models, which we describe in Section 4.2. However, we note that one benefit of continued pretraining for adaptation is that it does not require parallel text, and could therefore benefit from text which could not be used for a translation-based approach. For continued pretraining, we use a batch size of 32 and a learning rate of 2e-5. We train for a total of 40 epochs. Each adapted model starts from the same version of XLM-R, and is adapted individually to each target language, which leads to a different model for each language. We denote models adapted with continued pretraining as +MLM . Finetuning To finetune XLM-R, we follow the approach of Devlin et al. (2019) and use an additional linear layer. We train on either the English MNLI data or the machine-translated Spanish data, and we call the final models XLM-R (en) and XLM-R (es), respectively. Following Hu et al. (2020), we use a batch size of 32 and a learning rate of 2e-5. We train for a maximum of 5 epochs, and evaluate performance every 2500 steps on the XNLI development set. We employ early stopping with a patience of 15 evaluation steps and use the best performing checkpoint for the final evaluation. All finetuning is done using the Huggingface Transformers library (Wolf et al., 2020) with up to two Nvidia V100 GPUs. Using Lacoste et al. (2019), we estimate total carbon emissions to be 75.6 kgCO 2 eq. We also experiment with two translation-based approaches, translate-train and translate-test, detailed below along with the translation model used. Translation Models For our translation-based approaches, we train two sets of translation models: one to translate from Spanish into the target language, and one in the opposite direction. We use transformer sequence-to-sequence models (Vaswani et al., 2017) with the hyperparameters proposed by Guzmn et al. (2019). Parallel data used to train the translation models can be found in Table B.1. We employ the same model architecture for both translation directions, and we measure translation quality in terms of BLEU (Papineni et al., 2002) and ChrF (Popovic, 2015), cf. Table 3. We use fairseq (Ott et al., 2019) to implement all translation models. 5 Translate-train For the translate-train approach, the Spanish training data provided by XNLI is translated into each target language. It is then used to finetune XLM-R for each language individually. Along with the training data, we also translate the Spanish development data, which is used for validation and early stopping. We discuss the effects of using a translated development set in Section F.1. Notably, we find that the finetuning hyperparameters defined above do not reliably allow the model to converge for many of the target languages. To 5 The code for translation models can be found at https: //github.com/AmericasNLP/americasnlp2021 find suitable hyperparameters, we tune the batch size and learning rate by conducting a grid search over {5e-6, 2e-5, 1e-4} for the learning rate and {32, 64, 128} for the batch size. In order to select hyperparameters which work well across all languages, we evaluate each run using the average performance on the machine-translated Aymara and Guaran development sets, as these languages have moderate and high ChrF scores, respectively. We find that decreasing the learning rate to 5e-6 and keeping the batch size at 32 yields the best performance. Other than the learning rate, we use the same approach as for zero-shot finetuning. Translate-test For the translate-test approach, we translate the test sets of each target language into Spanish. This allows us to apply the model finetuned on Spanish, XLM-R (es), to each test set. Additionally, a benefit of translate-test over translate-train and the adapted XLM-R models is that we only need to finetune once overall, as opposed to once per language. For evaluation, we use the checkpoint with the highest performance on the Spanish XNLI development set. Zero-shot Models We present our results in Table 4. Results for the development set are presented in Table E.1. Zero-shot performance is low for all 10 languages, with an average accuracy of 38.48% and 37.99% for the English and Spanish model, respectively. However, in all cases the performance is higher than the majority baseline. As shown in 6284 FT aym bzd cni gn hch nah oto quy shp tar Avg. Table E.3 in the appendix, the same models achieve an average of 74.20% and 75.35% accuracy respectively, when evaluated on the 15 XNLI languages. Interestingly, even though code-switching with Spanish is encountered in many target languages, finetuning on Spanish labeled data on average slightly underperforms the model trained on English, however performance is better for 3 of the languages. The English model achieves a highest accuracy of 42.59%, when evaluated on Nahuatl, while the Spanish model achieves a highest accuracy of 39.51%, when evaluated on Quechua. The lowest performance is achieved when evaluating on Aymara and Rarmuri, for the English and Spanish model, respectively. We find that model adaptation via continued pretraining improves both models, with an average gain of 5.22 percentage points for English and 5.86 percentage points for Spanish. Notably, continued pretraining increases performance for Quechua by 24.53 percentage points when finetuning on English, and 22.89 points when finetuning on Spanish. Performance decreases for Bribri and Otom when finetuning on English, however performance for all languages improves when using Spanish. Translate-test Performance of the translate-test model improves over both zero-shot baselines. We see the largest increase in performance for Guaran and Quechua, with gains of 7.16 and, respectively, 11.87 points over the best performing zero-shot model without adaptation. Considering the translation metrics in Table 3, models for Guaran and Quechua achieve the two highest scores for both metrics. On average, translate-test does worse when compared to the adapted zero-shot models, and in all but two cases, both adapted models perform better than translate-test. We hypothesize that translate-test is more sensitive to noise in the translated data; sentences may lose too much of their original content, preventing correct classification. Translate-train The most surprising result is that of translate-train, which considerably outperforms the performance of translate-test for all languages, and outperforms the zero-shot models for all but two languages. Compared to the best non-adapted zero-shot model, the largest performance gain is 20.40 points for Quechua. For the language with the lowest performance, Otom, translate-train performs 2.32 points worse than zero-shot; however, it still outperforms translate-test. When averaged across all languages, translate-train outperforms the English zero-shot model by 10.64 points, and translate-test by 8.9 points. It is important to note that the translation performance from Spanish to each target language is not particularly high: when considering ChrF scores, the highest is 0.33, and the highest BLEU score is 3.26. Performance of both translation-based models is correlated with ChrF scores, with a Pearson correlation coefficient of 0.82 and 0.83 for translate-train and translate-test. Correlations are not as strong for BLEU, with coefficients of 0.37 and 0.59. The sizable difference in performance between translate-train and the other methods suggests that translation-based approaches may be a valuable asset for cross-lingual transfer, especially for low-6285 resource languages. While the largest downsides to this approach are the requirement for parallel data and the need for multiple models, the potential performance gain over other approaches may prove worthwhile. Additionally, we believe that the performance of both translation-based approaches would improve given a stronger translation system, and future work detailing the necessary level of translation quality for the best performance would offer great practical usefulness for NLP applications for low-resource languages. As shown by Gururangan et al. (2018), SNLI and MNLI  the datasets AmericasNLI is based on  contain artifacts created during the annotation process which models exploit to artificially inflate performance. To analyze whether similar artifacts exist in AmericasNLI and if they can also be exploited, we train and evaluate models using only the hypothesis, and present results in Table 5. We can see that the average performance across languages is better than chance for all models except for XLM-R without adaptation. Translate-train obtains the highest result with 44 . 23% accuracy, and as shown in Table E.2, hypothesis-only performance of translate-test is higher than standard performance for 5 languages. Thus, as with SNLI and MNLI, artifacts in the hypotheses can be used to predict, to some extent, the correct labels. However all but 1 zero-shot and translate-train models perform better in the standard setting, indicating that the models are learning something beyond just exploiting artifacts in the hypotheses, even with the additional challenge of unseen languages. Following Conneau et al. (2018), AmericasNLI was created by translating sentences individually, in order to prevent additional context being added into the hypotheses. However, this strategy may break the original semantic relationship between the premise and the hypothesis. Furthermore, for some examples the logical relationship may be dependent on context or subtext which can be lost through translation, or simply not make sense in the target language. To verify the validity of the labels of AmericasNLI, we conduct a human evaluation experiment, focusing on examples translated to Bribri. We create a balanced, random sample of 450 examples taken from the Bribri development set. An annotator familiar with the task was then asked to classify the pairs of sentences. For comparison, we also annotate parallel examples taken from the English and Spanish development sets. For Bribri, we recover the original XNLI label for 76.44% of examples. For English and Spanish, we achieve 81.78% and 71.56% accuracy, respectively. Due to the relatively small differences in performance across languages, we conclude that translation to Bribri has a minimal effect on the semantic relationship between the premise and the hypothesis. While the case study above provides strong evidence for the validity of our Bribri examples, we cannot currently generalize this claim to the remaining languages. For future work, we plan on extending our human evaluation to more languages and provide a more detailed analysis. Additionally, due to the limited availability of annotators and the difficulties of translation for languages that are less frequently studied, the size of the AmericasNLI test set is relatively small. As such, care must be taken to carefully evaluate conclusions drawn using the dataset; following Card et al. (2020) we present a power analysis of our results in Section D.1. Future work expanding the dataset size will help create a stronger baseline. Furthermore, while we do not make any model-specific assumptions in our experiments, our results are based on only one pretrained model and adaptation method. Methods using vocabulary extension or adapters may offer additional improvements. Similarly, other pretrained models could perform differently, depending on, e.g., the model size or the set of languages in their pretraining data. In Table F.3, we present results using XLM-R Large, and find that, while the relationship between the approaches differs from the main experiments, the overall highest average performance is still achieved by the translate-train approach with XLM-R Base. We provide a longer discussion in Section F.3. To better understand the zero-shot abilities of pretrained multilingual models for semantic tasks in unseen languages, we present AmericasNLI, a parallel NLI dataset covering 10 low-resource lan-6286 guages indigenous to the Americas. We conduct experiments with XLM-R, and find that the model's zero-shot performance, while better than a majority baseline, is poor. However, it can be improved by model adaptation via continued pretraining. Additionally, we find that translation-based approaches outperform a zero-shot approach, which is surprising given the low quality of the employed translation systems. We hope that this work will not only spur further research into improving model adaptation to unseen languages, but also motivate the creation of more resources for languages not frequently studied by the NLP community. In this work, we present a new dataset created through the translation of an existing resource, XNLI (Conneau et al., 2018). While this allows for results that are directly comparable, it also means that this dataset inherits any biases and flaws which are contained in the previous dataset. Furthermore, research involving languages spoken by Indigenous communities raises ethical concerns regarding the exploitation of these languages and communities: it is crucial that members of the community are able to directly benefit from the research. Translation for AmericasNLI was done by either paper authors or translators who were compensated at a rate based on the average rate for translation and the minimum wage in their country of residence. Additionally, many authors are members of, and/or have a record of close work with communities who speak a language contained in AmericasNLI. We thank the following people for their work on the translations: Francisco Morales for Bribri, Feliciano Torres Ros for Ashninka, Perla Alvarez Britez for Guaran, Silvino Gonzlez de la Crz for Wixarika, Giovany Martnez Sebastin, Pedro Kapoltitan, and Jos Antonio for Nahuatl, Jos Mateo Lino Cajero Velzquez for Otom, Liz Chvez for Shipibo-Konibo, and Mara del Crmen Sotelo Holgun for Rarmuri. We would also like to thank Dallas Card for his help with power analysis. This work would not have been possible without the financial support of Facebook AI Research, Microsoft Research, Google Research, the Institute of Computational Linguistics at the University of Zurich, the NAACL Emerging Regions Fund, Comunidad Elotl, and Snorkel AI.
abstain abstain abstain method objective result abstain abstain abstain abstain other abstain abstain abstain method abstain abstain objective objective method abstain abstain method result abstain result method objective other other other other other other other other other other other other method other other other other other method other objective abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method abstain abstain method method result other method result method method method method method result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain result abstain result objective abstain abstain abstain abstain abstain other other other