We show that unsupervised sequence-segmentation performance can be transferred to extremely low-resource languages by pretraining a Masked Segmental Language Model (Downey et al., 2021) multilingually. Further, we show that this transfer can be achieved by training over a collection of low-resource languages that are typologically similar (but phylogenetically unrelated) to the target language. In our experiments, we transfer from a collection of 10 Indigenous American languages (AmericasNLP, Mager et al., 2021) to K'iche', a Mayan language. We compare our multilingual model to a monolingual (from-scratch) baseline, as well as a model pre-trained on Quechua only. We show that the multilingual pre-trained approach yields consistent segmentation quality across target dataset sizes, exceeding the monolingual baseline in 6/10 experimental settings. Our model yields especially strong results at small target sizes, including a zero-shot performance of 20.6 F1. These results have promising implications for low-resource NLP pipelines involving human-like linguistic units, such as the sparse transcription framework proposed by Bird (2020). Unsupervised sequence segmentation (at the word, morpheme, and phone level) has long been an area of interest in languages without whitespace-delimited orthography (e.g. Chinese, Uchiumi et al., 2015; Sun and Deng, 2018), morphologically complex languages without rule-based morphological analyzers (Creutz and Lagus, 2002), and automatically phone-transcribed speech data (Goldwater et al., 2009; Lane et al., 2021), respec- respecEqual contribution from starred authors, sorted by last name. Sincere thanks to: Gina-Anne Levow, Shane Steinert Threlkeld, and Sara Ng for helpful comments and discussion; Francis Tyers for access to the K'iche' data; Manuel Mager for access to the morphologically-segmented validation data tively. It has been particularly important for lower-resource languages in which there is little or no gold-standard data on which to train supervised models (Joshi et al., 2020). In modern neural end-to-end systems, unsupervised segmentation is usually performed via information-theoretic algorithms such as BPE (Sen-nrich et al., 2016) and SentencePiece (Kudo and Richardson, 2018). However, the segmentations they produce are largely non-sensical to humans (Park et al., 2021). The motivating tasks listed above instead require unsupervised approaches that correlate more closely with human judgements of the boundaries of linguistic units. For example, in a human-in-the-loop framework such as the sparse transcription proposed by Bird (2020), lexical items are automatically proposed to native speakers for confirmation, and it is important that these candidates be (close to) sensical, recognizable pieces of language. In this paper, we investigate the utility of recent models that have been developed to conduct unsupervised surface morpheme segmentation as a byproduct of a language modeling objective (e.g. Kawakami et al., 2019; Downey et al., 2021, see Section 2). The key idea is that recent breakthroughs in crosslingual language modeling and transfer learning (Conneau and Lample, 2019; Artetxe et al., 2020, inter alia ) can be leveraged to facilitate transferring unsupervised segmentation performance to a new target language, using these types of language models. Specifically, we investigate the effectiveness of multilingual pre-training in a Masked Segmental Language Model (Downey et al., 2021) when applied to a low-resource target. We pre-train our model on the ten Indigenous languages of the 2021 AmericasNLP shared task dataset (Mager et al., 2021), and apply it to another low-resource, Indigenous, and morphologically complex language of Central America: K'iche' (quc), which at least 5331 phylogenetically is unrelated to the pre-training languages (Campbell et al., 1986). We hypothesize that multilingual pre-training on similar, possibly contact-related languages, will outperform both a monolingual baseline trained from scratch and a model pre-trained on a single language (Quechua) with the same amount of pretraining data. We also expect that the pre-trained models will perform increasingly better than the monolingual baseline the smaller the target corpus is. Indeed, our experiments show that a pre-trained multilingual model provides stable performance across all dataset sizes and far exceeds the monolingual baseline at low-to-medium target sizes. We additionally show that the multilingual model achieves a zero-shot segmentation performance of 20.6 F1 on the K'iche' data, where the monolingual baseline yields a score of zero. These results suggest that transferring from a multilingual model can greatly assist unsupervised segmentation in very low-resource languages, even those that are morphologically rich. The results also provide evidence for the idea that transfer from multilingual models works at a more moderate scale than is typical for recent crosslingual models (3.15 million parameters for our models). In the following section, we overview work relating to unsupervised segmentation, crosslingual pre-training, and transfer-learning (Section 2). We then introduce the multilingual data used in our experiments, and the additional pre-processing we performed to prepare the data for pre-training (Sec-tion 3). Next we provide a brief overview of the type of Segmental Language Model used in our experiments, as well as our multilingual pre-training process (Section 4). After this, we describe our experimental process applying the pre-trained and from-scratch models to varying target data sizes (Section 5). Finally, we discuss the results of our experiments and their significance for low-resource pipelines, both within unsupervised segmentation and for other NLP tasks more generally (Sections 6 and 7). Work related to the present study largely falls either into the field of (unsupervised) word segmentation, or the field(s) of crosslingual language modeling and transfer learning. To our knowledge, we are the first to propose a multilingual model for unsupervised word/morpheme-segmentation. Unsupervised Segmentation Current state-of-the-art unsupervised segmentation has largely been achieved with Bayesian models such as Hierarchical Dirichlet Processes (Teh et al., 2006; Goldwater et al., 2009) and Nested Pitman-Yor (Mochihashi et al., 2009; Uchiumi et al., 2015). Adaptor Grammars (Johnson and Goldwater, 2009) have been successful as well. Models such as Morfessor (Creutz and Lagus, 2002), which are based on Minimal Description Length (Rissanen, 1989) are also widely used for unsupervised morphology. As Kawakami et al. (2019) note, most of these models have weak language modeling ability, being unable to take into account much other than the immediate local context of the sequence. Another line of techniques has focused on models that are both strong language models and good for sequence segmentation. Many are in some way based on Connectionist Temporal Classification (Graves et al., 2006), and include Sleep-WAke Networks (Wang et al., 2017), Segmental RNNs (Kong et al., 2016), and Segmental Language Models (Sun and Deng, 2018; Kawakami et al., 2019; Wang et al., 2021; Downey et al., 2021). In this work, we conduct experiments using the Masked Segmental Language Model of Downey et al. (2021), due to its good performance and scalability, the latter usually regarded as an obligatory feature of multilingual models (Conneau et al., 2020a; Xue et al., 2021, inter alia ). Crosslingual and Transfer Learning Crosslingual modeling and training has been an especially active area of research following the introduction of language-general encoder-decoders in Neural Machine Translation, offering the possibility of zero-shot translation (i.e. translation for language pairs not seen during training; Ha et al., 2016; Johnson et al., 2017). The arrival of crosslingual language model pretraining (XLM, Conneau and Lample, 2019) further demonstrates that large models pre-trained on multiple languages yield state-of-the-art performance across an abundance of multilingual tasks including zero-shot text classification (e.g. XNLI, Conneau et al., 2018), and that pre-trained transformer encoders provide great initializations for MT systems and language models in very low-resource languages. single out which components of crosslingual training contribute to transferability from one language to another (e.g. Conneau et al., 2020b). Others have questioned the importance of multilingual training, and have instead proposed that even monolingual pre-training can provide effective transfer to new languages (Artetxe et al., 2020). Though some like Lin et al. (2019) have tried to systematically study which aspects of pre-training languages/corpora enable effective transfer, in practice the choice is often driven by availability of data and other ad-hoc factors. Currently, large crosslingual successors to XLM such as XLM-R (Conneau et al., 2020a), MASS (Song et al., 2019), mBART (Liu et al., 2020), and mT5 (Xue et al., 2021) have achieved major success, and are the starting point for a large portion of multilingual NLP systems. These models all rely on an enormous amount of parameters and pre-training data, the bulk of which comes from very high-resource languages. In contrast, in this paper we assess whether multilingual pre-training on a suite of very low-resource languages, which combine to yield a moderate amount of unlabeled data, can provide good transfer to similar languages which are also very low-resource. We draw data from three main datasets. We use the AmericasNLP 2021 open task dataset (Mager et al., 2021) to pre-train our multilingual models. The multilingual dataset from Kann et al. (2018) serves as segmentation validation data for our pre-training process in these languages. Finally, data from Tyers and Henderson (2021) is used as the training set for our experiments transferring to K'iche', and Richardson and Tyers (2021) provides the validation and test data for these experiments. AmericasNLP 2021 The AmericasNLP data consists of train and validation files for ten low-resource Indigenous languages of Central and South America: Ashninka (cni), Aymara (aym), Bribri (bzd), Guaran (gug), Hhu (oto), Nahuatl (nah), Quechua (quy), Rarmuri (tar), Shipibo Konibo (shp), and Wixarika (hch). For each language, AmericasNLP also includes parallel Spanish sets, which we do not use. The data was originally curated for the AmericasNLP 2021 shared task on low-resource Machine Translation. (Mager et al., 2021). We augment the Ashninka and Shipibo-Konibo training sets with additional available monolingual data from Bustamante et al. (2020), 2 which is linked in the official AmericasNLP repository. We add both the training and validation data from this corpus to the training set of our splits. To pre-process for a multilingual language modeling setting, we first remove lines that contain urls, copyright boilerplate, or that contain no alphabetic characters. We also split lines that are longer than 2000 characters into sentences/clauses where evident. Because we use the Nahuatl and Wixarika data from Kann et al. (2018) as validation data, we remove any overlapping lines from the AmericasNLP set. We create a combined train file as the concatenation of the training data from each of the ten languages, as well as a combined validation file likewise. Because the original ratio of Quechua training data is so high compared to all other languages (Figure 1), we downsample it to 2 15 examples, the closest order of magnitude to the next-largest training set. A plot of the balanced (final) composition of our AmericasNLP train and validation sets is seen in Figure 2. To compare the effect of multilingual and monolingual pre-training, we also pre-train a model on Quechua alone, since it has by far the most data (Figure 1). However, the full Quechua training set has about 50k fewer lines than our balanced AmericasNLP set (Figure 2). To create a fair comparison between multilingual and monolingual pre-training, we additionally create a downsampled version of the AmericasNLP set of equal size to the Quechua data (120,145 lines). The detailed composition of our data is available in Appendix A. Kann et al (2018) The data from Kann et al. (2018), originally curated for a segmentation task on polysynthetic low-resource languages, contains morphologically segmented sentences for Nahuatl and Wixarika. We use these examples as validation data for segmentation quality during the pretraining process. We clean this data in the same manner as the AmericasNLP sets. K'iche' data The K'iche' data used in our study was curated for Tyers and Henderson (2021). The 1 https://github.com/AmericasNLP/ americasnlp2021 2 https://github.com/iapucp/ multilingual-data-peru 5333 Figure 1: Original (imbalanced) language composition of the AmericasNLP training set Figure 2: Final language composition of our AmericasNLP splits after downsampling Quechua raw (non-gold-segmented) data, used as the training set in our transfer experiments, comes from a section of this data web-scraped by the Crbadn project (Scannell, 2007). This data is relatively noisy, so we clean it by removing lines with urls or lines where more than half of the characters are non-alphabetic. We also remove duplicate lines. The final data consists of 47,729 examples and is used as our full-size training set for K'iche'. Our experiments involve testing transfer at different resource levels, so we also create smaller training sets by downsampling the original to lower orders of magnitude. For evaluating segmentation performance on K'iche', we use the segmented sentences from Richardson and Tyers (2021), 3 which were created for a shared task on morphological segmen-3 https://github.com/ftyers/ global-classroom tation. These segmentations were created by a hand-crafted FST, then manually disambiguated. Because gold-segmented sentences are so rare, we concatenate the original train/validation/test splits and then split them in half into final validation and test sets. This section gives an overview of the Masked Segmental Language Model (MSLM), introduced in Downey et al. (2021), along with a description of our pre-training procedure. MSLMs An MSLM is a variant of a Segmental Language Model (SLM) (Sun and Deng, 2018; Kawakami et al., 2019; Wang et al., 2021), which takes as input a sequence of characters x and outputs a probability distribution for a sequence of segments y such that the concatenation of y is equivalent to x :  ( y ) = x . An MSLM is composed of a Segmental Transformer Encoder and an LSTM-based Segment Decoder (Downey et al., 2021). See Figure 3. The MSLM training objective is based on the prediction of masked-out spans. During a forward pass, the encoder generates an encoding for every position in x , for a segment up to k symbols long; the encoding at position i  1 corresponds to every possible segment that starts at position i . Therefore, the encoding approximates p ( x i : i +1 , x i : i +2 , ..., x i : i + k | x <i , x  i + k ) To ensure that the encodings are generated based only on the portions of x that are outside of the predicted span, the encoder uses a Segmental Attention Mask (Downey et al., 2021) to mask out tokens inside the segment. Figure 3 shows an example of such a mask with k = 2 . Finally, the Segment Decoder of an SLM determines the probability of the j th character of the segment of y that begins at index i , y ij , using the encoded context: p ( y ij | y i 0: j , x <i , x  i + k ) = Decoder ( h ij  i , y ij  1 ) The output of the decoder is not conditional on the determination of other segment boundaries. The probability of y is modeled as the marginal probability over all possible segmentations of x . Because directly marginalizing is computationally intractable, the marginal is computed using dynamic programming over a forward-pass lattice. The maximum-probability segmentation is determined by Viterbi decoding. The training objective optimizes language-modeling performance, which is measured in Bits Per Character (bpc). Pre-training Procedure In our experiments, we test the transferability of multilingual and monolingual pre-trained MSLMs. The multilingual models are trained on the AmericasNLP 2021 data (see Section 3). Since SLMs operate on plain text, we can train the model directly on the multilingual concatenation of this data, and evaluate it by its language modeling performance on the concatenated validation data. As mentioned in Section 3, we create two versions of the multilingual pre-trained model: one trained on the full AmericasNLP set (  172k lines) and the other trained on the downsampled set, which is the same size as the Quechua training set (  120k lines). We designate these models MULTI-PT full and MULTI-PT down , respectively. Our pre-trained monolingual model is trained on the full Quechua set (QUECHUA-PT ). Each model is an MSLM with four encoder layers, hidden size 256, feedforward size 512, and four attention heads. Character embeddings are initialized using Word2Vec (Mikolov et al., 2013) over the training data. The maximum segment size is set to 10. The best model is chosen as the one that minimizes the Bits Per Character (bpc) loss on the validation set. For further pre-training details, see Appendix B. To evaluate the effect of pre-training on the segmentation quality for languages within the pretraining set, we also log MCC between the model output and gold-segmented secondary validation sets available in Nahuatl and Wixarika (Kann et al., 2018, see Section 3). Figure 4 shows the unsupervised segmentation quality for Nahuatl and Wixarika almost monotonically increases during pre-training (MULTI-PT full ). To do this, we pre-train SLMs on one or all of the AmericasNLP 2021 languages (Mager et al., 2021) and transfer it to a new target language: K'iche' (Tyers and Henderson, 2021). K'iche' is a morphologically rich Mayan language with several classes of in-flectional prefixes and suffixes (Txchajchal Batz et al., 1996). An example sentence can be found in Table 1, which also shows our model's input and target output format. As a baseline, we train a monolingual K'iche' model from scratch. We evaluate performance with respect to the size of the target training set, simulating varying degrees of low-resource setting. To do this, we downsample the K'iche' training set to 8 smaller sizes, for 9 total: {256, 512, ... 2 15 , 47.7k (full)}. For each size, we both train a monolingual baseline and fine-tune the pre-trained models we describe in Section 4. 4 4 All of the data and software required to run these experiments can be found at https://github.com/ cmdowney88/XLSLM 5335 Orthography kinch'aw ruk' le nunan Linguistic Segmentation k-in-ch'aw r-uk' le nu-nan Translation I speak with my mother Model Input kinch'awruk'lenunan Target Output k in ch'aw r uk' le nu nan Table 1: Example K'iche' sentence from Tyers and Henderson (2021). The only difference is that the baseline model is initialized with a character vocabulary only covering the particular K'iche' training set (size-specific). The character vocabulary of the K'iche' data is a subset of the AmericasNLP vocabulary, so we are able to transfer the multilingual models without changing the embedding and output layers. The Quechua vocabulary is not a superset of the K'iche', so we add the missing characters to the Quechua model's embedding block before pre-training (these are randomly initialized). The character embeddings for the baseline are initialized using Word2Vec (Mikolov et al., 2013) on the training set (again, size-specific). Evaluation Metrics SLMs can be trained in either a fully unsupervised or lightly supervised manner (Downey et al., 2021). In the former case, only the language modeling loss (Bits Per Character, bpc) is used to pick parameters and checkpoints. In the latter, the segmentation quality on gold-segmented validation data can be considered. Though our validation set is gold-segmented, we pick the best parameters and checkpoints based on bpc only, simulating the unsupervised case. However, to monitor the change in segmentation quality during training, we also use Matthews Correlation Coefficient (MCC). This measure frames segmentation as a character-wise binary classification task (i.e. boundary vs. no boundary), and measures correlation with the gold segmentation. To make our results comparable with the wider word-segmentation literature, we use the scoring script from the SIGHAN Segmentation Bakeoff (Emerson, 2005) for our final segmentation F1. For each model and target size, we choose the best checkpoint (by bpc), apply the model to the combined validation and test set, and use the SIGHAN script to score the output. For comparison to the Chinese Word-Segmentation and speech literature, any whitespace segmentation in the validation/test data is discarded before it is fed to the model. However, SLMs can also be trained to treat spaces like any other character, and thus could be able to take advantage of existing segmentation in the input. We leave this for future work. Parameters and Trials For our training procedure (both training the baseline from scratch and fine-tuning the pre-trained models) we tune hyperparameters on three of the nine dataset sizes (256, 2048, and full) and choose the optimal parameters by bpc. For each of the other sizes, we directly apply the chosen parameters from the tuned dataset of the closest size (on a log scale). We tune over five learning rates and three encoder dropout values. As in pre-training, we set the maximum segment length to 10. For more details on our training procedure, see Appendix B. 6 Results The results of our K'iche' transfer experiments at various target sizes can be found in Table 2. In general, the (full) pre-trained multilingual model (MULTI-PT full ) demonstrates good performance across dataset sizes, with the lowest segmentation performance (20.6 F1) being in the zero-shot case and the highest (40.7) achieved on 2 14 examples. The monolingual baseline outperforms MULTIPT full at the two largest target sizes, as well as at size 4096 (achieving the best overall F1 of 44.8), but performs very poorly under 2048 examples, and has no zero-shot ability (unsurprisingly, since it is a random initialization). Interestingly, other than in the zero-shot case, QUECHUA-PT and the comparable MULTI-PT down perform very similarly to each other. However, the 5336 zero-shot transferability of MULTI-PT down is almost twice that of the model trained on Quechua only. MULTI-PT full exceeds both MULTI-PT down and QUECHUA-PT by a wide margin in every setting. Finally, all models show increasing performance until about size 4096, after which more target examples don't provide a large increase in segmentation quality. Interpretation These results show that MULTIPT full provides consistent performance across target sizes as small as 512 examples. Even for size 256, there is only a 9% (relative) drop in quality from the next-largest size. Further, the pre-trained model's zero-shot performance is impressive given the baseline is effectively 0 F1. On the other hand, the performance of the monolingual baseline at larger sizes seems to suggest that given enough target data, it is better to train a model devoted to the target language only. This is consistent with previous results (Wu and Dredze, 2020; Conneau et al., 2020a). However, it should also be noted that MULTI-PT full never trails the baseline by more than 5.2 F1. One less-intuitive result is the dip in the base-line's performance at sizes 8192 and 2 14 . We believe this discrepancy may be partly explainable by sensitivity to hyperparameters in the baseline. Though the best baseline trial at size 2048 exceeds MULTI-PT full by a small margin, the baseline shows large variation in performance across the top-four hyperparameter settings at this size, where MULTI-PT full actually performs better on average and much more consistently (Table 3). We thus believe the dip in performance for the baseline at sizes 8192 and 2 14 may be due to an inability to extrapolate hyperparameters from other experimental settings. Standing of Hypotheses Within the framework of unsupervised segmentation, these results provide strong evidence that relevant linguistic patterns can be learned over a collection of low-resource languages, and then transferred to a new language without much (or any) target training data. Further, it is shown that the target language need not be (phylogenetically) related to any of the pre-training languages, even though details of morphological structure are ultimately language-specific. baseline at smaller target sizes is also strongly supported. This result is consistent with related work showing this to be a key advantage of the multilingual approach (Wu and Dredze, 2020). The hypothesis that multilingual pre-training also yields better performance than monolingual pre-training given the same amount of data seems to receive mixed support from our experiments. On one hand, the comparable multilingual model has a clear advantage over the Quechua model in the zero-shot setting, and outperforms the latter in 5/10 settings more generally. However, because the Quechua data lacks several frequent K'iche' characters (and these embeddings remain randomly ini-tialized), it is unclear how much of this advantage comes from the multilingual training per-se . Instead, the advantage may be due to the multilingual model's full coverage of the target vocabulary an advantage which may disappear at larger target sizes. Further analysis of this hypothesis will require additional investigation. Significance The above results, especially the strong zero-shot transferability of segmentation performance, suggest that the type of language model used here learns some abstract linguistic pattern(s) that are generalizable across languages, and even to new ones. It is possible that these generalizations could take the form of abstract stem/affix or word-order patterns, corresponding roughly to the lengths and order of morphosyntactic units. Because MSLMs operate on the character level (and in these languages orthographic characters mostly correspond to phones), it is also possible the model could recognize syllable structure in the data (the ordering of consonants and vowels in human languages is relatively constrained), and learn to segment on syllable boundaries. It is also helpful to remember that we select the training suite and target language to have some characteristics in common that may help facilitate transfer. The AmericasNLP languages are almost all morphologically rich, with many considered polysynthetic (Mager et al., 2021), a feature that K'iche' shares (Surez, 1983). Further, all of the languages, including K'iche', are spoken in countries where either Spanish or Portuguese is the official language, and have very likely had close contact with these Iberian languages and borrowed lexical items. Finally, the target language family (Mayan) has also been shown to have close historical contact with the families of several of the 5337 Model Target Language Segmentation F1 0 256  512 1024 2048  4096 8192 2 14 2 15 47,729 (full)  MULTI-PT full 20.6 34.0 37.4 37.4 38.2 40.5 38.6 40.7 38.9 38.2 MULTI-PT down 15.0 25.1 25.7 29.3 32.5 33.2 33.3 31.5 33.6 31.9 QUECHUA-PT 7.6 29.9 31.0 30.4 30.7 31.0 29.9 33.6 31.8 33.3 MONOLINGUAL 0.002 4.0 3.3 10.3 39.2  44.8 29.4 39.5 44.1 43.2 Table 2: Segmentation quality on the combined validation and test set for each model, at each target training set size. It is possible that one or several of these shared characteristics facilitates the strong transfer shown here, in both our multilingual and monolingual pretrained models. However, our current study does not conclusively show this to be the case. Lin et al. (2019) show that factors like linguistic similarity and geographic contact are often not as important for transfer success as non-linguistic features such as the raw size of the source dataset. Indeed, the fact that our Quechua pre-trained model performs similarly to the comparable multilingual model (at least at larger target sizes) suggests that the benefit to using MULTI-PT full could be interpreted as a combined advantage of pre-training data size and target vocabulary coverage. The nuanced question of whether multilingual pre-training itself enables better transfer than monolingual pre-training requires more study. However, taking a more pragmatic point of view, multilingual training can be seen as a methodology to 1) acquire more data than is available from any one language and 2) ensure broader vocabulary overlap with the target language. Our character-based model is of course different from more common wordor subword-based approaches, but with these too, attaining pre-trained embeddings that cover a novel target language is an important step in cross-lingual transfer (Garcia et al., 2021; Conneau et al., 2020a; Artetxe et al., 2020, inter alia ) Future Work We believe some future studies would shed light on the nuances of segmentation transfer-learning. First, pre-training either multilingually or monolingually on languages that are not linguistically similar to the target language could help isolate the advantage given by pre-training on any language data (vs. similar language data). Second, we have noted that monolingual pretraining on a language that does not have near-full vocabulary coverage of the target language leaves some embeddings randomly initialized, yielding worse performance at small target sizes. Pretraining a model on a single language that happens to have near-complete vocabulary coverage of the target could give a better view of whether multilingual training intrinsically yields advantages, or whether monolingual training is disadvantaged mainly due to this lack of vocabulary coverage. Finally, because none of the present authors have any training in the K'iche' language, we are unable to perform a linguistically-informed error analysis of our model's output (e.g. examining the types of words and morphemes which are erroneously 5338 (un)segmented, rather than calculating an overall precision and recall for the predicted and true morpheme boundaries, as we do in this study). However, we make all of our model outputs available in our public repository, so that future work may provide a more nuanced analysis of the types of errors unsupervised segmentation models are prone to make. This study has shown that unsupervised sequence segmentation ability can be transferred via multilingual pre-training to a novel target language with little or no target data. The target language also need not be from the same family as a pre-training language for successful transfer. While training a monolingual model from scratch on large amounts of target data results in good segmentation quality, our experiments show that pre-trained models, especially multilingual ones, far exceed the baseline at small target sizes (  1024), and seem to be much more robust to hyperparameter variation at medium sizes (2048, 8192, 2 14 ). One finding that may have broader implications is that pre-training can be conducted over a set of low-resource languages with some typological or geographic connection to the target, rather than over a crosslingual suite centered around high-resource languages like English and other European languages. Most modern crosslingual models have huge numbers of parameters (XLM has 570 million, mT5 has up to 13 billion, Xue et al., 2021), and are trained on enormous amounts of data, usually bolstered by hundreds of gigabytes in the highest-resource languages (Conneau et al., 2020a). In contrast, our results suggest that effective transfer may be possible at smaller scales, by combining the data of low-resource languages and training moderately-sized, more targeted pre-trained multilingual models (our model has 3.15 million parameters). Of course, this study can only support this possibility within the unsupervised segmentation task, so future work will be needed to investigate whether transfer to and from low-resource languages can be extended to other tasks.
result result abstain method result result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective method result result abstain result method method method method result other objective other other other other other other method other other other other other other other objective abstain method abstain method abstain method abstain abstain abstain method abstain method method method method abstain method abstain method abstain method abstain method other method method method method other abstain method abstain abstain abstain abstain abstain abstain other abstain other abstain abstain abstain abstain method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method other abstain method method abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result result abstain method abstain abstain other abstain method abstain method method objective abstain result abstain abstain objective objective