Language identification for code-switching (CS), the phenomenon of alternating between two or more languages in conversations, has traditionally been approached under the as-sumption of a single language per token. However, if at least one language is morphologically rich, a large number of words can be composed of morphemes from more than one language (intra-word CS). In this paper, we extend the language identification task to the subword level, such that it includes splitting mixed words while tagging each part with a language ID. We further propose a model for this task, which is based on a segmental recurrent neural network. In experiments on a new SpanishWixarika dataset and on an adapted GermanTurkish dataset, our proposed model performs slightly better than or roughly on par with our best baseline, respectively. Considering only mixed words, however, it strongly outperforms all baselines. In settings where multilingual speakers share more than one language, mixing two or more languages within a single piece of text, for example a tweet, is getting increasingly common (Grosjean, 2010). This constitutes a challenge for natural language processing (NLP) systems, since they are commonly designed to handle one language at a time. Code-switching (CS) can be found in multiple non-exclusive variants. For instance, sentences in different languages can be mixed within one text, or words from different languages can be combined into sentences. CS can also occur on the subword level, when speakers combine morphemes from different languages ( intra-word CS ). This last phenomenon can mostly be found if at least one of the languages is morphologically rich. An example for intra-word CS between the Ro-(a) ne'iwa pecansadoxi  WIX MIXED my.brother you-are.tired.PPFV mance language Spanish and the Yuto-Aztecan language Wixarika 1 is shown in Figure 1. CS language identification (LID) , i.e., predicting the language of each token in a text, has attracted a lot of attention in recent years (cf. Solorio et al. (2014); Molina et al. (2016)). However, intra-word mixing is mostly not handled explicitly: words with morphemes from more than one language are simply tagged with a mixed label. While this works reasonably well for previously studied language pairs, overlooking intra-word CS leads to a major loss of information for highly pol-synthetic languages. A mixed word is unknown for NLP systems, yet a single word contains much more information, cf. Figure 1 (b). Furthermore, we find intra-word CS to be much more frequent for SpanishWixarika than for previously studied language pairs, such that it is crucial to handle it. Motivated by these considerations, we extend the LID task to the subword level (from (a) to (b) in Figure 1). We introduce a new CS dataset for SpanishWixarika (ESWIX) and modify an existing GermanTurkish (DETR) CS corpus (Cetinoglu, 2016) for our purposes. We then intro-1 Wixarika, also known as Huichol, is a polysynthetic Mexican indigenous language. duce a segmental recurrent neural network (Seg-RNN) model for the task, which we compare against several strong baselines. Our experiments show clear advantages of SegRNNs over all baselines for intra-word CS. The task of LID for CS has been frequently studied in the last years (Al-Badrashiny and Diab, 2016; Rijhwani et al., 2017; Zhang et al., 2018), including two shared tasks on the topic (Solorio et al., 2014; Molina et al., 2016). The best systems (Samih et al., 2016; Shirvani et al., 2016) achieved over 90% accuracy for all language pairs. However, intra-word CS was not handled explicitly, and often systems even failed to correctly assign the mixed label. For NepaliEnglish, Barman et al. (2014) correctly identified some of the mixed words with a combination of linear kernel support vector machines and a k -nearest neighbour approach. The most similar work to ours is Nguyen and Cornips (2016), which focused on detecting intra-word CS for DutchLimburgish (Nguyen et al., 2015). The authors utilized Morfessor (Creutz and Lagus, 2002) to segment all words into morphemes and Wikipedia to assign LID probabilities to each morpheme. However, their task definition and evaluation are on the word level. Furthermore, as this method relies on large monolingual resources, it is not applicable to low-resource languages like Wixarika, which does not even have its own Wikipedia edition. Subword-level LID consists of both segmentation and tagging of words. An earlier approach to handle a similar scenario was the connectionist temporal classification (CTC) model developed by Graves et al. (2006). The disadvantage of this model was the lack of prediction of the segmentation boundaries that are necessary for our task. Kong et al. (2016) later proposed the SegRNN model that segments and labels jointly, with successful applications on automatic glossing of polysynthetic languages (Micher, 2017, 2018). Segmentation of words into morphemes alone has a long history in NLP (Harris, 1951), including semior unsupervised methods (Goldsmith, 2001; Creutz and Lagus, 2002; Hammarstrom and Borin, 2011; Gronroos et al., 2014), as well as supervised ones (Zhang and Clark, 2008; Ruokolainen et al., 2013; Cotterell et al., 2015; Kann et al., 2018). Formally, the task of subword-level LID consists of producing two sequences, given an input sequence of tokens X = (cid:104) x 1 , . . . , x i , . . . , x | X | (cid:105) . The first sequence contains all words and splits X s = (cid:104) x s 1 , . . . , x si , . . . , x s | X | (cid:105) , where each x si is an m -tuple of variable length 0 < m  | x i | , where | x i | is the number of characters in x i . The second sequence is such that T s = (cid:104) t s 1 , . . . , t si , . . . , t s | X | (cid:105) , where | T s | = | X s | = | X | and each t si  T s is an n -tuple of tags from a given set of LID tags. An inputoutput example for a DETR mixed phrase is shown in Figure 2. Input (cid:104) Yerim', seni', ,', danke', Schatzym' (cid:105) Output (cid:104) (Yerim), (seni), (,), (danke), (Schatzy, m) (cid:105) (cid:104) (TR), (TR), (OTHER), (DE), (DE, TR) (cid:105) Figure 2: Subword-level LID in GermanTurkish. GermanTurkish The GermanTurkish Twitter Corpus (Cetinoglu and Coltekin, 2016) consists of 1029 tweets with 17K tokens. They are manually normalized, tokenized, and annotated with language IDs. The language ID tag set consists of TR (Turkish), DE (German), LANG3 (other lan-guage), MIXED (intra-word CS), AMBIG (ambigu-ous language ID in context), and OTHER (punctu-ation, numbers, emoticns, symbols, etc.). Named entities are tagged with a combination of NE and their language ID: NE.TR , NE.DE , NE.LANG3 . In the original corpus, some Turkish and mixed words undergo a morphosyntactic split, 2 with splitting points not usually corresponding to language boundaries. For the purpose of subword-level LID, these morphosyntactic splits are merged back into single words. We then manually segment MIXED words at language boundaries, and replace their labels with more fine-grained language ID tags. The total percentage of mixed words is 2 . 75% . However, the percentage of sentences with mixed words is 15.66%. The complete dataset statistics can be found in Table 1. SpanishWixarika Our second dataset consists of 985 sentences and 8K tokens in Spanish and Wixarika. Wixarika is spoken by approximately 2 E.g., separating copular suffixes from roots they are attached to, cf. Cetino  glu and C  oltekin (2016) for details. 50 , 000 people in the Mexican states of Durango, Jalisco, Nayarit and Zacatecas (Leza and Lopez, 2006) and is polysynthetic, with most morphemes ocurring in verbs. The data is collected from public postings and comments from Facebook accounts. To ensure the public characteristic of these posts, we manually collect data that is accessible publicly without being logged in to Facebook, to comply with the terms of use and privacy of the users. These posts and comments are taken from 34 users: 14 women, 10 men, and the rest does not publically reveal their gender. None of them have publically mentioned their age. To get a dataset that focuses on the LID task, we only consider threads where the CS phenomenon appears. We replace usernames with @username in order to preserve privacy. Afterwards, we tokenize the text, segment mixed words, and add language IDs to words and segments. The tag set is parallel to that of German Turkish: ES (Spanish), WIX (Wixarika), EN (En-glish), AMBIG (ambiguous) OTHER (punctuation, numbers, emoticons, etc), NE.ES , NE.WIX and NE.EN (named entities). Mixed words are segmented and each segment is labeled with its corresponding language ( ES , WIX , EN ). Table 2 shows a detailed description of the dataset. The percentage of mixed words is higher than in the DETR dataset: 3 . 13% of the tokens and 4 . 26% of the types. The most common combination is Spanish roots with Wixarika affixes. Furthermore, 16.55% of the sentences contain mixed words. We split the DETR corpus and the ESWIX corpus into training and test sets of sizes 800:229 Tokens All % Unique Unique % ES 4218 50.73 1527 45.76 WIX 2019 24.28 1191 35.69 EN 24 0.29 21 0.63 AMBIG 28 0.34 25 0.75 OTHER 1664 20.01 288 8.63 NE.ES 96 1.15 85 2.55 NE.WIX 77 0.93 49 1.47 NE.EN 11 0.13 9 0.27 MIXED 177 2.13 142 4.26 ES WIX 35 19.77 31 21.83 WIX ES 122 68.93 93 65.49 WIX ES WIX 17 9.60 31 10.56 WIX EN 1 0.07 1 0.07 EN ES 1 0.07 1 0.07 Table 2: Number of tokens classified by language tags seen in the Spanish-Wixarika dataset. and 770:216, respectively. Error analysis and hyperparameter tuning are done on the training set via 5-fold cross-validation. We present results on the test sets. Both datasets are available at https://www.ims.uni-stuttgart. de/institut/mitarbeiter/ozlem/ NAACL2019.html 4 Experiments Our main system is a neural architecture that jointly solves the segmentation and language identification tasks. We compare it to multiple pipeline systems and another joint system. We suggest a SegRNN (Kong et al., 2016) would be the best fit for our task because it models a joint probability distribution over possible segmentations of the input and labels for each segment. The model is trained to optimize the following objective, which corresponds to the joint log-likelihood of the segment lengths e and the language tags t : L (  )= (cid:88) ( x,t,e ) D  log p ( t, e | x ) (1) D denotes the training data,  is the set of model parameters, x the input, t the tag sequence and e the sequence of segment lengths. Our inputs are single words. 3 As hyperparame-3 We also experimented with entire phrases as inputs, and the achieved scores were slightly worse than for word-based inputs. ters we use: 1 RNN layer, a 64-dimensional input layer, 32 dimensions for tags, 16 for segments, and 4 for lengths. For training, we use Adam (Kingma and Ba, 2014). BiLSTM+Seq2Seq/BiLSTM+CRF Our first baselines are pipelines. First, the input text is tagged with language IDs. Language IDs of a mixed word are directly predicted as a combination of all language ID tags of the word (i.e., WIX ES ). Second, a subword-level model segments words with composed language ID tags. For word-level tagging, we use a hierarchical bidirectional LSTM ( BiLSTM ) that incorporates both tokenand character-level information (Plank et al., 2016), similar to the winning system (Samih et al., 2016) of the Second Code-Switching Shared Task (Molina et al., 2016). 4 For the subword level, we use two supervised segmentation methods: a CRF segmenter proposed by Ruokolainen et al. (2013), that models segmentation as a labeling problem and a sequence-to-sequence ( Seq2Seq ) model trained with an auxiliary task as proposed by Kann et al. (2018). CRFTag+Seq2Seq/CRFTag+CRF Since our datasets might be small for training neural networks, we substitute the BiLSTM with a CRF tagger (Muller et al., 2013, CRFTag ) in the first step. For segmentation, we use the same two approaches as for the previous baselines. CharBiLSTM We further employ a BiLSTM to tag each character with a language ID. For training, each character inherits the language ID of the word or segment it belongs to. At prediction time, if the characters of a word have different language IDs, the word is split. 4 For all BiLSTM models input dimension is 100 with a hidden layer size of 100. For training we use a stochastic gradient descent (Bottou, 2010), 30 epochs, with a learning rate of 0.1. A 0.25 dropout factor is applied. We use two metrics for evaluation. First, we follow Kong et al. (2016) and calculate precision (P), recall (R), and F1, using segments as units (an unsegmented word corresponds to one segment). We also report a tagging accuracy (Char Acc.) by assigning a language ID to each character and calculating the ratio of correct language tags over all characters. Table 4 shows all test results for the entire datasets. We find the following: (i) For ESWIX, SegRNN performs slightly better for tagging than the best baseline, both in terms of F1 and character accuracy. For DETR, SegRNN and BiLSTM+CRF are the best segmentation models, but the BiLSTM models slightly outperform SegRNN for tagging. (ii) The CRF pipelines perform slightly worse than the best word-level BiLSTM models for both datasets and all evaluations. Table 3 shows the results of tagging and segmentation only for the mixed words in our datasets. Here, we can see that: (i) Our SegRNN model achieves the best performance for segmentation. Differences to the other approaches are  10% , showing clearly why these models are good for the task when the number of words belonging to two languages is high. (ii) The pipeline BiLSTM models work best for tagging of the DE TR data with a slight margin, but underperform TRDEDETRNE . (iii) Both CRFTag models achieve very low results for both segmentation and tagging. (iv) CharBiLSTM performs better than the CRFTag models on both tasks, but is worse than all other approaches in our experiments. More generally, we further observe that recall on mixed words for the DETR pair is low for all systems, as compared to ESWIX. This effect is especially strong for the CRFTag and CharBiLSTM models, which seem to be unable to correctly identify mixed words. While this tendency can also be seen for the ESWIX pair, it is less extreme. We suggest that the better segmentation and tagging of mixed words for ESWIX might mostly be due to the higher percentage of available examples of mixed words in the training set for ESWIX. Overall, we conclude that SegRNN models seem to work better on language pairs that have more intra-word CS, while pipeline approaches might be as good for language pairs where the number of mixed words is lower. Error analysis. Figure 3 shows confusion matrices for SegRNN and BiLSTM+Seg2Seg. Both models achieve good results assigning monolingual tags ( ES , WIX , DE , TR ) and punctuation symbols ( OTHERS ). The hardest labels to classify are named entities ( NE , NE.TR , NE.TR , NE.WIX , NE.ES ), as well as third language and ambiguous tags ( LANG3 , EN , AMBIG ). Performance on multilingual tags ( DE TR , WIX ES , ES WIX , WIX ES WIX ) is mixed. For DE TR , BiL-STM+Seq2Seq gets slightly better classifications, but for the ESWIX tags SegRNN achieves better results. Regarding oversegmentation problems, BiL-STM+Seq2Seq (0.8% for DETR and 2.0% for ESWIX) slightly underperforms SegRNN (0.7% for DETR and 1.13% for ESWIX). The BiLSTM+Seq2Seq (2.4%) makes fewer undersegmentation errors for DETR than SegRNN (2.7%). However, for ESWIX, SegRNN performs better with 3.81% undersegmentation errors compared to 4.2% of BiLSTM+Seq2Seq. In this paper, we extended the LID task to the subword level, which is particularly important for code-switched text in morphologically rich languages. We further proposed a SegRNN model for the task and compared it to several strong baselines. Investigating the behaviour of all systems, we found that pipelines including a BiLSTM tagger work well for tagging DETR, where the number of mixed tokens is not that high, but that our proposed SegRNN approach performs better than all other systems for ESWIX. Also, SegRNNs have clear advantages over all baselines if we consider mixed words only. Our subword-level LID datasets for ESWIX and DETR are publicly available. We would like to thank Mohamed Balabel, Sam Bowman, Agnieszka Falenska, Ilya Kulikov and Phu Mon Htut for their valuable feedback. We also want to thank Jeffrey Micher for his help with the setup of the SegRNN code. This project has bene-fited from financial support to Manuel Mager and Ozlem Cetinoglu by DFG via project CE 326/1-1 Computational Structural Analysis of German-Turkish Code-Switching, to Manuel Mager by DAAD Doctoral Research Grant, and to Katharina Kann by Samsung Research.
abstain abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective abstain abstain abstain abstain method result other other other other other other other other other other abstain other other abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain method abstain other abstain abstain method method abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain method abstain method abstain abstain abstain result abstain abstain abstain result result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective method abstain other other other