Simplified Chinese to Traditional Chinese character conversion is a common preprocessing step in Chinese NLP. Despite this, current approaches have insufficient performance because they do not take into account that a simplified Chinese character can correspond to multiple traditional characters. Here, we propose a model that can disambiguate between mappings and convert between the two scripts. The model is based on subword segmentation, two language models, as well as a method for mapping between subword sequences. We further construct benchmark datasets for topic classification and script conversion. Our proposed method outperforms previous Chinese Character conversion approaches by 6 points in accuracy. These results are further confirmed in a downstream application, where 2kenize is used to convert pretraining dataset for topic classification. An error analysis reveals that our method's particular strengths are in dealing with code mixing and named entities. The code and dataset is available at https: //github.com/pranav-ust/2kenize 1 Introduction Chinese character (or script) conversion is a common preprocessing step for Chinese NLP practitioners (Zhang, 2014; Shi et al., 2011). Traditional Chinese (TC) and Simplified Chinese (SC) are the two standardized character sets (or scripts) for written Chinese. TC is predominantly used in Taiwan, Hong Kong, and Macau, whereas SC is mainly adopted in mainland China and SC characters are simplified versions of TC characters in terms of strokes and parts. Therefore, Chinese NLP practitioners apply script converters 1 to translate the 1 Most of these tools like OpenCC, Mafan, Hanziconv are generally widely used within Chinese NLP community, which can be attested popularity of their Github repos. dataset into their desired language. This is especially useful for TC NLP practitioners because TC is less widely used and under-resourced as compared to SC. Converting from TC to SC is generally straightforward because there are one-to-one correspondences between most of the characters, so conversion can be performed using mapping tables (Denisowski, 2019; Chu et al., 2012). However, conversion from SC to TC is an arduous task as some SC characters can be mapped to more than one TC character depending on the context of the sentence. A detailed analysis by Halpern and Kerman (1999) shows that SC to TC conversion is a challenging and crucial problem, as 12% of SC characters have one-to-many mappings to TC characters. Our experiments show that current script converters achieve sentence accuracy results of 55-85% (3). Another issue is that varying tokenization would lead to different results as Chinese is an unsegmented language, see Table 1 for an example. Off-the-shelf script converters would translate   into   , 2 whereas the correct conversion is  2 Throughout this paper, we color code ambiguous SC characters with brown, ambiguous TC characters with violet, vernacular Cantonese characters with teal. By scripts, we refer as to character sets, and we interchangeably use them in this paper. . Here, the SC character  ( hair, issue ) has two TC mappings,  ( hair, issue ) and  ( hair, issue ), depending on the context and tokenization; which shows that this task is non-trivial. Despite this being an important task, there is a lack of benchmarks, 3 which implies that this problem is understudied in NLP. In this study, we propose 2kenize , a subword segmentation model which jointly considers Simplified Chinese and forecasting Traditional Chinese constructions. We achieve this by constructing a joint Simplified Chinese and Traditional Chinese language model based Viterbi tokenizer. Performing mapping disambiguation based on this tokenization method improves sentence accuracy by 6 points as compared to off-the-shelf converters and supervised models. Our qualitative error analysis reveals that our method's particular strengths are in dealing with code-mixing and named entities. Additionally, we address the issue of a lack of benchmark datasets by constructing datasets for script conversion and TC topic classification. We employ subword tokenization, as it addresses the issue of rare and unknown words (Mikolov et al., 2012) and has been shown advantageous for the language modelling of morphologically-rich languages (Czapla et al., 2018; Mielke and Eisner, 2019). This achieves improvements in accuracy for neural machine translation (NMT) tasks and has now become a prevailing practice (Denkowski and Neubig, 2017). The most widely-utilized method is Byte Pair Encoding (BPE, Sen-nrich et al. (2016)), a compression algorithm that combines frequent sequences of characters, which results in rare strings being segmented into subwords. Unigram (Kudo, 2018) and BPE-Drop (Provilkov et al., 2019) use subword ambiguity as noise, as well as stochastically-corrupted BPE segmentation to make it less deterministic. For NMT tasks generally, subword segmentation is seen as a monolingual task and applied independently on source and target corpora. We hypothesize that translation tasks, and specifically conversion tasks, as investigated here, would have a bet-3 The ChineseNLP website states that script conversion benchmarks and experiments currently do not exist: https://chinesenlp.xyz/#/docs/simplified_traditional_Chinese_conversion ter performance if segmentation were performed jointly. Hence, in this section, we describe our proposed method 2kenize , which jointly segments by taking the source and its approximate target sentences into account. This motivates the main idea of this paper: We propose 2kenize which jointly considers the source sentence and its corresponding target conversions by doing lookaheads with mappings. Given the possible SC character sequence s = s 1 s 2 : : : s n and TC character sequence t = t 1 t 2 : : : t n , we want to find the most likely t , which is given by the Bayes decision rule as follows: where T (cid:3) denotes the set of all strings over symbols ( t i ) in T (Kleene star). We divide this problem into two parts: finding the mapping sequence (2) and finding the TC sequence from mappings (7). We define a mapping, which is given by m i = ( s i ; t i ) = ( s j : k ; t j : k ) . Here, t j : k = f t 1 j : k : : : t nj : k g is a set of TC characters that correspond to the SC character in the mapping. Thus, a mapping sequence can be defined as a concatenation of mappings, which is m = m 1 m 2 : : : m l . Let M be the superset of all possible mapping sequences and M ( s ) be the all mapping sequences resulting from s . Then, the best possible mapping sequence is given by m = arg max m  2M ( s ) p ( m  ) (2) Morever, p ( m ) can be expanded as such: p ( m ) = p ( m 1 m 2 : : : m l ) (3) = p ( s 1 s 2 (cid:1) (cid:1) (cid:1) s l t 1 t 2 (cid:1) (cid:1) (cid:1) t l ) (4) (cid:25) p ( s 1 s 2 : : : s l ) + p ( t 1 t 2 : : : t l ) (5) = p LM ( s 1 : l ) +  t 2  i t i p LM ( t 1: l ) (6) After expanding the mapping sequences (4), we take an approximation by estimating this as the sum of likelihoods of two sequences formed due to co-segmentations (5). The set of possible TC sequences is given by the Cartesian product of t i . These likelihoods can then be estimated using language model (LM) probabilities as shown in (6). Once the mapping sequence m has been found, all possible TC sequences are found from the set m t , which is the Cartesian product for all t i in m . From (7), we calculate approximate final sequence using beam search. Viterbi, a dynamic programming (DP) algorithm, considers phrases (or subsequences) and performs segmentation in a bottom-up' fashion (Nagata, 1994; Sproat et al., 1996). RNN-based language models are theoretically considered to be  1 '-gram (Khandelwal et al., 2018), which consitutes a challenge. Consider this sentence,    . A potential challenge could be to adquately estimate the probability of    . As this sequence occurs infrequently in the beginning of sentences in the corpus, an RNN would under-estimate the probability of this subsequence. Moreover, an RNN would likely lose some useful context and perform worse without it (Kim et al., 2019). So for Viterbi to perform well with an RNN, we train the language model on subsequences. We approach this by training our model in such a way that it samples subsequences ran-Viterbi Tokeniser       Approximate constructions from mapping table              TC LSTM  Beam sum SC LSTM Viterbi Tokenizer Subwordperplexities SC sentence TC Sentence Tokenized SC sentence Beam search on the segments using TC LSTM Converted TC sentence TC LSTM Figure 2: From the given SC sentence, we create possible TC sequences using mappings. domly in each epoch. As shown in Fig 1, we randomly split the sentence and use subsequences in separate epochs. Using Kudo (2018) regularization method, we sample from the n -best segmentations in each epoch. This is done so that the model can understand different segmentations of a subsequence using a similar motivation as above. Recent works have shown that varying subword segmentations lead to a better downstream model performance (Provilkov et al., 2019; Kudo, 2018; Hiraoka et al., 2019); therefore, we use it as a data augmentation strategy. Once we get the n -best segmentations with scores, we normalize them, and then use the normalized scores as sampling probabilities (see Fig 1). As opposed to other subword tokenizers where the vocabulary size is fixed, we do not limit the vocabulary in our model. Hence, there are numerous possibilities of segment combinations which raises a need of caching most frequent tokens. Inspired by the work related to cache-based LMs (Kawakami et al., 2017) and ghost batches (Hoffer et al., 2017), we only consider the top-k tokens in the main network memory and keep track of gradients of less recently used token embeddings (commonly known as LRU, Least Recently Used policy). This could be thought of as 7260 HK Literature HK News TW Literature TW News Sources Liu (1962) Singpao (2017-2018) Jiubadao (2011) AS subset Emerson (2005) Lau Yee (1972) Mingpao (2017-2018) Ko (2010) Liberty Times (2017-2018) Foon (1988) CityU subset Emerson (2005) Yao (1964) United Daily News (2017-2018) Average Length 194.8 214.6 188.2 223.6 IAA 0.982 0.979 0.981 0.971 Mapping Examples  [  ,  ,  ,  ]  [  ,  ,  ]  [  ,  ,  ]  [  ,  ,  ]  [  ,  ]  [  ,  ]  [  ,  ]  [  ,  ] Table 2: An overview of the dataset used for intrinsic evaluation. virtual embeddings as delayed gradient accumulation allows to accommodate larger number of tokens. This virtual size embedding architecture is related to the continuous cache implementation and stochastic tokenization architectures (Grave et al., 2016; Hiraoka et al., 2019). This optimal sequencing problem can be formulated as an overlapping subsequence approach, which can be solved using LM based Viterbi (Na-gata, 1994; Sproat et al., 1996). Fig. 2 explains this process of joint subword modelling. Here, we take Eq. (6) as the objective function for finding the mapping sequence, however, we use subword perplexities (Cotterell et al., 2018; Mielke et al., 2019; Mielke, 2019) in our implementation. For the TC LSTM, we add the probabilities of the beams of the possible sequences. As discussed in 2.1 and Eq. (7), beam search is needed to select the best subword sequence for TC. Once the sentences are tokenized, the mapping table is used to convert each SC token to the corresponding TC token. We extract the final TC sentence by resolving ambiguities through beam search using the TC LSTM (Fig. 2). We construct a gold standard corpus for both Chinese scripts consisting of 4 domains: HK Literature and Newswire, and Taiwanese Literature and Newswire (Table 2) with each domain containing 3000 sentences. SC-TC mapping tables are constructed from existing resources (Denisowski, 2019; Chu et al., 2012). We heuristically convert selected TC sentences to SC using OpenCC. We asked the annotators to manually correct any incorrect conversions. 4 4 A detailed data statement is given in the appendix. We choose the SIGHAN-2005 Bakeoff dataset to train the segmentation-based language model (Emerson, 2005). For SC, we select the PKU and MSR partitions, and for TC, we use the Academia Sinica and CityU partitions. We apply maximal matching (or heuristic dictionary-based word segmenter) to pre-process these datasets by segmenting words into subwords (Wong and Chan, 1996). Here, dictionary' refers to the word-list in the mapping table. We then train a 2-layer LSTM language model LSTM with tied weights, and embedding and hidden sizes of 512 (Sundermeyer et al., 2012) on this segmented dataset with subsequence sampling and stochastic tokenization as discussed in 2.2. Off-the-shelf Converters: Hanziconv 6 and Mafan 7 are dictionary-based script character converters. Evaluating this could be useful to understand the lower accuracy bound. OpenCC 8 uses a hybrid of characters and words (specifically trie based tokenizer) for script conversion (Pranav A et al., 2019). Language Model Disambiguation: A strong baseline to this problem would be to build a language model to disambiguate between the characters, which is quite similar to STCP (Xu et al., 2017). We use a 2-layer LSTM language model trained on Traditional Chinese corpus. Neural Sequence Models: We heuristically convert Traditional Chinese Wikipedia to Simplified Chinese using OpenCC and use it for training the seq2seq model (Sutskever et al., 2014). We 5 If in case you are looking for Related Work' section. 6 https://github.com/berniey/hanziconv 7 https://github.com/hermanschaaf/mafan 8 https://github.com/BYVoid/OpenCC 7261 Conversion System HK Lit HK News TW Lit TW News Overall DED SA DED SA DED SA DED SA DED SA Dictionary based conversion, Hanziconv 34.1 54.7 37.7 59.1 31.3 60.0 39.3 58.9 34.2 55.6 Dictionary based conversion, Mafan 14.7 71.2 17.7 72.5 14.5 73.8 13.3 72.7 14.4 72.6 Trie dictionary based conversion, OpenCC 5.5 87.3 5.1 83.4 4.1 84.7 3.8 88.5 4.3 85.3 Language Model Disambiguation, STCP 6.3 85.6 5.4 79.9 4.7 84.1 5.2 83.9 5.3 84.0 Convolutional Sequence Models 6.7 85.8 5.3 79.3 4.8 84.5 5.2 83.9 5.4 84.4 2kenize with word tokenization 11.2 84.3 12.1 81.3 11.3 82.1 10.0 81.1 11.5 82.7 2kenize with maximal matching 5.2 88.7 3.3 93.1 4.0 88.6 4.8 87.7 4.5 88.9 2kenize with Unigram subwords 3.4 91.9 3.8 90.9 4.3 88.1 3.9 87.8 3.7 89.3 2kenize with joint LSTM modelling 2.8 94.9 3.1 93.7 3.8 91.3 2.9 91.9 3.0 92.4 Table 3: Results of the intrinsic evaluation experiments which are reported as a mean across 10 different seeds. construct a 20-layer neural convolutional sequence model (Gehring et al., 2017) (both in encoder and decoder) using fairseq (Ott et al., 2019). Dictionary substrings : We apply maximal string matching, which is a dictionary based greedy tokenizer (Pranav A et al., 2019; Wong and Chan, 1996). Unigram from Sentencepiece: Subword segmentation is performed by sampling unigram language model perplexity values (Kudo, 2018). We evaluate our models using the metrics of disambiguation error density (DED) and sentence accuracy (SA). DED is the average of total edit distances per 1000 ambiguous Simplified characters, which is  edit distances  ambiguous Simplified characters (cid:2) 1000 . SA is the number of sentences correctly converted in percentages. Contrary to previous papers, we do not report character based accuracy values, as generally most characters have straightforward mappings  a reason why we opt for a less forgiving metric like SA where every character in a sentence has to be correctly converted. Results are shown in Table 3, broken down by domain, and overall. Our model attains an average DED of 3.0 and a SA of 92.4% overall, whereas the best existing converter, OpenCC, only achieves a DED of 4.3 and a SA of 85.3%. We 9 https://github.com/fxsjy/jieba find that seq2seq and LM based disambiguation perform almost on par with OpenCC, due to the large number of false positive errors by these models. Jieba achieves an average DED of 11.2 as it does not handle OOV words well. For maximal matching of segmented words and Unigram subwords, it achieves an overall DED of 4.5 and 3.7, respectively  showing that joint segmentation yields better results. We observe that accuracy values are slightly worse on news text, due to the relatively high number of new entities in those datasets. We find that seq2seq and LM based disambiguation gives rise to many false positives. Heuristically converting TC to SC results in certain conversion errors in the training dataset; and additionally, seq2seq approaches tend to reword the target sentence, which shows that they are unsuitable for this task. We manually inspect incorrect conversions in the intrinsic evaluation and find four interesting recurring linguistic patterns which confused the converters. We instructed the annotators to classify the items in the dataset (overall 12000 sentences in intrinsic evaluation dataset) if the sentences contain any of these patterns. In Table 4, we provide an overview of statistical information of these patterns and the performance by the converters. Code mixing : Vernacular Cantonese characters ( zh-yue ) are a subset of TC characters but do not follow the norms of the standard written Chinese (Snow, 2004). We find that some of the sentences in our dataset are code-mixed with zh-yue (e.g. speech transcription) or English (e.g. named entities). Consider the snippet, ...  BENZ 190E  ..., which is code-mixed with 7262 Case Method SA Example Code mixing with Cantonese (34 cases, 0.3%)  ,  ? both zh-yue and English. The characters BENZ 190E,  and  are not a part of standard written Chinese. We find that OOV words are 2kenize d into single-character tokens which results in:    j  j B j E j N j Z j 1 j 9 j 0 j E j  j  j   Thus, 2kenize distributes the entropy over multiple tokens rather than a single token (generally UNK is used in such cases). This allows the language model to have more space for multiple guesses, which shows a massive advantage over word models or just UNK ing it, a reason why subword tokenizers outperform closed-vocabulary models (Merity, 2019). Disguised Named Entities : Take the recurring sentence:   . Observe that the sentence contains a frequent word  (China). However, the actual meaning and English translation do not include China at all. This is an interesting linguistic trait of Chinese, where words often appear in the sentence, but are not being interpreted. This could easily trip up a tokenizer, as the probability of  being a token independently is high. Having  as a separate token in the sentence could lead into an incorrect conversion (Table 1). We find in 2kenize r's trellis 10 that    j  j   has a higher probability than other possible segmentations. Substructure lookups and beam search in our setup considerably reduces the probability of getting wrong tokenization. The sen-10 This is a probability lookup table in Viterbi to keep track of the segment information in a subsequence. tence is 2kenize d into   j  j  j  j   j  , which results in the correct conversion   . Repetitions : We find that in 3.57% of sentences, named entities are repeated. Interestingly, STCP, which uses a language model for disambiguation, often only converts one out of the repeated tokens correctly, which we can see in the table. As also shown, STCP prefers  over  in the first occurrence, but then prefers  11 in the second occurrence as it gets more context. 2kenize converts both of the entities correctly, very likely due to substructure lookups. Failure Cases : Dictionary-based converters (OpenCC, HanziConv and Mafan) only use the first conversion candidate 12 if multiple candidates are available. STCP often converts named entities wrongly, especially the ones which have long-range dependencies and repetitions. Although we find that 2kenize converts some of the unseen named entities perfectly, some of the errors caused were due to infrequent characters. Few cases are mainly related to variant characters 13 which are often used interchangeably. An accurate script converter should produce a less erroneous dataset, which should in turn improves the accuracy of the downstream tasks. In this section, we demonstrate the effect of script conversion on topic classification tasks to examine this assumption. We also study the impact of tokenization and pooling on the accuracy of topic classification. We apply the converter to the language modelling corpus (Wikitext), then train a classifier for informal and formal topic classification on that translated data. This allows us to measure the performance of the converter compared to other ones for a specific downstream task. This section describes the dataset that we used for extrinsic evaluation experiments. It involves a pretraining dataset which consists Chinese Wikipedia and topic classification datasets. We use Chinese Wikipedia articles for pretraining the language model. Script conversion is an issue in Chinese Wikipedia, and currently, they use a server-side mechanism to automatically convert the scripts (dictionary-based) based on the location of the user. However, Wikipedia provides an option to view the article without conversion, which we use in the corpus. 14 We use zh-CN , zh-HK and zh-yue wikis to retrieve articles originally written SC, TC and vernacular Cantonese + TC respectively with the help of wikiextractor. 15 We pretrain the formal text classification models on articles from zh-HK and converted zh-CN ; and classification models for informal text on articles from zh-HK , zh-yue , and converted zh-CN . We choose two classification tasks: formal news and informal topic classification (Table 5). For formal news, we scrape recent articles (2017-2019) from Singtao, 16 for informal topics, we scrape posts (2017-2018) from LIHKG. 17 4.2 Performance of various classifiers For classification baselines, we use character-based SVM (Support Vector Machines, Joachims (1998)), CNN (Convolutional Nets, Zhang et al. (2015)) and Chinese BERT (Devlin et al., 2019). We also employ a state-of-the-art text classifier, MultiFiT (Eisenschlos et al., 2019), a lightweight RNN-based language model based classifier, which has shown to achieve a performance competitive with BERT (Devlin et al., 2019) and ULMFiT (Howard and Ruder, 2018). The base architecture of MultiFiT is a 4-layer QRNN (Brad-bury et al., 2016) with classifier head. We choose rectified Adam (Liu et al., 2019) with Lookahead (Zhang et al., 2019) as the optimizer. We employ the cosine cyclic learning scheduler (Smith, 2015), where the limits of learning rate cycles are found by increasing the learning rate logarithmically and computing the evaluation loss for each learning rate (Smith, 2018). To compute the batch size, we apply gradient noise scale to each batch size candidate and pick the one which gives the highest gradient noise scale (McCandlish et al., 2018). We apply label smoothing (Szegedy et al., 2015) and use mixed precision training on RTX 2080. We implement our experiments using Pytorch (Paszke et al., 2019) and FastAI (Howard and Gugger, 2020). MultiFiT uses concat pooling after the last layer of QRNN, which means that the last time step is concatenated with an average and maximum 14 Note that it would not be straight-forward to compare against Wikipedia's conversions directly, because they also perform some degree of manual post-processing (Contribu-tors, 2019). pooled over previous time steps. Studies show that in LM based classifiers, different layers capture different types of knowledgethe last layer would be domain-specific and initial layers would be more generalized (Yosinski et al., 2014; Peters et al., 2019). We speculate that concat pooling only on the last layer limits the information available to the classifier head and we hypothesise that the classifer would perform better if domain-specific as well as generalized knowledge were available to the head. For this reason, we augment the original MultiFIT architecture with layer pooling , which is concat pooling from all the layers, and pass that to the dense layer in the classifier, as shown in Fig 3. We fine-tune the BERT language model and pretrain the MultiFiT language model on Chinese Wikipedia subsets (4.1.1). All classifiers are then trained on the given training set (character based models) and evaluated on the test set in terms of accuracy as number of items in each class are roughly equal. This experiment (and subsequent experiments in this section) is repeated across ten different seeds (Reimers and Gurevych, 2018) and data splits (Gorman and Bedrick, 2019) and the results are shown in Table 6. Layer pooling shows an absolute improvement of 0.4% improvement over concat pooling on formal and informal topic clas-Pretraining data of MultiFiT Formal Informal No Conversions 89.0 70.3 Including conversions with OpenCC 91.7 75.6 Including conversions with STCP 92.3 73.4 Including conversions with 2kenize 93.2 77.9 Table 7: Ablation test of MultiFiT on different script converters. For each converter (OpenCC, STCP, 2kenize ), we translate zh-CN wiki dataset and augment it with the TC wiki dataset. Then, we pretrain on this dataset, finetune on the domain data and train MultiFiT with layer pooling on these three datasets. We demonstate test set accuracies in Table 7. The dataset translated by 2kenize outperforms other converters, giving an absolute improvement of 0.9 % on formal and 4.5% over second-best converters on informal topic classification. These results emphasise that better script conversion improves the quality of the pretraining dataset, which boosts the performance of the downstream tasks like topic classification. Studies show that tokenization affects classification accuracy; open-vocabulary methods generally perform best (Eisenschlos et al., 2019; Hiraoka et al., 2019). For this experiment, we perform further ablations on our previous best classifier setup (MultiFiT with layer pooling on 2kenize ) to understand the effect of various subword tokenizers. Pretraining generally takes a long time (1-2 GPU days), hence we pretrain the classifier once for each tokenized corpus and do not perform sub-7265 10 0 10 1 10 2 10 3 10 4 Order Rank 10 0 10 1 10 2 10 3 F r e q u e n c y Character, 1.703 Word, 1.412 BPE Drop, 1.31 BPE, 1.27 Unigram, 1.26 1kenize, 1.10 Figure 4: Log-log plots for different tokenizers. word sampling for this experiment. For closed vocabulary methods, we use character and word segmentations (here with Jieba). Likewise, for open-vocabulary methods, we employ BPE, BPE-Drop and Unigram subword tokenizers. Subword tokenizers mostly rely on frequency and do not take likelihood (something similar to n -gram language model) of tokenized sentence into consideration. Hence, we choose LM-based Viterbi segmentation (henceforth referred as 1kenize ), and here the LM would be the TC LSTM described in 2.2. We report results in Table 8. We find that for formal classification, 1kenize and Unigram perform best. 1kenize outperforms other subword tokenizers for the noisier informal dataset, giving an absolute improvement of 0.5% over the second best method, which is BPE-Drop. We plot a log frequency of tokens vs log order rank, which is shown in Figure 4. This distribution is based on the LIHKG dataset, which is noisier than other domains. We observe that character and word distributions are steeper than language model based subword tokenizers. This indicates that subword tokenizers produce a less skewed token distribution. Subword tokenizers like BPE and Unigram are deterministic and rely on frequency for segmentation. Since 1kenize is contextual, being LM-based, we find that it produces the least skewed distribution (lowest Zipf's law coefficient (Zipf, 1949)), which also reduces variance, a reason why this simple segmentation method outperforms others for informal text classification. The contributions of our work are: 2kenize , a subword segmentation model, which jointly segments source sentence and its corresponding approximate target conversions. An unsupervised script converter based on 2kenize which shows a significant improvement over existing script converters and supervised models. 1kenize , a variant of 2kenize which performs tokenization on only Traditional Chinese sentences which improves accuracy on topic classification tasks. Character conversion evaluation datasets : spanning Hong Kong and Taiwanese literature and news genres. Traditional Chinese topic Classification datasets : formal (scraped from Singtao) and informal (scraped from LIHKG) styles spanning genres like news, social media discussions, and memes. The key findings of our work are:  Our script converter shows a strong performance when dealing with code mixing and named entities. Supervised models are prone to anaphora and unseen entities related errors. A simple LM-based Viterbi segmentation model outperforms other subword tokenizers on topic classification tasks and reduces skewness of token distribution on a noisy dataset. We leave some open questions to explore:  How can we exploit subword variations to reduce skewness in the NLU tasks? Would subword-segmentation-transfer be helpful for other NMT-NLU task pairs like we did for 2kenize (script conversion) to 1kenize (classification)? We anticipate that this study would be useful to TC NLP practitioners, as we address several research gaps, namely script conversion and a lack of benchmark datasets. The first author would like to thank Dayta AI Limited, S.F. Hui, I-Tsun Cheng, Ishaan Batra, Conrad Ho, Roy Fork, Abhishek Gupta, Ajay Singh, Eugene Ho, Patrick Tu, Alex Chu, and Leland So for making valuable additions to this work. The second author would like to acknowledge funding from the Swedish Research Council for the project under grant agreement 2019-04129, which partly funded this work.
abstain abstain objective abstain method objective abstain result other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method abstain abstain abstain objective result abstain result method result abstain abstain abstain abstain other objective objective method abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain method abstain method other abstain abstain abstain abstain method abstain method abstain abstain abstain method method abstain method abstain abstain method method method abstain method abstain abstain abstain abstain method method abstain other abstain method abstain abstain abstain abstain abstain abstain result other abstain abstain abstain result abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain method abstain method method method method abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other