The evolution of language follows the rule of gradual change. Grammar, vocabulary, and lexical semantic shifts take place over time, resulting in a diachronic linguistic gap. As such, a considerable amount of texts are written in languages of different eras, which creates obstacles for natural language processing tasks, such as word segmentation and machine translation. Although the Chinese language has a long history, previous Chinese natural language processing research has primarily focused on tasks within a specific era. Therefore, we propose a cross-era learning framework for Chinese word segmentation (CWS), CROSSWISE, which uses the Switch-memory (SM) module to incorporate era-specific linguistic knowledge. Experiments on four corpora from different eras show that the performance of each corpus significantly improves. Further analyses also demonstrate that the SM can effectively integrate the knowledge of the eras into the neural network. As a human-learnable communication system, language does not remain static but instead evolves over time. The rate of change between different aspects of language, such as grammar, vocabulary, and word meaning, vary due to language contact and many other factors, which has led to the diachronic linguistic gap. An example of this can be seen in, That slepen al the nyght with open ye (That sleep all the night with open eye), which is a sentence from The Canterbury Tales, written in Middle English by Geoffrey Chaucer at the end of the 14th century. It is difficult for people without an understanding of Middle English to make sense of this sentence. Furthermore, some discourses  Corresponding author Sample from MSR Golds (wait) (who) (come) (slove) (ne) contain both modern English and Old English due to citation or rhetorical need. For example, Shake-speare's fourteen lines of poetry are often quoted in contemporary novels. This kind of era-hybrid text creates barriers to natural language processing tasks, such as word segmentation and machine translation. The Chinese language has the honor of being listed as one of the world's oldest languages and, as such, has seen several changes over its long history. It has undergone various incarnations, which are recognized as Archaic (Ancient) Chinese, Middle Ancient Chinese, Near Ancient Chinese, and Modern Chinese. Notably, most Chinese NLP tasks skew towards Modern Chinese. Previous research has primarily focused on addressing the CWS problem in Modern Chinese and has achieved promising results, such as Chinese Word Segmentation (CWS) (Zheng et al., 2013; Chen et al., 2015; Zhang et al., 2016; Xu and Sun, 2016; Shao et al., 2017; Yang et al., 2017; Zhang et al., 2018; Tian et al., 2020b,a). Although CWS for ancient Chinese has been recognized in recent years, the processing of language-hybrid texts is still an open question. As shown in 7830 Table 1, PKUSeg (Luo et al., 2019a) is a Chinese segmenter that is trained with a modern Chinese corpus; while it can segment modern Chinese sentences correctly, its accuracy drops sharply when applied to ancient Chinese. Conversely, the ancient Chinese segmenter JiaYan 1 performs well on ancient Chinese text but fails to perform well on Modern Chinese texts. Therefore, it is necessary to develop appropriate models to undertake cross-era NLP tasks. To address this need, we propose CROSSWISE (CROsS-ear Segmentation WIth Switch-mEmory), which is a learning framework that deals with cross-era Chinese word segmentation (CECWS) tasks. The framework integrates era-specific knowledge with the Switch-memory mechanism to improve CWS for era-hybrid texts. More specifically, we utilized the abilities of both CWS and sentence classification tasks to predict segmentation results and era labels. We also incorporated the Switch-memory module to include knowledge of different eras, which consists of key-value memory networks (Miller et al., 2016) and a switcher. In order to store era-specific knowledge by several memory cells, key-value memory networks are used. The sentence discriminator is considered to be a switcher that governs the quantity of information in each memory cell that is integrated into the model. For each memory cell, we map candidate words from the dictionary and word boundary information to keys and values.. Cross-era learning is introduced for CWS, we share all the parameters with a multi-task architecture. The shared encoder is used to capture information that several datasets from different eras have in common. This single model can produce different words segmentation granularity according to different eras. The Switch-memory mechanism is used to integrate era-specific knowledge into the neural network, which can help improve the performance of out of vocabulary (OOV) words. This study proposes two switcher modes ( hard-switcher and soft-switcher ) to control the quantity of information that each cell will feed into the model. Experimental results from four CWS datasets with different eras confirm that the performance of each corpus improves significantly. Further analyses also demonstrate that this model is flexible for cross-era Chinese word segmentation. Chinese word segmentation is generally considered to be a sequence labeling task, namely, to assign a label to each character in a given sentence. In recent years, many deep learning methods have been successfully applied to CWS (Zheng et al., 2013; Chen et al., 2015; Zhang et al., 2016; Xu and Sun, 2016; Shao et al., 2017; Yang et al., 2017; Kurita et al., 2017; Liu et al., 2018; Zhang et al., 2018; Ye et al., 2019a; Higashiyama et al., 2019; Huang et al., 2020b; Tian et al., 2020b,a,c; Liu et al., 2021). Among these studies, some indicate that context features and external knowledge can improve CWS accuracy (Kurita et al., 2017; Yang et al., 2017; Zhang et al., 2018; Liu et al., 2018; Tian et al., 2020b,a,c). Studies from Liu et al. (2018) and Zhang et al. (2018) leveraged the dictionary to improve the task; n-gram is also an effective context feature for CWS (Kurita et al., 2017; Tian et al., 2020b; Shao et al., 2017). The use of syntactic knowledge generated by existing NLP toolkits to improve CWS and part-of-speech (POS) has been established by Tian et al. (2020b). Furthermore, Tian et al. (2020c) incorporated wordwood information for neural segmenters and achieved a state-of-the-art performance at that time. It is common practice to jointly train CWS and other related tasks based on a multi-task framework. Chen et al. (2017) took each segmentation criterion as a single task and proposed an adversarial multitask learning framework for multi-criteria CWS by extracting shared knowledge from multiple segmentation datasets. Yang et al. (2017) investigated the effectiveness of several external sources for CWS by a globally optimized beam-search model. They considered each type of external resource to be an auxiliary classification, and then leveraged multi-task learning to pre-train the shared parameters used for the context modeling of Chinese characters. Liu et al. (2018) jointly trained the CWS and word classification task by a unified framework model. Inspired by these successful studies, this study also incorporated ideas from the multi-task framework, and jointly trained the CWS task and 7831  M k a i + n a i+1 a i a 1 h i + n h i+1 h 1 h i  M 1  O I B I B   / Switch Memory Encoder Decoder Prediction Output Input Dis     Key Value         D 1 Memory cell VBVE----VS---VS------VS-----VS-----VS---VBVE--VB-VE  yellow river Yellow River enter sea mouth Haikou estuary  Figure 1: CROSSWISE for cross-era Chinese word segmentation. Recently, some studies have noticed the linguistic gap due to the differences between eras. Ceroni et al. (2014) proposed a time-aware re-contextualization approach to bridge the temporal context gap. Chang et al. (2021) reframed the translation of ancient Chinese texts as a multi-label prediction task, then predicted both translation and its particular era by dividing ancient Chinese into three periods. The use of key-value memory networks were introduced to the task of directly reading documents and answering questions by Miller et al. (2016), which helped to bridge the gap between direct methods and the use of human-annotated or automatically constructed Knowledge Bases. Tian et al. (2020c) applied this mechanism to incorporate n-grams into the neural model for CWS. Encouraged by the above works, this study designed a multi-task model for cross-era CWS by jointly training the sentence classification task and CWS through the use of a unified framework model. Key-value memory networks are used to integrate era-specific knowledge into the neural network, as was done in research by Tian et al. (2020c). 3 The Proposed Framework 3.1 BERT-CRF model for Chinese word Segmentation Chinese word segmentation is generally viewed as a character-based sequence labeling task. Specifically, given the sentence X = { x 1 , x 2 , ...x T } , each character in the sequence is labeled as one of L = { B, M, E, S } , indicating the location of the character as at the beginning, middle, or end of a word, or that the character is a single-character word. CWS aims to determine the ground truth of labels Y  = { y  1 , y  2 , . . . y  T } : Y  = arg max P ( Y | X ) Y L T (1) The universal end-to-end neural CWS architecture usually contains an encoder and a decoder. The framework used in this study is shown in Figure 1; the functions of each part are explained below. Encoding layer. According to Fu et al. (2020), although BERT-based (Devlin et al., 2019) models for CWS are imperfect, BERT is superior, in many aspects, to models that have not been pre-trained. For example, BERT is more suitable for dealing with long sentences; therefore, this study utilizes BERT released by Google Devlin et al. (2019) as the shared encoder, which is pre-trained with a large amount of unlabeled Chinese data. where h i is the representation for x i from the encoder. Decoding layer. This study is able to use a shared decoder for samples from different eras because era-aware representation have been combined for each character by the Switch-memory module. There are various algorithms that can be implemented as decoders, such as conditional random fields (CRF) (Lafferty et al., 2001) and softmax. According to (Tian et al., 2020c), CRF performs better in word segmentation tasks. Therefore, considering the framework of this study, CRF is used as the decoder. In the CRF layer, P ( Y | X ) in Eq. 1 can be represented as: P ( Y | X ) =  ( Y | X ) (cid:80) Y  L T  ( Y  | X ) (3) where,  ( Y | X ) is the potential function, and only interactions between two successive labels are considered. ( Y | X ) = T (cid:89) i =2  ( X, i, y i  1 , y i ) (4)  ( x , i, y  , y ) = exp ( s ( X, i ) y + b y  y ) (5) where b y  y  R is trainable parameters respective to label pair ( y  , y ) . The score function s ( X , i )  R |L| calculate the score of each lable for i th character: s ( X , i ) = W  s a i + b s (6) where a i is the final representation for i th character. W s  R d a  L and b s  R |L| are trainable parameters. The Switch-memory consists of d memory cells and a switcher. For an input sentence, there are d memory cells for each character. The switcher governs how much information in each cell will be integrated into the network. And the state of the switcher depends on the sentence classifier. The dictionary has been a useful external source to improve the performance of CWS in many stud-ies.(Yang et al., 2017; Liu et al., 2018; Zhang et al., 2018). However, the ability to incorporate the dictionary into previous research has been limited by either concatenating candidate words and character embeddings or the requirement of handcrafted templates. In this study, key-value memory networks are utilized to incorporate dictionary information, which is initially applied to the Question Answering (QA) task for improved storage of prior knowledge required by QA. Furthermore, this network structure can also be used to store the existing knowledge that is required by cross-era CWS. Ancient Chinese is not a static language but is instead a diachronic language. Ancient Chinese has three development stages: Ancient, Middle Ancient, and Near Ancient. Each stage has a specific Rule V i,j x i is the beginning character of w i,j . lexicon and word segmentation granularity. Therefore, this research has constructed four dictionaries D = { D 0 , D 1 , D 2 , D 3 } , associating with the four development stages of Chinese, respectively, and each dictionary is era-related. When input a sentence, four memory cells are generated for each character in the sentence according to the four dictionaries, and each memory cell maps candidate words and word boundary information to keys and values. Candidate words as keys. Following Tian et al., for each x i in the input sentence, each dictionary has many words containing x i , we only keep the n-grams from the input sentence and appear in each dictionary, resulting w di = { w di, 1 , w di, 2 , ...w di,j ...w di,m i } , x i is a part of word w di,j  D d , d  [0 , 3] . We use an example to illustrate our idea. For the input sentence show in Figure 1, there are many n-grams containing x 3 =   (sea), we only retain ones that appear in D 0 for the first memory cell, thus, w 03 = {    (HaiKou) ,     (estuary) ,   (sea) } . Similarly, we can generate w 13 , w 23 , w 33 for the second, third and fourth memory cell according to D 1 , D 2 , D 3 . Then, the memory cell compute the probability for each key (which are denoted as e wi,j for each w di,j ), here h i is the embedding for x i , which is encoded by the encoder. Word boundary information as values. As we know, CWS aims to find the best segment position. However, each character x i may have different positions in each w di,j . For example, x i may be at the beginning, middle, ending of w di,j , or x i maybe a single word. Different positions convey different information. Therefore, we use the boundary information of candidate words as values for key-value networks. As shown in Table 2, a set of word boundary values { VB , VE , VS } with embeddings { e VB , e VE , e VS } represent the x i 's different positions in w di,j , and we map x i 7833 to different value vectors according to its positions. As a result, each w di for x i has a values list V di = [ v di, 1 , v di, 2 , ...v di,j , ...v di,m i ] . In Figure 1, x 3 =   (sea), for the first memory cell, we can map candidate word boundary information to the value list V 03 = [ VS , VB ] . Four cells for x i has a values list V i = [ v 0 i , v 1 i , v 2 i , v 3 i ] . Then the d th memory cell embedding for x i is computed from the weighted sum of all keys and values as follow. m i (cid:88) where e v d i,j is the embedding for v di,j . Next, the final character embedding is the element-wise sum of o i and h i , or their concatenation, passing through a fully connected layer as follow: a i = W o  ( o i  h i ) (9) where  operation could be sum or concatenate, W o  RT is a trainable parameter and the output a i  RT is the final representation for the i th character. o i is the final memory embedding for the i th character, and can be calculated as follow. where Switcher is used to control how much information in each memory cell will be combined with the output of the encoder. Inspired by the benefits of multi-task, a classifier has been added on top of the encoder to predict the era label of the input sentence. The discriminator predicts the probability of the correct era label, z , conditioned on the hidden states of the encoder, H , which is the output of [CLS] from BERT. The loss function of the discriminator is J disc =  logP ( z | H ) , through minimizing the negative cross-entropy loss to maximizes P ( z | H ) . In this study, H is fed into a fully-connected layer and let it pass through a softmax layer to obtain probabilities for each era label. Switch mode. For the switcher, we propose two switcher modes, hard-switcher and soft-switcher . Hard-switcher switches memory cells according to the predicted final result from the discriminator. For the input sentence in Figure 1, if the predicted result is the modern era, then the switcher will switch to the memory cell associated with modern Chinese, and o i = o di . Soft-switcher switches memory cells according to the predicted probability, which is calculated by the weight of each memory cell. Soft-switcher means that the information from all four dictionaries may be fused into the current character's representation. For example, the predicted probability list is [0 . 1 , 0 . 2 , 0 . 1 , 0 . 6 ] ; therefore, the final memory representation for the i th character is o i = o 0 i  0 . 1 + o 1 i  0 . 2 + o 2 i  0 . 1 + o 3 i  0 . 6 . In this framework, the discriminator is optimized jointly with the CWS task, which both share the same encoding layer. Different weights are assigned to the loss of the two tasks, the final loss function is: J =  J cws + (1   ) J disc (11) where  is the weight that controls the interaction of the two losses. J cws is the negative log likelihood of true labels on the training set. where N is the number of samples in the training set, and Y n is the ground truth tag sequence of the n th sample. The model proposed in this study has been evaluated on four CWS datasets from Academia Sinica Ancient Chinese Corpus 2 (ASACC) and SIGHAN 2005 (Emerson, 2005). The statistics of all the datasets are listed in Table 3. Among these datasets, PKIWI, DKIWI, AKIWI from ASACC, correspond to near ancient Chinese, middle ancient Chinese, ancient Chinese, respectively, and MSR from SIGHAN 2005 is a modern Chinese CWS dataset. It should be noted that PKIWI, DKIWI, and AKIWI are traditional Chinese and were translated into simplified Chinese prior to segmentation. For PKIWI, DKIWI, and AKIWI, 5K examples were randomly picked as a test set; then, 10% of examples were randomly selected from training set as the development set. Similar to previous work (Chen et al., 2017), all datasets are pre-processed by 2 http://lingcorpus.iis.sinica.edu.tw/ ancient 7834 Datasets Words Chars Word types Char Types Sents OOV Rate ASACC AKIWI Train 2.8M 3.2M 65.3K 7.5K 59.7K Test 0.2M 0.3M 15.7K 4.4K 5K 4.35% DKIWI Train 2.2M 2.8M 44.3K 6.0K 50.1K Test 0.2M 0.3M 13.0K 3.8K 5K 4.91% PKIWI Train 6.4M 7.8M 117.0K 7.2K 144.1K Test 0.2M 0.3M 18.6K 4.4K 5K 1.71% SIGHAN05 MSR Train 2.4M 4.1M 88.1K 5.2K 86.9K Test 0.1M 0.2M 12.9K 2.8K 4.0K 2.60% Table 3: Detail of the four datasets. replacing Latin characters, digits, and punctuation with a unique token. In the cross-era learning scenarios, all of the training data from four eras corpora were used as the training set. Then, all of the test data from four corpora were used as the cross-era test set to evaluate the model. Finally, F1 and OOV recall rates ( R oov ) were computed according to the different eras. In our experiments, for the encoder BERT, we follow the default setting of the BERT (Devlin et al., 2019). The key embedding size and value embedding size are the same as the output of the encoder, and they have been randomly initialized. For the baseline model Bi-LSTM, the character embedding size is set to 300, and the hidden state is set to 100. For the transformer, the same settings as Qiu et al. (2020) were followed. The loss weight coefficient  is a hyper-parameter that balances classification loss and segmentation loss; the model achieves the best performance when  is set to 0.7, which was identified by searching from 0 to 1 with the equal interval set to 0.1. The words from the training set are used as the internal dictionary, and each training set generates its own dictionary. The simplified Chinese dictionary sourced from jieba 3 is used as the external dictionary for MSR, and words from The ErYa (an ancient dictionary) and ancient Chinese textbooks were extracted as the external dictionary for AWIKI. For PWIKI and DWIKI, high-frequency bi-grams and tri-grams were extracted from the corresponding period corpus 4 as external dictionaries. To begin, in this section, the experimental results of the proposed model on the test sets from the four cross-era CWS datasets are provided, which can be seen in Table 4. Several observations can be made from the data provided in Table 4. First, BERT-CRF in single-era scenarios (ID:1 in Table 4) and cross-era learning without the SM module (ID:6) are compared. As can be seen in the table, when mixing four datasets, the average F1 value of all datasets decreases slightly. Single-era dataset learning has an average F1 value of 97.61, while cross-era learning without the Switch-memory module has a 97.32 average F1 value. This indicates that performance cannot be improved by merely mixing several datasets. Second, the models with the SM mechanism (ID:3,5,7) outperformed the baseline models (ID:2,4,6) in terms of F1 value and R oov on all datasets. For example, the average F1 score for BERT-CRF with SM module (ID:7) improved by 0.92% when compared to BERT-CRF (ID:6), and the average R oov went from 76.15 to 82.37. This indicates that the Switch-memory can help improve segmentation and R oov performance by integrating era-specific knowledge. Third, among different encoders, the improvement of the pre-trained encoder BERT on the F1 value is still significant. When using Bi-LSTM as the encoder (ID:2,3), the average F1 value and the R oov are 89.15 and 90.66, respectively. When using BERT as the encoder (ID:6,7), the F1 value improves by approximately 8%. The reason for this may be that the pre-training processing supplements some effective external knowledge. To further illustrate the validity and effectiveness of this model, the best results from this study are compared to works that have been previously 7835 NO. identified as state-of-the-art. Various aspects of multi-domain and multi-criteria Chinese word segmentation are very similar to the tasks in this study; therefore, this study reproduced experiments on several previous word segmentation models using the four datasets identified in this research (Luo et al., 2019b; Qiu et al., 2020; Huang et al., 2020a). For the multi-domain segmenter PKUSeg (Luo et al., 2019b), four datasets were trained with the pre-trained mixed model. The comparison is shown in Table 5; the model from this study outperforms previous methods. Table 6 shows the effectiveness of each component in the SM module. The first ablation study is conducted to verify the effectiveness of memory cells. In this experiment, the sentence classification task is no longer a switcher but simply a joint training task with word segmentation. We can see that the ancient Chinese datasets (AWIKI, DWIKI, PWIKI) are more sensitive to memory cells than MSR. This may be explained by the fact that the encoder is pre-trained with a large quantity of modern Chinese data, and the memory cells in this study incorporate some ancient era knowledge into the model, which helps to boost the performance of the three ancient Chinese datasets. The second ablation study is to evaluate the effect of the switcher. For this experiment, the average of four embedded memory cells is used as the final memory representation. The comparison between the second and the third line indicates that the switcher is an important component when integrating era-specific information. In summary, in terms of average performance, the switcher and the memory cells can both boost the performance of R oov considerably. In this section, the effect of the switcher mode and the combination mode (concatenate or sum) of memory embedding and character embedding is investigated. To better understand the effect of the different configurations, this study examines the four pair settings to train the model on the four datasets in this study; the results are shown in Figure 2, and different color bars represent different datasets. As can be seen, soft-switcher significantly improves the F1 value on MSR compared to hard-switcher , while the other three datasets prefer hard-switcher , which suggests that the forward direction of knowledge dissemination from ancient Chinese to mod-7837 ern Chinese can help modern Chinese word segmentation, and that the reverse knowledge dissemination will have a negative impact on ancient Chinese word segmentation. Concatenating memory embedding and character embedding from the encoder outperforms the combination of the two; therefore, this study chose the pair of configurations, hard +concat, to obtain the experimental results in the last row of Table 4 and Table 5. 4.6 Case study This study further explores the benefits of the SM mechanism by comparing some cases from BERT-CRF and CROSSWISE. Table 7 lists three examples from the test sets of Ancient Chinese and modern Chinese datasets. According to the results, in the first sentence,   (swept) and   (grass) are two words in ancient Chinese, BERT-CRF treats these two words as a single word; BERT-CRF gives the second sentence the wrong boundary prediction in   (middle) and   (through). However, this study's CROSSWISE achieves all exact segmentation of these instances. The third sample is a sentence written in both ancient and modern Chi-nese,       , which is a famous classical sentence in ancient Chinese. CROSSWISE also can split the sentence correctly. Therefore, it can be concluded that the model is flexible for Chinese word segmentation of era-hybrid texts and can produce different segmentation granularity of words according to the era of the sentence. Concurrently, it shows that the SM mechanism is effective in integrating era-specific linguistic knowledge according to different samples. In this study, a flexible model, called CROSSWISE, for cross-era Chinese word segmentation is proposed. This model is capable of improving the performance of each dataset by fully integrating era-specific knowledge. Experiments on four corpora show the effectiveness of this model. In the future, the incorporation of other labeling tasks into CROSSWISE, such as POS tagging and named entity recognition, may prove to be insightful. This research is supported by the NSFC project the Construction of the Knowledge Graph for the History of Chinese Confucianism (Grant No. 72010107003). We would like to thank Professor Jun Wang and Hao Yang for their insightful discussion. The datasets used in this paper are open datasets and do not involve any ethical issues.
abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain objective abstain result abstain abstain abstain abstain abstain abstain objective objective objective objective objective objective other other other other other other other other other other other method other other other other other method other other other other abstain other other method other other method other other method other other other other other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain result abstain abstain abstain abstain objective abstain abstain abstain other other method