Recently, word enhancement has become very popular for Chinese Named Entity Recognition (NER), reducing segmentation errors and increasing the semantic and boundary information of Chinese words. However, these methods tend to ignore the information of the Chinese character structure after integrating the lexical information. Chinese characters have evolved from pictographs since ancient times, and their structure often reflects more information about the characters. This paper presents a novel Multi-metadata Embedding based Cross-Transformer (MECT) to improve the performance of Chinese NER by fusing the structural information of Chinese characters. Specifi-cally, we use multi-metadata embedding in a two-stream Transformer to integrate Chinese character features with the radical-level embedding. With the structural characteristics of Chinese characters, MECT can better capture the semantic information of Chinese characters for NER. The experimental results obtained on several well-known benchmarking datasets demonstrate the merits and superiority of the proposed MECT method. 1 1 Introduction Named Entity Recognition (NER) plays an essential role in structuring of unstructured text. It is a sequence tagging task that extracts named entities from unstructured text. Common categories of NER include names of people, places, organizations, time, quantity, currency, and some proper nouns. NER is the basis for many Natural Language Processing (NLP) tasks such as event extraction (Chen et al., 2015), question answering (Diefenbach et al., 2018), information reCorresponding author. trieval (Khalid et al., 2008), knowledge graph construction (Riedel et al., 2013), etc. Compared with English, there is no space between Chinese characters as word delimiters. Chinese word segmentation is mostly distinguished by readers through the semantic information of sentences, posing many difficulties to Chinese NER (Duan and Zheng, 2011; Ma et al., 2020). Besides, the task also has many other challenges, such as complex combinations, entity nesting, and indefinite length (Dong et al., 2016). In English, different words may have the same root or affix that better represents the word's semantics. For example, physiology, psychology, sociology, technology and zoology contain the same suf-fix, -logy', which helps identify the entity of a subject name. Besides, according to the information of English words, root or affixes often determine general meanings (Yadav et al., 2018). The root, such as ophthalmo-' ( ophthalmology ), esophage-' ( esophagus ) and epithelio-' ( epithelium ), can help human or machine to better recognize professional nouns in medicine. Therefore, even the state-of-the-art methods, such as BERT (Devlin et al., 2019) and GPT (Radford et al., 2018), trained on large-scale datasets, adopt this delicate word segmentation method for performance boost. For Chinese characters, there is also a structure Radicals Denotation Examples  ( bird ) birds  ( chicken ),  ( duck ),  ( goose ),  ( eagle )  ( grass ) herbaceous plants  ( flower ),  ( grass ),  ( vegetable ),  ( tea )  ( meat ) body parts  ( kidney ),  ( foot ),  ( leg ),  ( brain ) Table 2: Some examples of Chinese radicals, including   '( bird ),   '( grass ) and   '( meat ). similar to the root and affixes in English. According to the examples in Table 1, we can see that the structure of Chinese characters has different decomposition methods, including the Chinese radical (CR), head and tail (HT) and structural components (SC). Chinese characters have evolved from hieroglyphs since ancient times, and their structure often reflects more information about them. There are some examples in Table 2. The glyph structure can enrich the semantics of Chinese characters and improve the performance of NER. For example, the Bi-LSTM-CRF method (Dong et al., 2016) firstly obtains character-level embedding through the disassembly of Chinese character structure to improve the performance of NER. However, LSTM is based on time series modeling, and the input of each cell depends on the output of the previous cell. So the LSTM-based model is relatively complicated and the parallel ability is limited. To address the aforementioned issues, we take the advantages of Flat-Lattice Transformer (FLAT) (Li et al., 2020) in efficient parallel computing and excellent lexicon learning, and introduce the radical stream as an extension on its basis. By combining the radical information, we propose a Multi-metadata Embedding based Cross-Transformer (MECT). MECT has the latticeand radical-streams, which not only possesses FLAT's word boundary and semantic learning ability but also increases the structure information of Chinese character radicals. This is very effective for NER tasks, and has improved the baseline method on different benchmarks. The main contributions of the proposed method include:  The use of multi-metadata feature embedding of Chinese characters in Chinese NER. A novel two-stream model that combines the radicals, characters and words of Chinese characters to improve the performance of the proposed MECT method. The proposed method is evaluated on several well-known Chinese NER benchmarking datasets, demonstrating the merits and superiority of the proposed approach over the state-of-the-art methods. The key of the proposed MECT method is to use the radical information of Chinese characters to enhance the Chinese NER model. So we focus on the mainstream information enhancement methods in the literature. There are two main types of Chinese NER enhancement methods, including lexical information fusion and glyph-structural information fusion. Lexical Enhancement In Chinese NER, many recent studies use word matching methods to enhance character-based models. A typical method is the Lattice-LSTM model (Zhang and Yang, 2018) that improves the NER performance by encoding and matching words in the lexicon. Recently, some lexical enhancement methods were proposed using CNN models, such as LR-CNN (Gui et al., 2019a), CAN-NER (Zhu and Wang, 2019). Graph networks have also been used with lexical enhancement. The typical one is LGN (Gui et al., 2019b). Besides, there are Transformer-based lexical enhancement methods, such as PLT (Xue et al., 2019) and FLAT. And SoftLexicon (Ma et al., 2020) introduces lexical information through label and probability methods at the character representation layer. Glyph-structural Enhancement Some studies also use the glyph structure information in Chinese NER. For example, Dong et al. (2016) were the first to study the application of radical-level information in Chinese NER. They used Bi-LSTM to extract radical-level embedding and then concatenated it with the embedding of characters as the final input. The radical information used in Bi-LSTM is structural components (SC) as shown in Table 1, which achieved state-of-the-art performance on the MSRA dataset. The Glyce (Meng et al., 2019) model used Chinese character images to extract features such as strokes and structure of Chinese characters, achieving promising perfor-TransformerEncoder  South 1 1  Capital 2 2  City 3 3  Long 4 4  River 5 5  Big 6 6  Bridge 7 7  Nanjing 1 2  NanjingCity 1 3  Mayor 3 4  YangtzeRiver 4 5  YangtzeRiver Bridge 4 7  Bridge 6 7 MASKB-LOC I-LOC I-LOC B-LOC I-LOC I-LOC I-LOC Figure 1: The input and output of FLAT. mance in Chinese NER. Some other methods (Xu et al., 2019; Song et al., 2020) also proposed to use radical information and Tencent's pre-trained embedding 2 to improve the performance. In these works, the structural components of Chinese characters have been proven to be able to enrich the semantics of the characters, resulting in better NER performance. The proposed method is based on the Flat-Lattice Transformer (FLAT) model. Thus, we first briefly introduce FLAT that improves the encoder structure of Transformer by adding word lattice information, including semantic and position boundary information. These word lattices are obtained through dictionary matching. Figure 1 shows the input and output of FLAT. It uses the relative position encoding transformed by head and tail position to fit the word's boundary information. The relative position encoding, R ij , is calculated as follows: R ij = ReLU ( W r ( p h i  h j  p h i  t j  p t i  h j  p t i  t j )) , (1) where W r is a learnable parameter, h i and t i represent the head position and tail position of the i-th character,  denotes the concatenation operation, and p span is obtained as in Vaswani et al. (2017): p (2 k ) span = sin( span 10000 2 k/d model ) , (2) p (2 k +1) span = cos( span 10000 2 k/d model ) , (3) where p span corresponds to p in Eq. (1), and span denotes h i  h j , h i  t j , t i  h j and t i  t j . Then the scaled dot-product attention is obtained by: Att ( A , V ) = softmax ( A ) V , (4) A ij = ( Q i + u ) (cid:62) K j + ( Q i + v ) (cid:62) R  ij , (5) [ Q , K , V ] = E x [ W q , W k , W v ] , (6) 2 https://ai.tencent.com/ailab/nlp/en/ embedding.html where R  ij = R ij  WR . To better integrate the information of Chinese character components, we use Chinese character structure as another metadata and design a two-stream form of multi-metadata embedding network. The architecture of the proposed network is shown in Figure 2a. The proposed method is based on the encoder structure of Transformer and the FLAT method, in which we integrate the meaning and boundary information of Chinese words. The proposed two-stream model uses a Cross-Transformer module similar to the self-attention structure to fuse the information of Chinese character components. In our method, we also use the multi-modal collaborative attention method that is widely used in vision-language tasks (Lu et al., 2019). The difference is that we add a randomly initialized attention matrix to calculate the attention bias for the two types of metadata embedding. Chinese characters are based on pictographs, and their meanings are expressed in the shape of objects. In this case, the structure of Chinese characters has certain useful information for NER. For example, the radicals such as   ' ( grass ) and   ' ( wood ) generally represent plants, enhancing Chinese medicine entity recognition. For another example,   ' ( body ) represents human body parts or organs, and   ' ( disease ) represents diseases, which benefits Chinese NER for the medical field. Besides, the Chinese have their own culture and belief in naming. Radicals   ' ( metal ),   ' ( wood ),   ' ( water ),   ' ( fire ), and   ' ( earth ) represented by the Wu-Xing (Five Elements) theory are often used as names of people or companies. But   ' ( rust ),   ' ( kill ),   ' ( dirt ),   ' ( disaster ) and   ' ( fall ) are usually not used as names, even if they contain some elements of the Wu-Xing theory. It is because the other radical components also determine the semantics of Chinese characters. Radicals that generally appear negative or conflict with Chinese cultural beliefs are usually not used for naming. Therefore, we choose the more informative Structural Components (SC) in Table 1 as radical-level features of Chinese characters and use Convolutional Neural Network (CNN) to extract character                          <pad><pad> <pad><pad><pad> <pad>              LatticeEmbedding Radical-levelEmbedding Split characters into SC South Capital City Long River Big Bridge CNNforRadical-levelEmbedding B-LOC I-LOC I-LOC B-LOC I-LOC I-LOC I-LOCMASKS o u t h C a p i t a l C i t y L o n g R i v e r B i g N a n j i n g C i t y M a y o r Y a n g t z e R i v e r Y a n g t z e R i v e r B r i d g e B r i d g e C o n s t r u c t l a tt i c e B r i d g e N a n j i n g S o u t h C a p i t a l C i t y L o n g R i v e r B i g B r i d g e features. The structure diagram of the CNN network is shown in Figure 3. We first disassemble the Chinese characters into SC and then input the radicals into CNN. Last, we use the max-pooling and fully connected layers to get the feature embedding of Chinese characters at the radical-level. After radical feature extraction, we propose a Cross-Transformer network to obtain the supplementary semantic information of the structure of Chinese characters. It also uses contextual and lexical information to enrich the semantics of Chinese characters. The Cross-Transformer network is illustrated in Figure 2b. We use two Transformer encoders to cross the lattice and radical information of Chinese characters, which is different from the self-attention method in Transformer. where EL and ER are lattice embedding and radical-level embedding, I is the identity matrix, and each W is a learnable parameter. Then we use the relative position encoding in FLAT to represent the boundary information of a word and calculate the attention score in our Cross-Transformer: Att L ( AR , VL ) = Softmax ( AR ) VL , (8) Att R ( AL , VR ) = Softmax ( AL ) VR , (9) AL ( R ) ,ij = ( QL ( R ) ,i + u L ( R ) ) (cid:62) ER ( L ) ,j + ( QL ( R ) ,i + v L ( R ) ) (cid:62) R  L ( R ) ,ij , (10) where u and v are learnable parameters for attention bias in Eq. (10), AL is the lattice attention score, and AR denotes the radical attention score. And R  ij = R ij  WR . WR are learnable parameters. The relative position encoding, R ij , is calculated as follows: R ij = ReLU ( W r ( p h i  h j  p t i  t j )) . We empirically found that the use of random attention in Cross-Transformer can improve the performance of the proposed method. This may be due to the requirement of attention bias in lattice and radical feature embedding, which can better adapt to the scores of two subspaces. Random attention is a randomly initialized parameter matrix B max len  max len that is added to the previous attention score to obtain a total attention score: V  L = Softmax ( AR + B ) VL , (12) V  R = Softmax ( AL + B ) VR . To reduce information loss, we directly concatenate the lattice and radical features and input them into a fully connected layer for information fusion: Fusion ( V  L , V  R ) = ( V  R  V  L ) W o + b , (14) where  denotes the concatenation operation, W o and b are learnable parameters. After the fusion step, we mask the word part and pass the fused feature to a Conditional Random Field (CRF) (Lafferty et al., 2001) module. In this section, we evaluate the proposed MECT method on four datasets. To make the experimental results more reasonable, we also set up two additional working methods for assessing the performance of radicals in a two-stream model. We use the span method to calculate F1-score (F1), precision (P), and recall (R) as the evaluation metrics. We use four mainstream Chinese NER benchmarking datasets: Weibo (Peng and Dredze, 2015; He and Sun, 2016), Resume (Zhang and Yang, 2018), MSRA (Levow, 2006), and Ontonotes 4.0 (Weischedel and Consortium, 2013). The corpus of MSRA and Ontonotes 4.0 comes from news, the corpus of Weibo comes from social media, and the corpus of Resume comes from the resume data in Sina Finance. Table 3 shows the statistical information of these datasets. Among them, the Weibo dataset has four types of entities, including PER, ORG, LOC, and GPE. Resume has eight types of entities, including CONT, EDU, LOC, PER, ORG, PRO, RACE, and TITLE. OntoNotes 4.0 has four types of entities: PER, ORG, LOC, and GPE. The MSRA dataset contains three types of entities, i.e. , ORG, PER, and LOC. We use the state of the art method, FLAT, as the baseline model. FLAT is a Chinese NER model based on Transformer and combined with lattice. Besides, we also compared the proposed method with both classic and innovative Chinese NER models. We use the more informative SC' as the radical feature, which comes from the online Xinhua Datasets Types Train Dev Test Weibo SentencesEntities 1.35k1.89k 0.27k0.39k 0.27k0.42k Resume SentencesEntities 3.8k1.34k 0.46k0.16k 0.48k0.15k OntoNotes SentencesEntities 15.7k13.4k 4.3k6.95k 4.3k7.7k MSRA SentencesEntities 46.4k74.8k --4.4k6.2k Table 3: Statistics of the benchmarking datasets. Dictionary 3 . The pre-trained embedding of characters and words are the same as FLAT. For hyper-parameters, we used 30 1-D convolution kernels with the size of 3 for CNN. We used the SMAC (Hutter et al., 2011) algorithm to search for the optimal hyper-parameters. Besides, we set a different learning rate for the training of the radical-level embedding with CNN. Readers can refer to the appendix for our hyper-parameter settings. In this section, we evaluate and analyze the proposed MECT method with a comparison to both the classic and state of the art methods. The experimental results are reported in Tables 4  7 4 . Each table is divided into four blocks. The first block includes classical Chinese NER methods. The second one reports the results obtained by state of the art approaches published recently. The third and 3 http://tool.httpcn.com/Zi/ . 4 In Tables 4  7,   ' denotes the use of external labeled data for semi-supervised learning and   ' denotes the use of discrete features. MECT method as well as the baseline models. Weibo: Table 4 shows the results obtained on Weibo in terms of the F1 scores of named entities (NE), nominal entities (NM), and both (Over-all). From the results, we can observe that MECT achieves the state-of-the-art performance. Compared with the baseline method, MECT improves 2.98% in terms of the F1 metric. For the NE metric, the proposed method achieves 61.91%, beating all the other approaches. Resume: The results obtained on the Resume dataset are reported in Table 5. The first block shows Zhang and Yang (2018) comparative results on the character-level and word-level models. We can observe that the performance of incorporating word features into the character-level model is better than other models. Additionally, MECT combines lexical and radical features, and the F1 score is higher than the other models and the baseline method. Ontonotes 4.0: Table 6 shows the results obtained on Ontonotes 4.0. The symbol ' indicates gold segmentation, and the symbol ' denotes automated segmentation. Other models have no segmentation and use lexical matching. Compared to the baseline method, the F1 score of MECT is increased by 0.47%. MECT also achieves a high recall rate, keeping the precision rate and recall rate relatively stable. MSRA: Table 7 shows the experimental results obtained on MSRA. In the first block, the result proposed by Dong et al. (2016) is the first method using radical information in Chinese NER. From the table, we can observe that the overall performance of MECT is higher than the existing SOTA methods. Similarly, our recall rate achieves a higher performance so that the final F1 has a certain performance boosting. With BERT: Besides the single-model evaluation on the four datasets, we also evaluated the proposed method when combining with the SOTA method, BERT. The BERT model is the same as FLAT using the BERT-wwm' released by Cui et al. (2020). The results are shown in the fourth block of each table. The results of BERT are taken from the FLAT paper. We can find that MECT further improves the performance of BERT significantly. There are two sub-modules in the proposed Cross-Transformer method: lattice and radical attentions. Figure 4 includes two heatmaps for the normalization of the attention scores of the two modules. From the two figures, we can see that lattice attention pays more attention to the relationship between words and characters so that the model can obtain the position information and boundary information of words. Radical attention focuses on global information and corrects the semantic information of Models P R F1 Chen et al. (2006) 91.22 81.71 86.20 Zhang et al. (2006)  92.20 90.18 91.18 Zhou et al. (2013) 91.86 88.75 90.28 Lu et al. (2016) -87.94 Dong et al. (2016) 91.28 90.62 90.95 Lattice-LSTM 93.57 92.79 93.18 CAN-NER 93.53 92.42 92.97 LR-CNN 94.50 92.93 93.71 LGN 94.19 92.73 93.46 PLT 94.25 92.30 93.26 SoftLexicon (LSTM) 94.63 92.70 93.66 + bichar 94.73 93.40 94.06 Baseline -94.12 MECT 94.55 94.09 94.32 BERT -94.95 BERT + MECT -96.24 Table 7: Results obtained on MSRA (%). each character through radical features. Therefore, lattice and radical attentions provide complementary information for the performance-boosting of the proposed MECT method in Chinese NER. We visualized the radical-level embedding obtained by the CNN network and found that the cosine distance of Chinese characters with the same radical or similar structure is smaller. For example, Figure 5 shows part of the Chinese character embedding trained on the Resume dataset. The highlighted dots represent Chinese characters that are close to the character   '. We can see that they have the same radicals or similar structure. It can enhance the semantic information of Chinese characters to a certain extent. We also examined the inference results of MECT and FLAT on Ontonotes 4.0 and found many exciting results. For example, some words with a Figure 5: Embedding visualization of the characters related to   ' in two-dimensional space. percentage like          (43.2%)' is incorrectly labelled as PER in the training dataset, which causes FLAT to mark the percentage of words with PER on the test dataset, while MECT avoids this situation. There are also some words such as    ' and    ' that appear in the lexicon, which was mistakenly identified as valid words by FLAT, leading to recognition errors. Our MECT addresses these issues by paying global attention to the radical information. Besides, in FLAT, some numbers and letters are incorrectly marked as PER, ORG, or others. We compared the PER label accuracy of FLAT and MECT on the test dataset. FLAT achieves 81.6%, and MECT reaches 86.96%, which is a very significant improvement. We use the same FLAT method to evaluate the parallel and non-parallel inference speed of MECT on an NVIDIA GeForce RTX 2080Ti card, using batch size = 16 and batch size = 1. We use the non-parallel version of FLAT as the standard and calculate the other models' relative inference speed. The results are shown in Figure 6. According to the figure, even if MECT adds a Transformer encoder to FLAT, the speed is only reduced by 0.15 in terms of the parallel inference speed. Our model's speed is considerable relative to LSTM, CNN, and some graph-based network models. Because Transformer can make full use of the GPU's parallel computing power, the speed of MECT does not drop too much, but it is still faster than other models. The model's parameter is between 2 and 4 million, determined by the max sentence length in the dataset and the d model size in the model. To validate the effectiveness of the main components of the proposed method, we set up two experiments in Figure 7. In Experiment A, we only use a single-stream model with a modified self-attention, which is similar to the original FLAT model. The difference is that we use a randomly initialized attention matrix (Random Attention) for the attention calculation. We combine lattice embedding and radical-level embedding as the input of the model. The purpose is to verify the performance of the two-stream model relative to the single-stream model. In Experiment B, we do not exchange the query's feature vector. We replace the cross-attention with two sets of modified self-attention and follow the two modules' output with the same fusion method as MECT. The purpose of experiment B is to verify the effectiveness of MECT relative to the two-stream model without crossover. Besides, we evaluate the proposed MECT method by removing the random attention module. Table 8 shows the ablation study results. 1) By comparing the results of Experiment A with the results of Experiment B and MECT, we can find that the two-stream model works better. The use of lattice-level and radical-level features as the two streams of the model helps the model to better understand and extract the semantic features of Chinese characters. 2) Based on the results of Experiment B and MECT, we can see that by exchanging the two query feature vectors, the model can extract features more effectively at the lattice and radical levels. They have different attention mechanisms to obtain contextual information, resulting in global and local attention interaction. This provides better information extraction capabilities for the proposed method in a complementary way. 3) Last, the performance of MECT drops on all the datasets by Lattice Embedding Radical-levelEmbedding Adapt Self Attention removing the random attention module (the last row). This indicates that, as an attention bias, random attention can eliminate the differences caused by different embeddings, thereby improving the model's performance further. This paper presented a novel two-stream network, namely MECT, for Chinese NER. The proposed method uses multi-metadata embedding that fuses the information of radicals, characters and words through a Cross-Transformer network. Additionally, random attention was used for further performance boost. Experimental results obtained on four benchmarks demonstrate that the radical information of Chinese characters can effectively improve the performance for Chinese NER. The proposed MECT method with the radical stream increases the complexity of a model. In the future, we will consider how to integrate the characters, words and radical information of Chinese characters with a more efficient way in two-stream or multi-stream networks to improve the performance of Chinese NER and extend it to other NLP tasks. This work was supported in part by the National Key Research and Development Program of China (2017YFC1601800), the National Natural Science Foundation of China (61876072, 61902153) and the Six Talent Peaks Project of Jiangsu Province (XYDXX-012). We also thank Xiaotong Xiang and Jun Quan for their help on editing the manuscript.
abstain abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain method objective abstain abstain abstain objective objective other method other other other other other other other other other other other other other other other other other objective other other other other other other method abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain objective other other other