Lexicon information and pre-trained models, such as BERT, have been combined to explore Chinese sequence labelling tasks due to their respective strength. However, existing methods solely fuse lexicon features via a shallow and random initialized sequence layer and do not integrate them into the bottom layers of BERT. In this paper, we propose Lexicon Enhanced BERT (LEBERT) for Chinese sequence labelling, which integrates external lexicon knowledge into BERT layers directly by a Lexicon Adapter layer. Compared with the existing methods, our model facilitates deep lexicon knowledge fusion at the lower layers of BERT. Experiments on ten Chinese datasets of three tasks including Named Entity Recognition, Word Segmentation, and Part-of-Speech Tagging, show that LEBERT achieves the state-of-the-art results. Sequence labeling is a classic task in natural language processing (NLP), which is to assign a label to each unit in a sequence (Jurafsky and Martin, 2009). Many important language processing tasks can be converted into this problem, such as part-of-speech (POS) tagging, named entity recognition (NER) and text chunking. The current state-of-the-art results for sequence labelling have been achieved by neural network approaches (Lample et al., 2016; Ma and Hovy, 2016; Chiu and Nichols, 2016; Gui et al., 2017). Chinese sequence labelling is more challenging due to the lack of explicit word boundaries in Chinese sentences. One way of performing Chinese sequence labelling is to perform Chinese word segmentation (CWS) first, before applying word sequence labelling (Sun and Uszkoreit, 2012; Yang et al., 2016). However, it can suffer from the segmentation errors propagated from the CWS system !"#$%&'"()" !"#$%&'"()" ! (Zhang and Yang, 2018; Liu et al., 2019). Therefore, some approaches (Cao et al., 2018; Shen et al., 2016) perform Chinese sequence labelling directly at the character level, which have been empirically proven to be more effective (Ng and Low, 2004; Liu et al., 2010; Zhang and Yang, 2018). There are two lines of recent work enhancing character-based neural Chinese sequence labelling. The first considers integrating word information into a character-based sequence encoder, so that word features can be explicitly modelled (Zhang and Yang, 2018; Yang et al., 2019; Liu et al., 2019; Ding et al., 2019; Higashiyama et al., 2019). These methods can be treated as designing different variants to neural architectures for integrating discrete structured knowledge. The second considers the integration of large-scale pre-trained contextualized embeddings, such as BERT (Devlin et al., 2019), which has been shown to capture implicit word-level syntactic and semantic knowledge (Goldberg, 2019; Hewitt and Manning, 2019). The two lines of work are complementary to each other due to the different nature of discrete and neural representations. Recent work considers the combination of lexicon features and BERT for Chinese NER (Ma et al., 2020; Li et al., 2020), Chinese Word Segmentation (Gan and Zhang, 2020) and Chinese POS tagging (Tian et al., 2020b). The main idea is to integrate contextual representations from BERT and lexicon features into a neural sequence labelling model (shown in Figure 1 (a)). However, these approaches do not fully exploit the representation power of BERT, because the external features are not integrated into the bottom level. Inspired by the work about BERT Adapter (Houlsby et al., 2019; Bapna and Firat, 2019; Wang et al., 2020), we propose Lexicon Enhanced BERT (LEBERT) to integrate lexicon information between Transformer layers of BERT directly. Specifically, a Chinese sentence is converted into a char-words pair sequence by matching the sentence with an existing lexicon. A lexicon adapter is designed to dynamically extract the most relevant matched words for each character using a char-to-word bilinear attention mechanism. The lexicon adapter is applied between adjacent transformers in BERT (shown in Figure 1 (b)) so that lexicon features and BERT representation interact sufficiently through the multi-layer encoder within BERT. We fine-tune both the BERT and lexicon adapter during training to make full use of word information, which is considerably different from the BERT Adapter (it fixes BERT parameters). We investigate the effectiveness of LEBERT on three Chinese sequence labelling tasks 1 , including Chinese NER, Chinese Word Segmentation 2 , and Chinese POS tagging. Experimental results on ten benchmark datasets illustrate the effectiveness of our model, where state-of-the-art performance is achieved for each task on all datasets. In addition, we provide comprehensive comparisons and detailed analyses, which empirically confirm that bottom-level feature integration contributes to span boundary detection and span type determination. Lexicon-based . Lexicon-based models aim to enhance character-based models with lexicon information. Zhang and Yang (2018) introduced a lat-1 https://github.com/liuwei1206/LEBERT 2 We follow the mainstream methods and regard Chinese Word Segmentation as a sequence labelling problem. tice LSTM to encode both characters and words for Chinese NER. It is further improved by following efforts in terms of training efficiency (Gui et al., 2019a; Ma et al., 2020), model degradation (Liu et al., 2019), graph structure (Gui et al., 2019b; Ding et al., 2019), and removing the dependency of the lexicon (Zhu and Wang, 2019). Lexicon information has also been shown helpful for Chinese Word Segmentation (CWS) and Part-of-speech (POS) tagging. Yang et al. (2019) applied a lattice LSTM for CWS, showing good performance. Zhao et al. (2020) improved the results of CWS with a lexicon-enhanced adaptive attention. Tian et al. (2020b) enhanced the character-based Chinese POS tagging model with a multi-channel attention of N-grams. Pre-trained Model-based . Transformer-based pre-trained models, such as BERT (Devlin et al., 2019), have shown excellent performance for Chinese sequence labelling. Yang (2019) simply added a softmax on BERT, achieving state-of-the-art performance on CWS. Meng et al. (2019); Hu and Verberne (2020) showed that models using the character features from BERT outperforms the static embedding-based approaches by a large margin for Chinese NER and Chinese POS tagging. Hybrid Model . Recent work tries to integrate the lexicon and pre-trained models by utilizing their respective strengths. Ma et al. (2020) concatenated separate features, BERT representation and lexicon information, and input them into a shallow fusion layer (LSTM) for Chinese NER. Li et al. (2020) proposed a shallow Flat-Lattice Transformer to handle the character-word graph, in which the fusion is still at model-level. Similarly, character N-gram features and BERT vectors are concatenated for joint training CWS and POS tagging (Tian et al., 2020b). Our method is in line with the above approaches trying to combine lexicon information and BERT. The difference is that we integrate lexicon into the bottom level, allowing in-depth knowledge interaction within BERT. There is also work employing lexicon to guide pre-training. ERNIE (Sun et al., 2019a,b) exploited entity-level and word-level masking to integrate knowledge into BERT in an implicit way. Jia et al. (2020) proposed Entity Enhanced BERT, further pre-training BERT using a domain-specific corpus and entity set with a carefully designed character-entity Transformer. ZEN (Diao et al., 2019) enhanced Chinese BERT with a multi-!"#$%&'&()&%*&%+& ! ! ! " ! 748,415&33$%69.:&-Figure 2: The architecture of Lexicon Enhanced BERT, in which lexicon features are integrated between k -th and ( k +1) -th Transformer Layer using Lexicon Adapter. Where c i denote the i -th Chinese character in the sentence, and ws i denotes matched words assigned to character c i . layered N-gram encoder but is limited by the small size of the N-gram vocabulary. Compared to the above pre-training methods, our model integrates lexicon information into BERT using an adapter, which is more efficient and requires no raw texts or entity set. BERT Adapter . BERT Adapter (Houlsby et al., 2019) aims to learn task-specific parameters for the downstream tasks. Specifically, they add adapters between layers of a pre-trained model and tune only the parameters in the added adapters for a certain task. Bapna and Firat (2019) injected task-specific adapter layers into pre-trained models for neural machine translation. MAD-X (Pfeiffer et al., 2020) is an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks. Wang et al. (2020) proposed K-ADAPTER to infuse knowledge into pre-trained models with further pre-training. Similar to them, we use a lexicon adapter to integrate lexicon information into BERT. The main difference is that our goal is to better fuse lexicon and BERT at the bottom-level rather than efficient training. To achieve it, we fine-tune the original parameters of BERT instead of fixing them, since directly injecting lexicon features into BERT will affect the performance due to the difference between that two information. The main architecture of the proposed Lexicon Enhanced BERT is shown in Figure 2. Compared to BERT, LEBERT has two main differences. First, LEBERT takes both character and lexicon features as the input given that the Chinese sentence is converted to a character-words pair sequence. Second, a lexicon adapter is attached between Transformer layers, allowing lexicon knowledge integrated into BERT effectively. In this section we describe: 1) Char-words Pair Sequence (Section 3.1), which incorporates words into a character sequence naturally; 2) Lexicon Adapter (Section 3.2), by injecting external lexicon features into BERT; 3) Lexicon Enhanced BERT (Section 3.3), by applying the Lexicon Adapter to BERT. A Chinese sentence is usually represented as a character sequence, containing character-level features solely. To make use of lexicon information, we extend the character sequence to a character-words pair sequence. Given a Chinese Lexicon D and a Chinese sentence with n characters s c = { c 1 , c 2 , ..., c n } , we find out all the potential words inside the sentence by matching the character sequence with D . Specifically, we first build a Trie based on the D , then traverse all the character subsequences of the sentence and match them with the Trie to obtain all potential words. Taking the truncated sentence      (American People) for example, we can find out four different words, namely    (Amer-ica),     (American),    (Compatriot),    (People). Subsequently, for each matched word, we assign it to the characters it contains. As shown in Figure 3, the matched word    (Amer-ica) is assigned to the character    and    since they form that word. Finally, we pair each character with assigned words and convert a Chinese sentence into a character-words pair sequence, i.e. s cw = { ( c 1 , ws 1 ) , ( c 2 , ws 2 ) , ..., ( c n , ws n ) } , where c i denotes the i -th character in the sentence and ws i denotes matched words assigned to c i . Each position in the sentence consists of two types of information, namely character-level and word-level features. In line with the existing hybrid models, our goal is to combine the lexicon feature with BERT. Specifically, inspired by the recent works about BERT adapter (Houlsby et al., 2019; Wang et al., 2020), we propose a novel Lexicon Adapter (LA) shown in Figure 4, which can directly inject lexicon information into BERT. A Lexicon Adapter receives two inputs, a character and the paired words. For the i -th position in a char-words pair sequence, the input is denoted as ( h ci , x wsi ) , where h ci is a character vector, the output of a certain transformer layer in BERT, and x wsi = { x wi 1 , x wi 2 , ..., x wim } is a set of word embed-dings. The j -th word in x wsi is represented as following: x wij = e w ( w ij ) (1) where e w is a pre-trained word embedding lookup table and w ij is the j -th word in ws i . To align those two different representations, we apply a non-linear transformation for the word vectors: v wij = W 2 (tanh( W 1 x wij + b 1 )) + b 2 (2) where W 1 is a d c -byd w matrix, W 2 is a d c -byd c matrix, and b 1 and b 2 are scaler bias. d w and d c denote the dimension of word embedding and the hidden size of BERT respectively. As Figure 3 shows, each character is paired with multiple words. However, the contribution to each task varies from word to word. For example, as for Chinese POS tagging, words    (America) and    (People) are superior to     (Amer-ican) and    (Compatriot), since they are ground-truth segmentations of the sentence. To pick out the most relevant words from all matched words, we introduce a character-to-word attention mechanism. Specifically, we denote all v wij assigned to i -th character as V i = ( v wi 1 , ..., v wim ) , which has the size m -byd c and m is the total number of the assigned ! where W attn is the weight matrix of bilinear attention. Consequently, we can get the weighted sum of all words by: z wi = m (cid:88) j =1 a ij v wij (4) Finally, the weighted lexicon information is injected into the character vector by:  h i = h ci + z wi (5) It is followed by a dropout layer and layer normalization. Lexicon Enhanced BERT (LEBERT) is a combination of Lexicon Adapter (LA) and BERT, in which LA is applied to a certain layer of BERT shown in Figure 2. Concretely, LA is attached between certain transformers within BERT, thereby injecting external lexicon knowledge into BERT. Given a Chinese sentence with n characters s c = { c 1 , c 2 , ..., c n } , we build the corresponding character-words pair sequence s cw = { ( c 1 , ws 1 ) , ( c 2 , ws 2 ) , ..., ( c n , ws n ) } as described in Section 3.1. The characters { c 1 , c 2 , ..., c n } are first input into Input Embedder which outputs E = { e 1 , e 2 , ..., e n } by adding token, segment and position embedding. Then we input E into Transformer encoders and each Transformer layer acts as following: G = LN( H l  1 + MHAttn( H l  1 )) H l = LN( G + FFN( G )) (6) where H l = { h l 1 , h l 2 , ..., h ln } denotes the output of the l -th layer and H 0 = E ; LN is layer normalization; MHAttn is the multi-head attention mechanism; FFN is a two-layer feed-forward network with ReLU as hidden activation function. To inject the lexicon information between the k -th and ( k + 1) -th Transformer, we first get the output H k = { h k 1 , h k 2 , ..., h kn } after k successive Transformer layers. Then, each pair ( h ki , x wsi ) are passed through the Lexicon Adapter which transforms the i th pair into  h ki :  h ki = LA( h ki , x wsi ) (7) Since there are L = 12 Transformer layers in the BERT, we input (cid:101) H k = {  h k 1 ,  h k 2 , ...,  h kn } to the remaining ( L  k ) Transformers. At the end, we get the output of L -th Transformer HL for the sequence labelling task. Considering the dependency between successive labels, we use a CRF layer to make sequence labelling. Given the hidden outputs of the last layer HL = { h L 1 , h L 2 , ..., h Ln } , we first calculate scores P as: O = W o HL + b o (8) For a label sequence y = { y 1 , y 2 , ..., y n } , we define its probability to be: p ( y | s ) = exp( (cid:80) i ( O i,y i + T y i  1 ,y i )) (cid:80)  y exp( (cid:80) i ( O i,  y i + T  y i  1 ,  y i )) (9) where T is the transition score matrix and y denotes all position tag sequences. Given N labelled data { s j , y j }| Nj =1 , we train the model by minimize the sentence-level negative log-likelihood loss as: L =  (cid:88) j log( p ( y | s )) (10) While decoding, we find out the label sequence obtaining the highest score using the Viterbi algorithm. We carry out an extensive set of experiments to investigate the effectiveness of LEBERT. In addition, we aim to empirically compare model-level and BERT-level fusion in the same setting. Standard F1-score (F1) are used as evaluation metrics. We evaluate our method on ten datasets of three different sequence labelling tasks, including Chinese NER, Chinese Word Segmentation and Chinese POS tagging. The statistics of the datasets is shown in Table 1. Chinese NER . We conduct experiments on four benchmark datasets, including Weibo NER (Peng and Dredze, 2015, 2016), OntoNotes (Weischedel et al., 2011), Resume NER (Zhang and Yang, 2018) and MSRA (Levow, 2006). Weibo NER is a social media domain dataset, which is drawn from Sina Weibo; while OntoNotes and MSRA dataset are in the news domain. Resume NER dataset consists of resume of senior executives, which is annotated by Zhang and Yang (2018). Chinese Word Segmentation . For Chinese word segmentation, we employ three benchmark datasets in our experiments, namely PKU, MSR and CTB6, where the former two are from SIGHAN 2005 Bakeoff (Emerson, 2005) and the last one is from Xue et al. (2005). For MSR and PKU, we follow their official training/test data split. For CTB6, we use the same split as that stated in Yang and Xue (2012); Higashiyama et al. (2019). Chinese POS Tagging . For POS-tagging, three Chinese benchmark datasets are used, including CTB5 and CTB6 from the Penn Chinese TreeBank (Xue et al., 2005) and the Chinese GSD Treebank of Universal Dependencies(UD) (Nivre et al., 2016). The CTB datasets are in simplified Chinese while the UD datasets is in traditional Chinese. Following Shao et al. (2017), we first convert the UD dataset into simplified Chinese before the POS-tagging experiments 3 . Besides, UD has both universal and language-specific POS tags, we follow previous works (Shao et al., 2017; Tian et al., 2020a), refering the corpus with two tagsets as UD1 and UD2, respectively. We use the official splits of train/dev/test in our experiments. Our model is constructed based on BERTBASE (Devlin et al., 2019), with 12 layers of transformer, and is initialized using the Chinese-BERT checkpoint from huggingface 4 . We use the 200-dimension pre-trained word embedding from Song et al. (2018), which is trained on texts of news and webpages using a directional skip-gram model. The lexicon D used in this paper is the vocab of the pre-trained word embedding. We apply the Lexicon Adapter between the 1 -st and 2 -nd Transformer in BERT and fine-tune both BERT and pre-trained word embedding during training. Hyperparameters . We use the Adam optimizer with an initial learning rate of 1e-5 for original parameters of BERT, and 1e-4 for other parameters introduced by LEBERT, and a maximum epoch number of 20 for training on all datasets. The max length of the sequence is set to 256, and the training batch size is 20 for MSRA NER and 4 for other datasets. Baselines . To evaluate the effectiveness of the proposed LEBERT, we compare it with the following approaches in the experiments. BERT . Directly fine-tuning a pre-trained Chinese BERT on Chinese sequence labelling task. BERT+Word . A strong model-level fusion baseline method, which inputs the concatenation of BERT vector and bilinear attention weighted word vector, and uses LSTM 5 and CRF as fu-3 The conversion tool we used is OpenCC. sion layer and inference layer respectively. ERNIE (Sun et al., 2019a). An extension of BERT using a entity-level mask to guide pretraining. ZEN . Diao et al. (2019) explicitly integrate N-gram information into BERT through an extra multi-layers of N-gram Transformer encoder and pre-training. Chinese NER . Table 2 shows the experimental results on Chinese NER datasets 6 . The first four rows (Zhang and Yang, 2018; Zhu and Wang, 2019; Liu et al., 2019; Ding et al., 2019) in the first block show the performance of lexicon enhanced character-based Chinese NER models, and the last two rows (Ma et al., 2020; Li et al., 2020) in the same block are the state-of-the-art models using shallow fusion layer to integrate lexicon information and BERT. The hybrid models, including existing state-of-the-art models, BERT + Word, and the proposed LEBERT, achieve better performance than both lexicon enhanced models and BERT baseline. This demonstrates the effectiveness of combining BERT and lexicon features for Chinese NER. Compared with model-level fusion models ((Ma et al., 2020; Li et al., 2020), and BERT+Word), our BERT-level fusion model, LEBERT, improves in F1 score on all four datasets across different domains, which shows that our approach is more efficient in integrating word and BERT. The results also indicate that our adapter-based method, LEBERT, 6 For fair comparison, in Table 2, we use * denotes training the model with the same pre-trained word embedding as ours;  means the model is also initialized using the Chinese BERT checkpoint from huggingface and evaluated using the seqeval tool. with an extra pre-trained word embedding solely, outperforms those two lexicon guided pre-training models (ERNIE and ZEN). This is likely because implicit integration of lexicon in ERNIE and restricted pre-defined n-gram vocabulary size in ZEN limited the effect. Chinese Word Segmentation . We report the F1 score of our model and the baseline methods on Chinese Word Segmentation in Table 3. Yang et al. (2019) applied a lattice LSTM to integrate word feature to character-based CWS model. Qiu et al. (2020) investigated the benefit of multiple heterogeneous segmentation criteria for single criterion Chinese word segmentation. Tian et al. (2020c) designed a wordhood memory networks to incorporate wordhood information into pre-trained-based CWS model and showed good performance. Compared with those approaches, the models (BERT+Word and LEBERT) that combine lexicon features and BERT perform better. Moreover, our proposed LEBERT outperforms both model-level fusion baseline (BERT+Word) and lexicon guided pretraining models (ERNIE and ZEN), achieving the best results. Chinese POS Tagging . We report the F1 score on four benchmarks of Chinese POS tagging in Table BERT STOA (with BERT) NER Weibo 10.63% 5.31% Ontonote4 10.71% 3.97% MSRA 18.71% 5.28% Resume 16.06% 7.11% CWS PKU 17.60% 11.46% MSR 36.41% 23.84% CTB6 17.88% 12.68% POS CTB5 23.73% 11.46% CTB6 10.07% 6.95% UD1 23.79% 12.25% UD2 19.17% 6.17% Table 5: The relative error reductions over different base models. 4. The state-of-the-art model (Tian et al., 2020a) jointly trains Chinese Word Segmentation and Chinese POS tagging using a two-way attention to incorporate auto-analyzed knowledge, such as POS labels, syntactic constituents, and dependency relations. Similar to BERT+Word baseline, Tian et al. (2020b) integrated character-Ngram features with BERT at model-level using a multi-channel attention. As shown in the Table 4, hybrid models ((Tian et al., 2020b), BERT+Word, LEBERT) that combine words information and BERT outperform BERT baseline, indicating that lexicon features can further improve the performance of BERT. LEBERT achieves the best results among these approaches, which demonstrates the effectiveness of BERT-level fusion. Consistent with results on Chinese NER and CWS, our BERT adapter based approach is superior to lexicon guided pre-training methods (ERNIE and ZEN). Our proposed model has achieved state-of-the-art results across all datasets. To better show the strength of our method, we also summary the relative error reduction over BERT baseline and BERT-based state-of-the-art models in Table 5. The results show that the relative error reductions are significant compared with baseline models. Compared with model-level fusion models, LEBERT directly integrates lexicon features into BERT. We evaluate those two types of models in terms of Span F1, Type Acc, and Sentence Length, choosing the BERT+Word as the model-level fusion baseline due to its good performance across all the datasets. We also compare with a BERT baseline since both LEBERT and BERT+Word are improved based on it. Span F1 & Type Acc . Span F1 means the correctness of the span for an Entity in NER or a word in POS-tagging, while Type Acc denotes the proportion of full-correct predictions to span-correct predictions. Table 6 shows the results of three models on the Ontonotes and UD1 datasets. We can find that both BERT+Word and LEBERT perform better than BERT in terms of Span F1 and Type Acc on the two datasets. The results indicate that lexicon information contributes to span boundary detection and span classification. Specifically, the improvement of Span F1 is larger than Type Acc on Ontonotes, but smaller on UD1. Compared with BERT+Word, LEBERT achieves more improvement, demonstrating the effectiveness of lexicon feature enhanced via BERT-level fusion. Sentence Length . Figure 5 shows the F1-value trend of the baselines and LEBERT on Ontonotes dataset. All the models show a similar performance-length curve, decreasing as the sentence length increase. We speculate that long sentences are more challenging due to complicated semantics. Even lexicon enhanced models may fail to choose the correct words because of the increased number of matched words as the sentence become longer. The F1-score of BERT is relatively low, while BERT+Word achieves better performance due to the usage of lexicon information. Compared with BERT+Word, LEBERT performs better and shows more robustness when sentence length increases, demonstrating the more effective use of lexicon information. Case Study . Table 8 shows examples of Chinese NER and Chinese POS tagging results on Ontonotes and UD1 datasets respectively. In the first example, BERT can not determine the entity boundary, but BERT+Word and LEBERT can segment it correctly. However, the BERT+Word model fails to predict the type of the entity       (Hulunbuir League) while LEBERT makes the correct prediction. This is likely because fusion at lower layer contributes to capturing more complex semantics provided by BERT and lexicon. In the second example, the three models 20< 40 60 80 100 >100 sentence length 0.78 0.80 0.82 0.84 0.86 0.88 F 1 v a l u e BERT BERT+Word LEBERT Figure 5: F1-value against the sentence length. can find the correct span boundary, but both BERT and BERT+Word make incorrect predictions of the span type. Although BERT+Word can use the word information, it is disturbed by the irrelevant word    (Seven and Eight) predicting it as NUM. In contrast, LEBERT can not only integrate lexicon features but also choose the correct word for prediction. Adaptation at Different Layers . We explore the effect of applying the Lexicon Adapter (LA) between different Transformer layers of BERT on Ontonotes dataset. Different settings are evaluated, including applying LA after one , multiple , and all layers of Transformer. As for one layer, we applied LA after k  { 1 , 3 , 6 , 9 , 12 } layer; and { 1 , 3 } , { 1 , 3 , 6 } , { 1 , 3 , 6 , 9 } layers for multiple layers. All layers represents LA used after every Transformer layer in BERT. The results show in Table 7. Shallow layer achieves the better performance, which can be due to the fact that shallow layer promotes more layered interaction between lexicon feature and BERT. Applying LA at multi-layers of BERT hurts the performance and one possible reason is that integration at multi-layers causes over-fitting. Tuning BERT or Not . Intuitively, integrating lexicon into BERT without fine-tuning can be faster #1 Example of Chinese NER Sentence (truncated)         (Hulunbuir League, Inner Mongolia) Matched Words   ,    ,        ,   ,   ,     ,      ,   Inner Mongolia, Inner Mongolia, Inner Mongolia Hulunbuir, Mongolia, Hulun, Hulunbuir, Hulunbuir League, Buir Characters         Gold Labels B-GPE I-GPE E-GPE B-GPE I-GPE I-GPE I-GPE E-GPE BERT B-GPE I-GPE I-GPE I-GPE I-GPE I-GPE I-GPE E-GPE BERT+Word B-GPE I-GPE E-GPE B-ORG I-ORG I-ORG I-ORG E-ORG LEBERT B-GPE I-GPE E-GPE B-GPE I-GPE I-GPE I-GPE E-GPE #2 Example of Chinese POS Tagging Sentence (truncated)       (Messy Relationship) Matched Words    ,   ,   ,   Mess, Seven and Eight, Bad News, Relationship Characters        Gold Labels B-ADJ I-ADJ I-ADJ E-ADJ S-PART B-NOUN E-NOUN BERT B-ADJ I-NUM I-NUM E-ADJ S-PART B-NOUN E-NOUN BERT+Word B-ADJ I-NUM I-NUM E-ADJ S-PART B-NOUN E-NOUN LEBERT B-ADJ I-ADJ I-ADJ E-ADJ S-PART B-NOUN E-NOUN Table 8: Examples of tagging result. (Houlsby et al., 2019) but with lower performance due to the different characteristic of lexicon feature and BERT (discrete representation vs neural rep-resentation). To evaluate its impact, we conduct experiments with and without fine-tuning BERT parameters on Ontonotes and UD1 datasets. From the results, we find that without fine-tuning the BERT, the F1-score shows a decline of 7.03 points (82.08  75.05) on Ontonotes and 3.75 points (96.06  92.31) on UD1, illustrating the importance of fine-tuning BERT for our lexicon integration. In this paper, we proposed a novel method to integrate lexicon features and BERT for Chinese sequence labelling, which directly injects lexicon information between Transformer layers in BERT using a Lexicon Adapter. Compared with model-level fusion methods, LEBERT allows in-depth fusion of lexicon features and BERT representation at BERT level. Extensive experiments show that the proposed LEBERT achieves the state-of-the-art performance on ten datasets of three Chinese sequence labelling tasks. We would like to thank the anonymous reviewers for their valuable comments and suggestions. Moreover, We sincerely thank Dr. Zhiyang Teng for his constructive collaboration during the development of this paper, and Dr. Haixia Chai, Dr. Jie Yang, and my colleague Junfeng Tian for their help in polishing the paper.
abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result objective other other other other other other other other other other other other other other other other other other method abstain other other other other other method other abstain abstain abstain abstain method abstain method method method method method abstain method abstain method method abstain abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain method abstain method method method method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain result objective abstain abstain other other