Chinese pre-trained language models usually exploit contextual character information to learn representations, while ignoring the linguistics knowledge, e.g., word and sentence information. Hence, we propose a task-free enhancement module termed as H eterogeneous L inguistics G raph ( HLG ) to enhance Chinese pre-trained language models by integrating linguistics knowledge. Specifically, we construct a hierarchical heterogeneous graph to model the characteristics linguistics structure of Chinese language, and conduct a graph-based method to summarize and concretize information on different granularities of Chinese linguistics hierarchies. Experimental results demonstrate our model has the ability to improve the performance of vanilla BERT, BERTwwm and ERNIE 1.0 on 6 natural language processing tasks with 10 benchmark datasets. Further, the detailed experimental analyses have proven that this kind of mod-elization achieves more improvements compared with previous strong baseline MWA. Meanwhile, our model introduces far fewer parameters (about half of MWA) and the training/inference speed is about 7x faster than MWA. Our code and processed datasets are available at https://github.com/ lsvih/HLG . Pre-trained Language Models (PLM) (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018; Yang et al., 2019) have recently demonstrated the effectiveness on a variety of natural language processing (NLP) tasks, such as machine translation and text summarization. For a specific downstream task, the parameters of PLMs can be fine-tuned  Both authors contributed equally to this work  Corresponding Author with accurately labeled instances or weakly labeled instances of the task to achieve better performance. In recent, there are a series of studies on adapting PLMs for Chinese (Meng et al., 2019; Sun et al., 2019; Cui et al., 2019a; Sun et al., 2020; Wei et al., 2019; Diao et al., 2020; Lai et al., 2021). Many researchers introduce the Chinese-specific linguistics knowledge such as word information into PLMs by conducting elaborate self-supervised tasks to pretrain Chinese PLMs from scratch. Nevertheless, pre-training a PLM is computationally expensive and time-consuming since it needs large-scale Chinese corpus and heavy computational resources. The high cost makes it difficult for researchers to pre-train a PLM from scratch. An alternative way is to integrate the Chinese-specific linguistics knowledge into pre-trained PLMs in the fine-tuning stage in downstream tasks directly. Following this idea, the task-free enhancement module is widely used in the fine-tuning stage by adding an additional adapter in PLMs to integrate external knowledge (Li et al., 2020). As shown in Figure 1, the enhancement module is inserted between PLMs and task-specific module, and its inputs are the hidden representations of PLMs and embeddings of external knowledge. To achieve the goal of integrating external knowledge into PLMs in the fine-tuning stage, the enhancement module should have the following characteristics. First, as a plug-in adapter module in fine-tuning stage, it should maintain consistent output formulation with PLM. Second, it should not introduce unacceptable time or space complexity for training and inference. Third, it should improve the performance of downstream tasks universally. With the core idea of the enhancement module, Li et al. (2020) proposed a multi-source word-aligned model (MWA) to enhance PLMs by integrating Chinese Word Segmentation (CWS) bound-1986     Pre-trained Language Model   Task-SpecificModel Fine-tuning     Pre-trained Language Model   Task-SpecificModel Fine-tuning Enhancement Module  ExternalKnowledge Figure 1: The diagram of Enhancement Module framework. aries information implicitly. It first exploits various CWS tools to generate multiple word sequences and then utilizes word-aligned attention with a mixed pooling to integrate the word information into characters. Experimental results show that MWA has the ability to utilize CWS segmentation information to enhance Chinese PLMs to achieve SOTA performance in many downstream NLP tasks. However, MWA has two weaknesses: 1) Efficiency Degradation : The model structure of MWA is naturally non-parallel and cannot benefit from GPU acceleration (detailed in 4.3.3), which results in time inefficiencies in both training and inference processes. 2) Linguistic Information Loss : MWA utilizes a pooling-based mechanism to perform interaction between characters and words. Such a heuristic method could not make full use of information, resulting in sub-optimal results. To tackle the aforementioned limitations, we propose H eterogeneous L inguistics G raph ( HLG ), which is Graph Neural Network (GNN) based method to integrate CWS information to enhance PLMs. Specifically, the hierarchical CWS information is first conducted by a heterogeneous graph, which contains character nodes, word nodes and sentence nodes. The edge between nodes indicates the inclusion relationship of the grammatical structure between the linguistic hierarchies. Then, a simple but effective multi-step information propagation (MSIP) is proposed to incorporate the linguistics knowledge of heterogeneous graph to enhance Chinese PLMs inductively. In this way, we can obtain adequate information interaction among characters, words and sentences. Furthermore, the internal implementation of HLG is highly parallelized, which is conducive to GPU accelerate and raises the operating efficiency. In summary, we abstract out an adapter component named enhancement module for PLMs to integrate external knowledge during the fine-tuning stage. In this paradigm, we further introduce HLG to integrate CWS information delicately and model it via an effective MSIP. Extensive experiments conducted on 10 benchmark datasets of 6 NLP tasks demonstrate that our model outperforms the BERT, BERTwwm and ERNIE 1.0 significantly and steadily. Comparing with MWA, a strong baseline that also incorporates CWS information to enhance PLMs, our model achieves a steady improvement with the same information. Meanwhile, compared with previous work, MWA, our proposed HLG introduces only half additional parameters and the training/inference speed is about 7x faster. As mentioned in 1, the pre-trained language models (PLMs) have achieved great success in many NLP applications with the 2-stage paradigm of pretraining and fine-tuning. The PLMs usually perform pre-training on large-scale unlabeled corpus in virtue of self-supervised reconstruction tasks. For example, BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) is a typical well-known PLM, which conducts masked language modeling and next sentence prediction as pre-training tasks. After completing the pre-training, the PLMs learn substantial contextualized text representations, and then adapt fine-tuning on specific downstream tasks. In Chinese NLP, PLMs are generally character-based models (Li et al., 2019; Cui et al., 2019a). Specifically, given a character sequence: S = [ c 1 , c 2 , ..., c n ] (1) the outputs of Chinese PLMs can be treated as the character-level representations H  R n  d , where the d is the dimension of representation. As the same as most East-Asian languages, Chinese language is written without explicit word delimiters and the character is the smallest morpheme unit in Chinese linguistic (Cai and Zhao, 2016). Although character-based models could achieve good performance (Li et al., 2019), Li et al. (2020) point out that introducing Chinese Word Segmentation 1987         [CLS] [SEP] Characters Words Sentences  mountain  forest  park S2  west S3  westernmountain  forestalpark [SEP]  Beijing [CLS] S1 Figure 2: Overview of HLG structure. We give a formality definition of segmenter and its partition strategy  . Given a sentence consisting of a sequence of characters as Eq. 1, a segmenter is defined as: SEGMENTER   : S  S (cid:48) where  is a partition strategy of sentence. Specifi-cally,  partition and group the character sequence S into the word sequence S (cid:48) :  ( S ) = S (cid:48) = [ w 1 , w 2 , ..., w m ] (2) where m  n and w i = [ c s , c s +1 , ..., c s + l  1 ] is the i -th segmented word with a length of l and s is the index of w i 's first character in S . Namely, the word w i is a sequence of characters { c s , c s +1 , ..., c s + l  1 } , and the sentence S (cid:48) is a sequence of words { w 1 , w 2 , ..., w m } . Li et al. (2020) carried out researches on integrating CWS information into Chinese PLMs. The authors brought an architecture named Multi-source Word-aligned Attention (MWA) to incorporate multi-granularity segmentation via pooling attention weights among characters within the word. Formally, given a character sequence S as Eq. 1 and its partition strategy  as Eq. 2. The character-based representation H could be gained via PLM, MWA conducted self-attention between characters: A = softmax (cid:16) ( KW k )( QW q ) T  d (cid:17) where Q and K are both H , d is defined in 2.1, and A represents the attention score matrix. We decompose A over columns as [ a 1 , a 2 , ..., a n ] , and then perform partition  on it:  ( A ) = [ { a 1 , a 2 } , { a 3 } , ... { a sc , ..., a s + l  1 c } ..., { a n  1 , a n } ] where s and l are defined in 2.2. Pooling each group of partitioned columns: a iw = MixPooling ( { a sc , ..., a s + l  1 c } ) in which MixPooling is defined in Yu et al. (2014). The gained A w = [ a 1 w , a 2 w , ..., a mw ]  R n  m is the character-to-word attention weight matrix. After performing alignment-wise multiply (Li et al., 2020) between character-to-word attention weight matrix A w and the character-based representation H , the enhanced character-based representation which integrates CWS information can be obtained. In essence, the MWA conducts interaction between characters and words via character-to-word attention weight matrix A w , implicitly summary the information from characters, and performs MixPooling to aggregate the word-based segmentation information and concrete the character-level representation. This section introduces the components of our model HLG which instantiates the enhancement module by exploiting the CWS information. We first briefly explain the graph convolutional network as our base encoder, and then describe the graph construction of HLG. Finally, we give the details of the multi-step information propagation (MSIP) to integrate the CWS information into PLMs. Graph Convolutional Network (GCN) (Bruna et al., 2014; Kipf and Welling, 2017; Defferrard et al., 2016) is a powerful tool to extend the convolution operation from the grid data to irregular graph data. The basic idea of GCN is to aggregate the representations of neighbors to obtain better representation expression of nodes in the graph. For instance, consider a homogeneous graph G = ( V , E ) constructed by nodes set V and edges set E . A  R |V||V| is a binary adjacency matrix where each element A ij denotes whether node i has an edge with node j in the edge set E . Formally, a standard GCN layer can be abstracted as: H out =  ( AH in W ) ,  A = Norm ( A ) (3) where H in denotes the input representation matrix, H out is the updated representation matrix, Norm (  ) means row normalizing function,  A is the normalized adjacency matrix,  (  ) is the ReLU function and W is a parameter matrix. After such convolution operation, the representation H in were aggregated rely on edge connections defined by A , and transformed into H out by linear multiplication and active function. We build a heterogeneous graph G = ( C , W , S , E ) to model the structure of Chinese linguistic, where C , W , S , E denote the character nodeset, word nodeset, sentence nodeset and edge set, respectively. Besides, different from homogeneous graph, HLG models relationship between three granularities of linguistic in a hierarchical way. As presented in Figure 2, G is composed of three hierarchies including characters, words and sentences. In this case, we employed three different CWS tools, and got three different segmentation results, which resulted in three sentences with slightly different semantics. Note that the same word segmentation results in the same position obtained by different CWS tools will be regarded as the same word node to enhance the interaction (e.g., Beijing and park in Figure 2). This purpose is to denoise the mistake word nodes brought by segmenter error. If a word is segmented by multiple segmenters at the same time, the corresponding word node will have a higher vertex degree. Such nodes with higher betweenness centrality will lead to a stronger influence on the followed information propagation and achieve the effect of denoising intuitively, like the vote-based multi-model ensemble. In HLG, only one adjacency matrix A is not enough to describe the hierarchical relationships between characters, words and sentences. Hence, we " "#$ " $#% " %#$ " $#"  ! conduct two interaction matrices  A c 2 w  R |W||C| and  A w 2 s  R |S||W| to indicate aforementioned relationships. To be specific, we take the  A c 2 w as an example (the one for  A w 2 s is analogous), the element  A c 2 w ij denotes whether word i has an edge with character j in the edge set E . Similar to Eq. 3, we also denote normalized interaction matrices as  A c 2 w and  A w 2 s . To model the granularities hierarchical relationships in G , we devise a multi-step information propagation to learn the linguistics knowledge. In CWS, the partition and group processes could be considered as the partition of semantic representation and the aggregation of separated semantic respectively (detailed in 2.2). Inspired by CWS processes, we introduce two operations into MSIP to simulate such processes and named as summarization and concretization. Figure 3 shows the information propagation procedure of MSIP. Summarization. The summarization operation focuses on generalizing hierarchical word and sentence representations (e.g., from character-level to word-level). Specifically, given a heterogeneous graph G and corresponding character representations H c from PLM, the summarization operation can be formulated as follows: H w =  ( A c 2 w H c W c 2 w ) , H s =  (  A w 2 s H w W w 2 s ) , (4) where the W c 2 w , W w 2 s are parameter matrices, H w , H s are the interim representations of words and sentences. Concretization. Concretization is the inverse operation of summarization, it is used to repartition the semantics from high-level to low-level (e.g. from sentence-level to word-level). To do so, we first calculate the normalized interaction matrices  A s 2 w and  A w 2 c , which can be simply obtained by first transposed then normalized the predefined interaction matrices  A w 2 s and  A c 2 w , respectively. Thus, we have:  A s 2 w = Norm (cid:0) ( A w 2 s ) (cid:62) (cid:1) ,  A w 2 c = Norm (cid:0) ( A c 2 w ) (cid:62) (cid:1) , (5) where (  ) (cid:62) is the transpose function. Afterward, the concretization operation is defined as follows: H w (cid:48) =  ( A s 2 w H s W s 2 w ) , H c (cid:48) =  ( A w 2 c H w (cid:48) W w 2 c ) , (6) where W s 2 w and W w 2 c are parameter matrices, H w (cid:48) and H c (cid:48) are also interim word and character representations, H w (cid:48) denote the final word representations defined in Eq. 7. Skip Connection. Intuitively, it is difficult to generate satisfied low-level representations from the high-level representations directly. For example, it is easy to learn a few sentence representations from dozens of word representations, but hard to generate dozens of word representations from a few sentence representations. To mitigate this problem, in this paper, we introduce the skip connection to enhance the MSIP, which is to simulate the self-loop in vanilla GCN. As shown in Figure 3, we add skip connections between the summarization representations and the concretization representations directly. Formally, the skip connection can be simply expressed as: H w (cid:48) = H w (cid:48) +  ( H w W w ) , H c (cid:48) = H c (cid:48) +  ( H c W c ) , (7) where W w and W c are parameter matrices. Furthermore, H c (cid:48) denote the final representations for characters, which incorporates the fine-grained linguistics knowledge in G . For a fair comparison with MWA, which also gives an enhancement module by incorporating CWS information. We conduct the same experiments on five NLP tasks with various benchmark datasets. Three frequently-used Chinese PLMs: BERT (De-vlin et al., 2019), ERNIE 1.0 (Sun et al., 2019) and BERTwwm (Cui et al., 2019a) are employed as the basic PLM to enhance. Three CWS tools: thulac (Sun et al., 2016a), ictclas (Zhang et al., 2003) and hanlp (He, 2014) are employed to gain the segmentation information. The time of pre-processing including applying CWS tools is ignored in the experimental report. In the production, preprocessing and inference can be asynchronously executed in parallel (while inference a batch of data, the subsequence data can be preprocessed with multiprocess) (Cheng et al., 2019), all three of the CWS tools we've introduced are fast enough to achieve this effect. According to rough estimates and technical reports, the processing speed of these tools are thulac 1221KB/s, ictclas 769KB/s, hanlp 1375KB/s, respectively. Specifically, we instantiate the enhancement module as HLG and incorporate with downstream task-specific model. To verify the effectiveness of HLG, we execute 5 times fine-tuning on 10 benchmark datasets of 6 NLP tasks and report the average score. The tasks include Sentiment Classification (SC), Document Classification (DC), Named Entity Recognition (NER), Sentence Pair Matching (SPM), Natural Language Inference (NLI) and Machine Reading Comprehension (MRC). Specifi-cally, the following benchmark datasets are chosen to evaluate the performance: 1) SC : ChnSenti 1 and weibo100k 2 sentiment datasets are used for evaluating the capacity of short text classification. 2) DC : THUCNews (Sun et al., 2016b) dataset contains 10 types of news for performing long text classification. 3) NER : Ontonotes 4.0 (Weischedel et al., 2011) and MSRA-NER (Levow, 2006a) are used for testing model in sequence tagging task. 4) SPM : LCQMC (Liu et al., 2018) and BQ (Chen et al., 2018) are used to evaluate the text matching ability of model. 5) NLI : We conduct experiments on the Chinese part of XNLI (Conneau et al., 2018) dataset, and adopt the same pre-processing strategy as ERNIE (Sun et al., 2019). 6) MRC : Commonly used datasets DRCD (Shao et al., 2018) and CMRC2018 (Cui et al., 2019b) are tested. CMRC2018 is only evaluated on dev set as same as (Wei et al., 2019; Sun et al., 2020). We implement the presented approach in Py-Torch and fine-tune the downstream tasks on multiple Nvidia Tesla V100 GPUs. The basic architecture of PLMs and pre-trained parameters are provided by Huggingface (Wolf et al., 2020). The initial learning rate and other hyper-parameters refer to the previous works reported (Cui et al., 2019a; Li et al., 2020; Sun et al., 2020). Since the parameters of PLMs have been optimized, while the parameters of HLG and the downstream tasks are untrained. Hence, the learning rate of HLG part is larger than PLM part, we manually tuned the learning rates of PLM and HLG separately. The experimental results are shown in Table 1. Overall, we can observe that both HLG and MWA outperform baseline models (BERT, BERTwwm and ERNIE 1.0). Comparing with WMA, HLG achieves further improvement and significantly outperforms baseline models on 10 tasks. In detail, HLG outperforms MWA on ChnSent, weibo100k, MSRA-NER, ontonotes, LCQMC, BQ, THUCNews and XNLI tasks, and obtains comparable results on DRCD and CMRC2018 datasets. For the text classification tasks, namely SC and DC , HLG respectively achieves 0.88% and 0.84% average improvement on ChnSenti and weibo100k dataset, while MWA gains 0.53% and 0.82%. Meanwhile, HLG obtains 0.35% improvement on the long text multi-classification benchmark THUCNews, and MWA gets 0.31% points. Comparing with text classification tasks, the improvements over NER tasks are more obvious. The main reason may be that CWS explicitly provides the word boundaries, which are important to recognize entities accurately. On the ontonotes dataset, the promotion of HLG (1.28% averagely) is distinctly higher than that of MWA (0.92% av-eragely). Compared to the strong baseline models, the F1 scores of MSRA-NER have improved average 0.13% and 0.10% by HLG and MWA, respectively. HLG achieves the best results on the text matching tasks ( SPM ) and its variant NLI , which brings 1.28% average improvement to LCQMC, 0.26% average improvement to BQ, and 0.78% average improvement to XNLI. The improvements of HLG are much higher than that of MWA (0.3%, 0.23% and 0.55%). As described in Chen et al. (2020) and 1991 No. Lyu et al. (2021), text matching tasks can benefit from the interaction between the paired sentences. HLG follows them to construct graphs over sentence pairs collectively, which naturally obtains advantages in text matching tasks. For MRC task, HLG and MWA achieve comparable results on those datasets. HLG gets an average improvement of 1.41 in EM and 0.85 in F1 score, while MWA gets 1.4 EM and 0.86 F1 score. However, HLG has dominant advantage in training speed and inference speed. Detail analysis of time efficiency is in 4.3.3. We conduct ablation experiments to explore the effectiveness of the number of CWS tools. The ablation experiments are organized on sentiment classification task, ChnSenti and weibo100k dev set. As shown in Table 2, 5 popular CWS tools are added into our model successively according to the order, and we also show the total number of word nodes in our HLG. Meanwhile, the information from multiple word segmentation tools can be integrated at the same time without increasing parameter size in HLG (only the A is changed). Figure 4 shows the performance of BERT+HLG with different numbers of CWS tools on ChnSenti and weibo100k dev sets. Experimental results demonstrate the effectiveness of introducing word segmentation information. We can observe that when the number of CWS tools is larger, the number of generated word nodes gradually increasing to converge, and the performance of the model slightly is not always increasing as the word count. The more CWS tools introduced will bring more diversity but also bring noise caused by segmenter error. In practice, we find using 4 or more CWS tools can slightly increase the performance but take much longer preprocessing time, hence we select the elbow of the curve as the number of CWS tools. That is, using 3 as the number of CWS tools might be a balance between the performance of model and the cost of preprocessing. This number also coincides with the configuration in MWA. In general, the enhancement module should be able to bring performance improvements without unacceptable space complexity. Therefore, we conduct a comparative experiment on XNLI dev set to explore the performance improvement and the space overhead between MWA and HLG. To be specific, the number of parameters in MWA depends on the dimension of PLM's representation and the number of CWS tools K . Concretely, MWA contains K transformer layers and 1 aggregation layer. Nevertheless, our HLG only depends on the dimension of PLM's representation and simply contains 4 basic GCN layers and 2 skip connections. Thus, the number of parameters of 1992 Model Params. size ( MWA ) = K  (4  d 2 ) Transformer + d size ( HLG ) = (4  d 2 ) GCN + 2  d 2 As discussed before, we employ 3 CWS tools in both MWA and HLG. Table 3 reports the performance of BERT, BERTwwm, and ERNIE 1.0 on the XNLI dev set. Obviously, HLG can get a greater performance improvement with only half additional parameters. It shows that as an enhancement module, HLG is superior to MWA in terms of parameter utilization efficiency. In addition, to verify the impact of the additional parameters , we also conduct an ablation experiment on XNLI dev set that utilizes the random tokenizer, the single-character tokenizer, and sole segmenter to obtain the different word segmentation results, and send those results to HLG to eliminate the additional benefit from the change of neural network structure and the increase of parameters. The results are shown in Table 4, which indicates that the increment of parameters can slightly affect character-based model performance, and the CWS information is significantly useful to promote the performance of character-based PLM. Time efficiency is an important indicator in the real-world production. Less training time and inference time means lower costs. In order to analyze the additional time cost of different enhancement modules, we conduct comparative experiments among BERT, BERT+MWA, and BERT+HLG on ChnSenti, LCQMC and XNLI datasets. For the fair comparison, we remain other hyper-parameters consistent for the three models. As shown in Figure 5, we compare time cost during training and inference between vanilla BERT, BERT+MWA and BERT+HLG. We can observe that the training time and inference time of Figure 5: The training time, inference time of vanilla BERT, BERT+MWA and BERT+HLG on ChnSenti, LCQMC and XNLI benchmarks. BERT+HLG are basically consistent with vanilla BERT. However, when MWA is introduced, the average training time increases by 7 times, and the average inference time increases by 7.6 times. This is because MWA must calculate aligned attention weights token by token, and it cannot benefit from CUDNN parallelization, resulting in terrible operating efficiency. On the contrary, HLG is composed of GCNs, and its internal implementation is basically the simplest non-linear transformation. Therefore, HLG could be maximally accelerated through the optimized matrix operation of CUDNN primitive, which only produces a negligible impact on time efficiency. Pre-training language models, such as ELMo (Pe-ters et al., 2018), BERT (Devlin et al., 2019), XL-NET (Yang et al., 2019) and GPT (Radford et al., 2018), have shown their powerful performances on various natural language processing tasks and have been applied in many applications. In recent past, there are studies adapting PLMs for Chinese with Chinese-specific features such as word information. Glyce (Meng et al., 2019) proposed to use the glyph information of Chinese characters to enhance PLMs. ERNIE 1.0/2.0 (Sun et al., 2019, 2020) and BERTwwm (Cui et al., 2019a) used the whole word mask to learn the structure of words or entities in the pre-training stage and conducted more and better pre-training tasks to perceive large-scale data. NEZHA (Wei et al., 2019) used a series of methods such as functional relative 1993 positional encoding and whole word masking to improve the pre-training tasks, which had brought improvement. ZEN (Diao et al., 2020) adopted n-gram masking to enhance pre-trained encoder and obtained outstanding performance. Lattice-BERT (Lai et al., 2021) introduced word lattice information (Zhang and Yang, 2018) into pre-training framework via lattice position attention. As a fundamental feature of Chinese, word segmentation information is flexibility, granularity, and easy-to-get (Sproat and Emerson, 2003; Levow, 2006b). Further, Zhang et al. (2018); Li et al. (2019, 2020) conducted detailed research and experiments on the application of CWS in deep learning, and found that CWS information can effectively improve the performance of Chinese character-based PLMs. Recently, a lot of works have been proposed to prompt NLP applications by constructing graph on text and modeling with graph neural networks. Yao et al. (2019) first constructed word co-occurrence graph between documents and introduced GCN to modeling and aggregating document representation for text classification. Chen et al. (2020); Lyu et al. (2021) constructed lattice graph to maintain multi-granularity information and external knowledge in Chinese short text matching task. Nguyen and Grishman (2018) proposed performing GCN over dependency trees to extract event trigger. Sui et al. (2019) conducted a character-word interaction graph and performed graph attention network on it to recognize Chinese named entities. Shu et al. (2020) introduced a bipartite-graph based transformer PLM for integrating hierarchical semantic information. In this paper, we propose HLG which acts as the enhancement module to enhance Chinese PLMs with CWS information. The HLG firstly constructs heterogeneous graph based on multiple word segmentations to model the hierarchy of Chinese. Then, the MSIP is proposed to model the fine-grained linguistics knowledge of the heterogeneous graph. Experimental results on 6 NLP tasks with 10 benchmark datasets demonstrate that the performance of our model outperforms previous work, MWA. Besides the performance improvements, HLG introduces only half the additional parameters of MWA and its training/inference speed is 7x faster than MWA. At present, the experimental results of HLG are lagging behind SOTA, and we will try to migrate it to some of the latest PLMs. Besides, HLG has the expansibility to introduce the representation layer of the CWS model directly, or introduce some other information sources such as the knowledge graph, etc. We leave these further improvements to the future. This work is supported by the National Key Research and Development Program of China (grant No.2021YFB3100600 and 2020YFB2103803), the Strategic Priority Research Program of Chinese Academy of Sciences (grant No.XDC02040400), the Youth Innovation Promotion Association of CAS (Grant No. 2021153) and National Natural Science Foundation of China (Grant No.61902394). This work performed while the first author was at IIE, CAS.
abstain objective objective objective abstain abstain other abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain result abstain abstain abstain objective result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain other other other other other other other other other other other other other other other objective abstain abstain objective abstain result abstain abstain other other