Most Chinese pre-trained models take character as the basic unit and learn representation according to character's external contexts, ignoring the semantics expressed in the word, which is the smallest meaningful utterance in Chinese. Hence, we propose a novel word-aligned attention to exploit explicit word information, which is complementary to various character-based Chinese pre-trained language models. Specifically, we devise a pooling mechanism to align the character-level attention to the word level and propose to alleviate the potential issue of segmentation error propagation by multi-source information fusion. As a result, word and character information are explicitly integrated at the fine-tuning procedure. Experimental results on five Chinese NLP benchmark tasks demonstrate that our method achieves significant improvements against BERT, ERNIE and BERT-wwm. Pre-trained language Models (PLM) such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), ERNIE (Sun et al., 2019), BERT-wwm (Cui et al., 2019) and XLNet (Yang et al., 2019) have been proven to capture rich language information from text and then benefit many NLP applications by simple fine-tuning, including sentiment classification (Pang et al., 2002), natural language inference (Bowman et al., 2015), named entity recognition (Sang and De Meulder, 2003) and so on. Generally, most popular PLMs prefer to use attention mechanism (Vaswani et al., 2017) to represent the natural language, such as word-to-word self-attention for English. Unlike English, in Chinese, words are not separated by explicit delimiters. Since without word boundaries information, it is  Corresponding author intuitive to model characters in Chinese tasks directly. However, in most cases, the semantic of a single Chinese character is ambiguous. For example, the character    in word    (bat) and    (auction) has entirely different meanings. Moreover, several recent works have demonstrated that considering the word segmentation information can lead to better language understanding, and accordingly benefits various Chinese tasks (Wang et al., 2017; Li et al., 2018; Zhang and Yang, 2018; Gui et al., 2019; Mengge et al., 2019). All these factors motivate us to expand the character-level attention mechanism in Chinese PLMs to represent the semantics of words 1 . To this end, there are two main challenges. (1) How to seamlessly integrate the segmentation information into character-based attention module of PLM is an important problem. (2) Gold-standard segmentation is rarely available in the downstream tasks, and how to effectively reduce the cascading noise caused by Chinese word segmentation (CWS) tools (Li et al., 2019) is another challenge. In this paper, we propose a new architecture, named M ulti-source W ord A ligned Attention (MWA), to solve the above issues. (1) Psycholinguistic experiments (Bai et al., 2008; Meng et al., 2014) have shown that readers are likely to pay approximate attention to each character in one Chinese word. Drawing inspiration from such find-ings, we introduce a novel word-aligned attention, which could aggregate attention weight of characters in one word into a unified value with the mixed pooling strategy (Yu et al., 2014). (2) For reducing segmentation error, we further extend our word-aligned attention with multi-source segmentation produced by various segmenters and deploy 1 Considering the enormous cost of re-training a language model, we hope to incorporate word segmentation information to the fine-tuning process to enhance performance, and leave how to improve the pre-training procedure for a future work. a fusion function to pull together their disparate outputs. As shown in Table 1, different CWS tools may have different annotation granularity. Through comprehensive consideration of multi-granularity segmentation results, we can implicitly reduce the error caused by automatic annotation. Extensive experiments are conducted on various Chinese NLP tasks including sentiment classification, named entity recognition, sentence pair matching, natural language inference and machine reading comprehension. The results and analysis show that the proposed method boosts BERT, ERNIE and BERT-wwm significantly on all the datasets 2 . The primary goal of this work is to inject the word segmentation knowledge into character-level Chinese PLMs and enhance original models. Given the strong performance of deep Transformers trained on language modeling, we adopt BERT and its updated variants (ERNIE, BERT-wwm) as the basic encoder in this work, and the outputs from the last layer of encoder are treated as the character-level enriched contextual representations H . Although character-level Chinese PLM has remarkable ability to capture language knowledge from text, it neglects the semantic information expressed in the word level. Therefore we apply a word-aligned layer on top of the encoder to integrate the 2 The source code of this paper can be obtained from https://github.com/lsvih/MWA . word boundary information into the representation of characters with an attention aggregation module. For an input sequence with n characters S = [ c 1 , c 2 , ..., c n ] , where c j denotes the j -th character, CWS tool  is used to partition S into nonoverlapping word blocks:  ( S ) = [ w 1 , w 2 , ..., w m ] , ( m  n ) (1) where w i = { c s , c s +1 , ..., c s + l  1 } is the i -th segmented word with a length of l and s is the index of w i 's first character in S . We apply self-attention operation with the representations of all input characters to get the character-level attention score matrix A c  R n  n . It can be formulated as: A c = F ( H ) = softmax (( KW k )( QW q ) T  d ) (2) where Q and K are both equal to the collective representation H at the last layer of the Chinese PLM, W k  R d  d and W q  R d  d are trainable parameters for projection. While A c models the relationship between two arbitrarily characters without regard to the word boundary, we argue that incorporating word as atom in the attention can better represent the semantics, as the literal meaning of each individual character can be quite different from the implied meaning of the whole word, and the simple weighted sum in the character level may lose word and word sequence information. To address this issue, we propose to align A c in the word level and integrate the inner-word attention. For ease of exposition, we rewrite A c as [ a 1 c , a 2 c , ..., a nc ] , where a ic  R n denotes the i -th row vector of A c , that is, a ic represents the attention score vector of the i -th character. Then we deploy  to segment A c according to  ( S ) . For example, if  ( S ) = [ { c 1 , c 2 } , { c 3 } , ..., { c n  1 , c n } ] , then  ( A c ) = [ { a 1 c , a 2 c } , { a 3 c } , ..., { a n  1 c , a nc } ] (3) In this way, an attention vector sequence is divided into several subsequences and each subsequence represents the attention of one word. Then, motivated by the psycholinguistic finding that readers are likely to pay similar attention to each character in one Chinese word, we devise an appropriate aggregation module to fuse the inner-word attention. Concretely, we first transform { a sc , ..., a s + l  1 c } into one attention vector a iw for w i with the mixed pooling strategy (Yu et al., 2014) 3 . Then we execute the piecewise upsampling operation over each a iw to keep input and output dimensions unchanged for the sake of plug and play. The detailed process can be summarized as: a iw =  Maxpooling ( { a sc , ..., a s + l  1 c } ) (4) + (1   ) Meanpooling ( { a sc , ..., a s + l  1 c } )  A c [ s : s + l  1] = e l  a iw (5) where   R 1 is a weighting trainable variable to balance the mean and max pooling, e l = [1 , ..., 1] T represents a l -dimensional all-ones vector, l is the length of w i , e l  a iw = [ a iw , ..., a iw ] denotes the kronecker product operation between e l and a iw ,  A c  R n  n is the aligned attention matrix. Eqs. 4 and 5 can help incorporate word segmentation information into character-level attention calculation process, and determine the attention vector of one character from the perspective of the whole word, which is beneficial for eliminating the attention bias caused by character ambiguity. Finally, we can obtain the enhanced character representation produced by word-aligned attention as follows:  H =  A c VW v (6) where V = H , W v  R d  d is a trainable projection matrix. Besides, we also use multi-head attention (Vaswani et al., 2017) to capture information from different representation subspaces jointly, thus we have K different aligned attention matrices  A k c (1  k  K ) and corresponding representation  H k . With multi-head attention architecture, the output can be expressed as follows: H = Concat (  H 1 ,  H 2 , ...,  HK ) W o (7) 2.3 Multi-source Word-aligned Attention As mentioned in Section 1, our proposed word-aligned attention relies on the segmentation results 3 Other pooling methods such as max pooling or mean pooling also works. Here we choose mixed pooling because it has the advantages of distilling the global and the most prominent features in one word at the same time. of CWS tool  . Unfortunately, a segmenter is usually unreliable due to the risk of ambiguous and non-formal input, especially on out-of-domain data, which may lead to error propagation and an unsatisfactory model performance. In practice, the ambiguous distinction between morphemes and compound words leads to the cognitive divergence of words concepts, thus different  may provide diverse  ( S ) with various granularities. To reduce the impact of segmentation error and effectively mine the common knowledge of different segmenters, it's natural to enhance the word-aligned attention layer with multi-source segmentation inputs. Formally, assume that there are M popular CWS tools employed, we can obtain M different representations H 1 , ..., HM by Eq. 7. Then we propose to fuse these semantically different representations as follows:  H = M (cid:88) m =1 tanh( H m W g ) (8) where W g is a parameter matrix and  H denotes the final output of the MWA attention layer. To test the applicability of the proposed MWA attention, we choose three publicly available Chinese pre-trained models as the basic encoder: BERT, ERNIE, and BERT-wwm. In order to make a fair comparison, we keep the same hyper-parameters (such maximum length, warm-up steps, initial learning rate, etc.) as suggested in BERT-wwm (Cui et al., 2019) for both baselines and our method on each dataset. We run the same experiment for five times and report the average score to ensure the reliability of results. Besides, three popular CWS tools: thulac (Sun et al., 2016), ictclas (Zhang et al., 2003) and hanlp (He, 2014) are employed to segment sequence. NLP tasks and six public benchmark datasets: Sentiment Classification (SC) : We adopt ChnSentiCorp 4 and weibo-100k sentiment dataset 5 in this task. ChnSentiCorp dataset has about 10k sentences, which express positive or negative emotion. weibo-100k dataset contains 1.2M microblog 4 https://github.com/pengming617/bert_ classification 5 https://github.com/SophonPlus/ ChineseNlpCorpus/ Dataset Task Max length Batch size Epoch lr  Dataset Size Train Dev Test ChnSentiCorp SC 256 16 3 3  10  5 9.2K 1.2K 1.2K weibo-100k 128 64 2 2  10  5 100K  10K 10K ontonotes NER 256 16 5 3  10  5 15.7K 4.3K 4.3K LCQMC SPM 128 64 3 3  10  5  239K 8.8K 12.5K XNLI NLI 128 64 2 3  10  5  392K 2.5K 2.5K DRCD MRC 512 16 2 3  10  5 27K 3.5K 3.5K Table 2: Summary of datasets and the corresponding hyper-parameters setting. Named Entity Recognition (NER) : this task is to test model's capacity of sequence tagging. We use a common public dataset Ontonotes 4.0 (Weischedel et al., 2011) in this task. Sentence Pair Matching (SPM) : We use the most widely used dataset LCQMC (Liu et al., 2018) in this task, which aims to identify whether two questions are in a same intention. Natural Language Inference (NLI) : this task is to exploit the contexts of text and concern inference relationships between sentences. XNLI (Con-neau et al., 2018) is a cross-language language understanding dataset; we only use the Chinese language part of XNLI to evaluate the language understanding ability. And we processed this dataset in the same way as ERNIE (Sun et al., 2019) did. Machine Reading Comprehension (MRC) : MRC is a representative document-level modeling task which requires to answer the questions based on the given passages. DRCD (Shao et al., 2018) is a public span-extraction Chinese MRC dataset, whose answers are spans in the document. We implement our model with PyTorch (Paszke et al., 2019), and all baselines are converted weights into PyTorch version. All experiments employ modified Adam (Devlin et al., 2019) as optimizer with 0.01 weight decay and 0.1 warmup ratio. All pre-trained models are configured to 12 layers and 768 hidden dimension. The detail settings are shown in Table 2. Table 3 shows the performances on five classical Chinese NLP tasks with six public datasets. Generally, our method consistently outperforms all baselines on all five tasks, which demonstrates the effectiveness and universality of the proposed approach. Moreover, the Wilcoxon's test shows that a significant difference ( p < 0 . 05 ) exits between our model and baseline models. In detail, on the two datasets of SC task, we observe an average of 0.53% and 0.83% absolute improvement in F1 score, respectively. SPM and NLI tasks can also gain benefits from our enhanced representation. For the NER task, our method obtains 0.92% improvement averagely over all baselines. Besides, introducing word segmentation information into the encoding of character sequences improves the MRC performance on average by 1.22 points and 1.65 points in F1 and Exact Match (EM) score respectively. We attribute such significant gain in NER and MRC to the particularity of these two tasks. Intuitively, Chinese NER is correlated with word segmentation, and named entity boundaries are also word boundaries. Thus the potential boundary information presented by the additional segmentation input can provide better guidance to label each character, which is consistent with the conclusion in (Zhang and Yang, 2018). Similarly, the span-extraction MRC task is to extract answer spans from document (Shao et al., 2018), which also faces the same word boundary problem as NER, and the long sequence in MRC exacerbates the problem. Therefore, our method gets a relatively greater improvement on the DRCD dataset. To demonstrate the effectiveness of our multi-source fusion method, we carry out experiments on the DRCD dev set with different segmentation inputs. Besides, we also design two strong baselines by introducing a Transformer layer ( 1 T ) and a random tokenizer model (WA random ) to exclude the benefits from additional parameters. As shown in Table 4, adding additional parameters by introducing an extra transformer layer can benefit the PLMs. Compared with 1 T and WA random , our proposed word-aligned attention gives quite stable improvements no matter what CWS tool we use, which again confirms the effectiveness and rationality of incorporating word segmentation information into character-level PLMs. Another observation is that Task SC NER SPM NLI MRC Dataset ChnSenti 2 , 3 weibo-100k 2 Ontonotes 4 LCQMC 2 , 3 , 4 XNLI 1 , 2 , 3 , 4 DRCD 2 , 3 [EM | F1] Prev. employing multiple segmenters and fusing them together could introduce richer segmentation information and further improve the performance. For fair comparison and demonstrating the improvement of our model is not only rely on more trainable parameters, we also conduct experiments on the DRCD dev set to explore whether the performance keeps going-up with more parameters by introducing additional transformer blocks on top of the representations of PLMs. In Table 5, +1 T denotes that we introduce another one Transformer layer on top of BERT-wwm and +2 T means additional 2 layers, M denotes million. As the experimental results showed, when the number of additional layers exceeds 1, the performance starts to decline, which demonstrates that using an extensive model on top of the PLM representations may not bring additional benefits. We can conclude that MWA doesn't introduce too many parameters, and MWA achieves better performance than +1T under the similar parameter numbers. Besides, we also make comparison with the current best Chinese PLM: Robust-BERT-wwm-ext-large (Cui et al., 2019), a 24-layers Chinese PLM with 13.5 times more pre-training data and 3.1 times more parameters than BERT-wwm, experimental results show that our model can achieve comparable performance, which again confirms the effectiveness of incorporating word segmentation information into character-level PLMs. In this paper, we develop a novel Multi-source Word Aligned Attention model (referred as MWA), which integrates word segmentation information into character-level self-attention mechanism to enhance the fine-tuning performance of Chinese PLMs. We conduct extensive experiments on five NLP tasks with six public datasets. The proposed approach yields substantial improvements compared to BERT, BERT-wwm and ERNIE, demonstrating its effectiveness and universality. Furthermore, the word-aligned attention can also be applied to English PLMs to bridge the semantic gap between the whole word and the segmented Word-Piece tokens, which we leave for future work. We would like to thank reviewers for their insightful comments. This work is supported by the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDC02040400.
abstain objective objective abstain objective other abstain abstain abstain abstain abstain abstain objective abstain abstain abstain objective abstain abstain objective abstain abstain result abstain abstain abstain method abstain other abstain other method abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method method method abstain method abstain other abstain method method abstain method method abstain abstain method abstain abstain abstain abstain abstain result result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result objective method abstain method other other other