Chinese pre-trained language models usually process text as a sequence of characters, while ignoring more coarse granularity, e.g., words. In this work, we propose a novel pre-training paradigm for Chinese  Lattice-BERT, which explicitly incorporates word representations along with characters, thus can model a sentence in a multi-granularity manner. Specifically, we construct a lattice graph from the characters and words in a sentence and feed all these text units into transformers. We design a lattice position attention mechanism to exploit the lattice structures in self-attention layers. We further propose a masked segment prediction task to push the model to learn from rich but redundant information inherent in lattices, while avoiding learning unexpected tricks. Experiments on 11 Chinese natural language understanding tasks show that our model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base -size models on the CLUE benchmarks. Further analysis shows that Lattice-BERT can harness the lattice structures, and the improvement comes from the exploration of redundant information and multi-granularity representations. 1 1 Introduction Pre-trained Language Models (PLMs) have achieved promising results in many Chinese Natural Language Understanding (NLU) tasks (Cui et al., 2019; Liu et al., 2020; Sun et al., 2020). These models take a sequence of fine-grained units  Chinese characters  as the input, following the English PLMs' practice (Devlin et al., 2019, BERT). Work done during an internship at Alibaba DAMO Academy. Corresponding author. 1 Our code will be available at https://github. com/alibaba/pretrained-language-models/LatticeBERT . However, the meanings of many Chinese words cannot be fully understood through direct compositions of their characters' meanings. For example,   / boss does not mean  / elder  / board . 2 The importance of word-level inputs in Chinese has been addressed in different tasks, including relation classification (Li et al., 2019), short text matching (Lai et al., 2019; Chen et al., 2020; Lyu et al., 2021), trigger detection (Lin et al., 2018), and named entity recognition (Zhang and Yang, 2018; Gui et al., 2019; Li et al., 2020a). The coarse-grained inputs benefit these tasks by introducing word-level semantics with multi-granularity representations, which is potentially complementary in character-level Chinese PLMs. In this work, we discuss how to pre-train a Chinese PLM over a word lattice structure to exploit multi-granularity inputs. We argue that by incorporating the coarse-grained units into PLM, models could learn to utilize the multi-granularity information for downstream tasks. Specifically, we organize characters and words in sentences as word lattices (see Figure 1), which enable the models to explore the words from all possible word segmentation results. However, it is not straightforward to learn a BERT-like PLM over the word lattices. The major challenges are two-folded. Firstly, BERT's original input is a sequence of characters ordered by their positions, making it difficult to consume the word lattices and preserve the positional relation-2 For clarity, we use   / English translation to represent an example in Chinese with its translation followed by. ship between multi-granularity units. Secondly, the conventional masked language modeling (MLM) task may make the word-lattice based PLMs learn unexpected tricks. The reason is that such a word lattice naturally introduces redundancy, that is, one character can be contained in multiple text units. In MLM, models may refer to the other text units overlapping with the randomly masked one instead of the real context, which brings information leakages. To address these challenges, we propose a Lattice-based Bidirectional Encoder Representation from Transformers (Lattice-BERT). Specifi-cally, we design a lattice position attention (LPA) to help the transformers directly exploit positional relationship and distances between text units in lattices. Moreover, we propose a masked segment prediction (MSP) task to avoid the potential leakage between overlapping text units in language modeling. With LPA and MSP, the Lattice-BERT could harness the multi-granularity structures in lattices, thus, directly utilize the lattice structures to aggregate the coarse-grained word information to benefit various downstream tasks. We evaluate our model on 11 Chinese NLU tasks in various paradigms, including the CLUE benchmarks (Xu et al., 2020) as well as two sequence labeling tasks. Compared with the baseline that only takes characters as inputs, Lattice-BERT bring an average increase of 1.5% and 2.0% under the settings of 12 and 6 layers, respectively. The 12-layer Lattice-BERT model beats all other base -size models on CLUE benchmarks. 3 Morever, we show that Lattice-BERT can harness the multi-granularity in-3 https://www.cluebenchmarks.com/rank. puts and utilize word-level semantics to outperform vanilla fine-grained PLMs. Our contributions can be summarized as 1) We propose Lattice-BERT to leverage multi-granularity representations from word lattices in Chinese PLMs. 2) We design lattice position attention and masked segment prediction to facilitate Chinese PLMs to exploit the lattice structures. 3) Lattice-BERT brings remarkable improvements on 11 Chinese tasks and achieves new state of the arts among base -size models at the CLUE benchmarks. This section, we detail the implementation of Lattice-BERT, and its overall framework is presented in Figure 2. BERTBERT (Devlin et al., 2019, Bidirectional Encoder Representations from Transformers) is a pre-trained language model comprising a stack of multi-head self-attention layers and fully connected layers. For each head in the l th multi-head self-attention layer, the output matrix H out ,l = (cid:110) h out ,l 1 , h out ,l 2 , . . . , h out ,l n (cid:111)  R n  d k satisfies: h out ,l i = n (cid:88) j =1 (cid:32) exp  lij (cid:80) j (cid:48) exp  lij (cid:48) h in ,l j W v,l (cid:33)  lij = 1  2 d k (cid:16) h in ,l i W q,l (cid:17) (cid:16) h in ,l j W k,l (cid:17) T (1) where H in ,l = (cid:110) h in ,l 1 , h in ,l 2 , . . . , h in ,l n (cid:111)  R n  d h is the input matrix, and W q,l , W k,l , W v,l  R d h  d k are learnable parameters. n and d h are sequence length and hidden size, and the attention size d k = d h /n h , where n h is the number of attention heads. To capture the sequential features in languages, previous PLMs adopt position embedding in either input representations (Devlin et al., 2019; Lan et al., 2020) or attention weights (Yang et al., 2019; Wei et al., 2019; Ke et al., 2020). For the input-level position embedding, the inputs of the first layer are (cid:101) h in , 0 i = h in , 0 i + P i , where P i is the embedding of the i th position. The other works incorporate position information in attention weights, i.e., (cid:101)  lij =  lij + f ( i, j ) , where f is a function of the position pair ( i, j ) . The BERT model is pre-trained on an unlabeled corpus with reconstruction losses, i.e., Masked Language Modeling (MLM) and Next Sentence Prediction (NSP), and then fine-tuned on downstream tasks to solve specific NLU tasks. Readers could refer to Devlin et al. (2019) for details. We adopt a word lattice to consume all possible segmentation results of a sentence in one PLM. Each segmentation can be a mixture of characters and words. As shown in Figure 1, a word lattice is a directed acyclic graph, where the nodes are positions in the original sentences, and each directed edge represents a character or a plausible word. Word lattices incorporate all words and characters so that models could explicitly exploit the inputs of both granularities, despite some of the words are redundant. In the rest of this work, we use lattice tokens to refer to text units, including the characters and words, contained in lattice graphs. As shown in Figure 2, we list the lattice tokens in a line and consume these tokens to transformers straightforwardly. However, the challenges of learning PLMs as BERT over the lattice-like inputs include: 1) encoding the lattice tokens while preserving lattice structures; 2) avoiding potential leakage brought by redundant information. Since the original BERT is designed for sequence modeling, it is not straightforward for BERT to consume a lattice graph. The word lattices encode not only the character sequences but also nested and overlapping words from different segmentations. To accurately incorporate positional information from lattice graphs into the interactions between Source T.2 (left-det) T.1 (self) T.7 (right-det.) T.3 (left-ovl.) T.6 (right-ovl.) T.5 (cted. by) T.4 (containing) Figure 3: An illustration of the positional relations. The lattice position attention aggregates the attention score of token representations,  ij in Eq. 1, with three position related attention terms, encoding the absolute positions, the distance, and the positional relationship, which can be formulated as: (cid:101)  ij =  ij + att ij + b ij + r ij (2) The att ij in Eq. att ij = 1  2 d k (cid:0)(cid:2) P Ss i ; P Ee i (cid:3) W q (cid:1) (cid:16)(cid:104) P Ss j ; P Ee j (cid:105) W k (cid:17) [  ;  ] means the concatenation of vectors. W q , W k  R 2 d e  d k are learnable parameters, d e and d k are embedding size and attention size. s i , e i are positions of start and end characters of the i th token. Taking the word   / research in Figure 1 as an example, it starts at the first character and ends at the second one, thus, its s i and e i are 1 and 2, respectively. PS and PE are learnable position embedding matrices. P St , P Et  R d e is the t th embedding vector of PS or PE . The att ij exploit the prior of attention weight between the start and end positions of the token pairs. The b ij in Eq. 2 is the attention term for the distance between the i th and j th tokens, which consists of four scaling terms considering the combinations of the start and end positions: b ij = b sss j  s i + b ses j  e i + b ese j  s i + b eee j  e i b sst reflects the attention weight brought by the relative distance t between the start positions of two tokens. The other terms, i.e., b set , b est , and b eet , have similar meanings. In practice, the distance t is clipped into [  128 , 128] . consider seven relations, including (1) self, (2) left and detached, (3) left and overlapped, (4) containing, (5) contained by, (6) right and overlapped, (7) right and detached. Figure 3 shows an illustration of these 7 relations. Formally, for the i th and j th tokens, they are overlapped means s i  s j < e i  e j or s j  s i < e j  e i , and if e i < s j or e j < s i , they are detached. If s i  s j  e j  e i and i (cid:54) = j , the i th token contains the j th token and the j th token is contained by the i th token. Intuitively, only two detached tokens can be concurrent in one Chinese word segmentation result. Moreover, the containing relation reflects a sort of lexical hierarchy in the lattices. We think r ij can explicitly model the positional relations between tokens in lattice graphs. We argue that the attention scores for distances and token relations capture different aspects of the multi-granularity structures in lattice graphs, thus, meeting the needs of various downstream tasks, such as distance for coreference resolution and positional relation for named entity recognition. With the information of absolute positions, distances, and positional relations, PLMs could accurately exploit the lattice structures in attention layers. The lattice position attention weights are shared over all layers. b ij , r ij , W q , and W k are diverse in different attention heads to capture diverse attention patterns. We follow Ke et al. (2020) to reset the positional attention scores related to [CLS] tokens, which is the special token as prefix of the input sequences to capture the overall semantics. Vanilla BERT is trained to predict the randomly masked tokens in the sentences, i.e., the masked language modeling (MLM). For the case of consuming multi-granularity inputs, the input tokens are redundant which means a character can occur in its character forms and multiple words it belongs to. Directly adopting the randomly masking strategy may simplify the prediction problem in our case because the masked token can be easily guessed via peeking the unmasked tokens overlapping with the masked one. Taking the word   / research in Figure 2 as an example, supposing the masked input is [M] /  /   , the model will consult   rather than the context to predict the masked token,  . tokens within a minimal segment of the lattice provide strong clue for the prediction of other tokens. A segment is a connected subgraph of a lattice where no token exists outside the subgraph that overlaps with any token inside the subgraph. To identify these minimal segments, we enumerate the character-level tokens in sentence order, checking if all the word-level tokens which contain this character end at this character. If so, all the tokens containing previous and current characters are considered as a segment, and the next segment starts from the next character, see the example in Figure 2. After the segment detection, we propose a masked segment prediction (MSP) task as a replacement of the MLM in the original BERT. In MSP, we mask out all the tokens in a segment and predict all these tokens (see Figure 2) to avoid the potential leakage. In addition to MSP, we also pre-train our models with the sentence order prediction (SOP) task in Lan et al. (2020), where the model predicts whether two consecutive sentences are swapped in inputs. We explore four kinds of downstream tasks, i.e., sentence/sentence-pair classification, multiple choices, sequence labeling, and span selection machine reading comprehension (MRC). For the sentence/sentence-pair classification, both vanilla and Lattice-BERT classify input instances base on logistic regressions over the representation of [CLS] tokens in the last layer. The circumstances are similar in multiple choice tasks, where softmax regressions are conducted over the representations of [CLS] tokens to choose the best options. However, for the span selection MRC, and the sequence labeling tasks like named entity recognition (NER), models need to perform token-wise classification. Vanilla BERT predicts labels for the input characters, but lattice-BERT has additional words. In Lattice-BERT, we extract the character chains (word pieces for numbers and English words) from lattices for training and prediction for a fair comparison with vanilla BERT. Pilot studies show that this strategy performs comparably with the more complex strategies, which supervise the labels over words and obtain a character's label via ensembles of all tokens containing that character. Lattice Construction. We construct the word lattices based on a vocabulary consisting of 102K high-frequency open domain words. All the substrings of the input sequence that appear in the vocabulary are considered lattices tokens of the input. With Aho-Corasick automaton (Aho and Corasick, 1975), this construction procedure can complete in linear time to the size of the corpus and the vocabulary. 4 To deal with English words and numbers where the substrings are meaningless, we use the character sequences for those out-of-vocabulary non-Chinese inputs and remain the in-vocabulary words and word pieces. We construct word lattices using all possible words according to a vocabulary instead of more sophisticated lattice construction strategies. Previous research efforts (Lai et al., 2019; Chen et al., 2020; Li et al., 2020b) on lattice construction suggests that using all possible words usually yields better performance. We think an overly-designed lattice construction method may bias our model on certain types of text, and would probably harm the generalization. So, in our case, we let the model learn by itself to filter the noise introduced by using all possible words during pre-training on a large-scale corpus. Pre-training Details. To compare with previous pre-training works, we implement the base -size models, which contains 12 layers, 768-dimensional of hidden size, and 12 attention heads. To demonstrate how lattice gains in shallower architectures and provide lightweight baselines, we also conduct the lite -size models with 6 layers, 8 attention heads, and the hidden size of 512. To avoid the large vocabulary introducing too many parameters in embedding matrix, we adopt the embedding decomposition trick following Lan et al. (2020, ALBERT). Consequently, the parameters of Lattice-BERT is 100M in base-size, only 11% more than its character-level counterpart (90M), and smaller than the RoBERTa-base (Liu et al., 2019) (102M) and AMBERT (Zhang and Li, 2020) (176M). The modeling of positional relation and distances in lattice position attention introduces only 12K parameters. A collection of Chinese text, including Chinese Wikipedia, Zhihu, and web news, is used in our BERT models' pre-training stage. The total number of characters in our unlabeled data is 18.3G. We follow Liu et al. (2019) and train the PLMs with a large batch size of 8K instances for 100K 4 Formally, the time complexity is O (( N + M )  L ) . We present the details of the Lattice-BERT fine-tuning results on 11 Chinese NLU tasks. Answering the following questions: (1) Whether the Lattice-BERT performs better than mono-granularity PLMs and other multi-granularity PLMs? (2) How the proposed lattice position attention and masked segment prediction contribute to the downstream tasks? (3) How Lattice-BERT outperforms the original character-level PLMs? We test our models on 11 Chinese NLU tasks, including the text classification and Machine Reading Comprehension (MRC) tasks in the Chinese Language Understanding Evaluation benchmark (Xu et al., 2020, CLUE), and two additional tasks to probe the effectiveness in sequence labeling. CLUE text classification : natural language inference CMNLI , long text classification IFLY-TEK ( IFLY. ), short text classification TNEWS , semantic similarity AFQMC , coreference resolution (CoRE) CLUEWSC 2020 ( WSC. ), and key word recognition (KwRE) CSL . CLUE MRC : Span selection based MRC CMRC 2018 ( CMRC ), multiple choice questions C 3 , and idiom cloze ChID . Sequence Labeling : Chinese word segmentation (CWS) MSR dataset from SIGHAN2005 (Emerson, 2005), and named entity recognition (NER) MSRA-NER (Levow, 2006). We probe our proposed Lattice-BERT model thoroughly with these various downstream tasks. The statistics and hyper-parameters of each task are elaborated in Appendix B. We tune learning rates on validation sets and report test results with the best developing performances for CLUE tasks. 5 For MSR and MSRA-NER, we run the settings with the best learning rates five times and report the average scores to ensure the reliability of results. RoBERTa (Cui et al., 2020) is the Chinese version RoBERTa model (Liu et al., 2019), which adopts the whole word masking trick and external pretraining corpus, known as RoBERTa-wwm-ext. 6 NEZHA (Wei et al., 2019) is one of the best Chinese PLMs with a bag of tricks, which also explores attention-level position embedding. AMBERT (Zhang and Li, 2020) is the state-of-the-art multi-granularity Chinese PLM, with two separated encoders for words and characters. BERT-word is a Chinese PLM baseline, taking words as single-granularity inputs. We obtain the results from Zhang and Li (2020) directly. BERT-our is our implemented BERT model, with the same pre-training corpus, model structures, hyper-parameters, and training procedure with Lattice-BERT, but taking characters as inputs. We also adopt the whole word masking trick. LBERT is our proposed Lattice-BERT model, with word lattices as inputs, equipping with lattice position attentions and masked segment prediction . In Table 1, we can see in text classification, MRC, and sequence labeling tasks, with both base and lite sizes, LBERT works better than our character-level baselines consistently. LBERTbase outperforms all previous base -size PLMs in average scores and obtain the best performances in 7 of the 11 tasks. Comparing with the mono-granularity PLMs in base -size, LBERT takes benefits from word-level information and outperforms its character-level counterpart, BERT-our, by 1.5% averagely. Meanwhile, LBERT performs better than the word-level model, BERT-word, remarkably on CLUE tasks. We think the lattice inputs incorporate 6 https://huggingface.co/hfl/ chinese-roberta-wwm-ext coarse-grained semantics while avoiding segmentation errors by combining multiple segmentation results. Therefore, with the multi-granularity treatments in word lattices, PLMs obtain better performances in downstream tasks than the mono-granularity settings. Furthermore, LBERT outperforms the previous state-of-the-art ( sota ) multi-granularity PLM, AMBERT (Zhang and Li, 2020), by 0.9% in text classification and 1.3% in MRC, averagely. Different from modeling the characters and words separately, the graph representations of word lattices could enhance the interaction between multi-granularity tokens and utilize all possible segmentation results simultaneously. As a result, LBERT achieves a new sota among the base -size models on the CLUE leaderboard as well as the sub-leaderboards for text classification and MRC tasks. 7 With lite -size settings, LBERT brings 2.0% improvement over BERT-our on average, which is larger than the case in base -size. In CWS, TNEWS, and CSL, the lite -size LBERT even outperforms the base -size BERT-our. With more coarse-grained inputs, the shallower architectures do not require complicated interactions to identify character combinations but utilizing word representations explicitly, thus, narrowing the gap with the deeper ones. Ablation Study. We conduct ablation experiments to investigate the effectiveness of our proposed lattice position attention (LPA) and masked segment prediction (MSP) in downstream tasks. To reduce the computational costs, we base our 7 https://www.cluebenchmarks.com/rank. pre-training settings on lite -size with the sequence length of 128 characters. We select one task from each of the task clusters. We use the entity-level F1 score for NER to highlight the influence on boundary prediction. We report the average scores over 5 runs and use the development sets for CLUE tasks. We can see in Table 2 that the ablation of either module (Dis.Rel. & MSP) leads to a substantial drop in the average scores. In particular, replacing MSP with vanilla MLM, the average score of  MSP drops by 1.6%. For the WSC. task, where long-range dependency is required to resolve the coreference, the gap is high up to 3.1%. We trace this drop into the pre-training procedure and observe the MLM accuracy for the MSP setting on the development set is 88.3%. However, if we mask the tokens within the segment and avoid potential leakages, the accuracy drastically drops to 48.8%, much lower than the performance of LBERT training with MSP (56.6%). This gap provides evidence that the MSP task prevents the PLMs from tricking the target by peeking the overlapping text units in one segment, thus encourages the PLMs to characterize the long-range dependency. For the LPA method, without the positional relation (Rel.), the entity-level F1 score on NER decreases by 0.4%, and the performance on CMRC decreases by 0.7%. The performance drops are similar to the case without distance information (Dis.). Without either of them (Dis. Rel.), the gaps widen to 0.5% and 2.8%, respectively. The boundary predictions in NER and CMRC are more sensitive to the local linguistic structures like nested words or overlapping ambiguity. With the positional relation and distance charaterized in attention, LBERT could accurately model the interaction between the nested and overlapping tokens in different segmentation results. Meanwhile, the accuracy of WSC. remarkably drops without distance information. The performance drops by 7.5% and 5.8% when the number of characters between the pronouns and candidate phrases is larger than 30, or between 20 to 30, respectively. For the rest cases, the drop is only 0.4%. With explicitly modeling of distance, LBERT predicts the long-distance coreference relations more accurately. Averagely, without the positional relation and distance modeling in LPA, the performance drops by 2.0% on the three tasks, showing the importance of LPA in assisting the PLMs to exploit the multi-granularity structures in word lattices. How LBERT Improves Fine-grained PLMs? We compare the prediction results of LBERT and the character-level BERT-our in base -size on development sets to investigate how the LBERT outperforms the vanilla fine-grained PLMs. Intuitively, the word-level tokens in lattices provide coarse-grained semantics, which argument the character-level inputs. We observe in TNEWS, the short text classification task, LBERT brings more improvement in the shorter instances, where the statements may be too short to provide enough context for predictions. By dividing the development set into five bins with equal size according to the sentence length, LBERT outperforms BERT-our by 2.3% and 1.3% in the shortest and second shortest bins, respectively, larger than the average gain on the rest instances (0.6%). We think the redundant tokens in word lattices provide rich context for the semantics of these short statements. For example, for the short title        / the cinema in our village , with the redundant words,   / movie ,   /cinema, and    / cinema , introduced in the lattice, LBERT classifies the instance as entertainment news instead of news stories. Another case is the CSL task, where the target is to predict whether the candidate words are keywords for a given paragraph. For those instances, where LBERT identifies more than two word-level tokens from each candidate word averagely, which accounts for 47% of the dataset, the performance gain is 3.0%, significantly larger than the average improvement of the rest, 1.0%. We think LBERT understands the key words from various aspects by exploiting the redundant expressions in lattices. For example, from the keyword candidate      / solar battery , the   /solar,    /solar energy,   /battery, and      / solar battery Figure 4: Visualization of the attention scores of        / Research life is very fulfilling . are lattice tokens. With these word-level tokens, LBERT could match this candidate with the expressions in the paragraph like   / positive electrode ,  / light ,   / electron ,   / ion , etc. On the other side, for MSRA-NER, LBERT reduces the errors in identifying entities with nested structures. Averagely, the number of error cases where the predicted entities are nested with the golden ones are reduced by 25% in LBERT. For example, the organization entity         / Palestine National Liberation Movement is nested with the location entity     / Palestine and ends with an indicator to organizations,   / movement . The character-level baseline model mistakenly recognizes the     / Palestine and  / move as a location and an organization, separately. While LBERT identifies this entity correctly after integrating the words,   / liberate ,     / Palestine , and   / movement . With the pre-trained multi-granularity representations, LBERT fuse the contextual information from words and characters simultaneously, and detects the correct entity in success. How does LBERT harness multi-Granularity representations? LBERT consumes all the words and characters from input sequences simultaneously, but how does the model utilizes such multi-granularity representations during pre-training and downstream tasks? To investigate this, we use the average attention scores that each lattice token receives among all layers and all heads to represent its importance. As the example shown in Figure 4, before fine-tuning, LBERT focuses on tokens including  / live ,   / fulfilling ,   / research ,    / graduate student ,  / investigate , etc. Before fine-tuning on specific tasks, the model captures various aspects of the sentence. After fine-tuning with MSRA-NER, the most focused words become   / fulfilling , Question : What is the game with the theme song which is sung by Chen Yiting and is composed by Zeng Zhihao? / very ,   / life , and   / research , i.e., the tokens from the golden segmentation result,    |   |  |   , which is intuitively beneficial for the NER tasks. The attention score of the wrong segmented word,    / graduate student , drops remarkably. On the other hand, after fine-tuning with the news title classification task, TNEWS, LBERT tends to focus on   / fulfilling ,    / graduate student ,   / life , etc. Although these tokens can not co-exist in one Chinese word segmentation result, LBERT can still utilize the redundant information from various plausible segmentations to identify the topics of inputs. These results indicate that Lattice-BERT can well manage the lattice inputs by shifting the attention to different aspects among the multi-granularity representations according to specific downstream tasks. Case Study. Table 3 shows an example in CMRC, a span selection MRC task, where models choose a text span from the given document to answer the question. In this case, the question asks for a game , restricted by its theme song . BERT-our incorrectly outputs a theme song, The Song of China , since there is no expression in the document explicitly related to game . However, LBERT find the correct answer, The Legend of Sword and Fairy V . One possible reason is that The Legend of Sword and Fairy is an entry in the vocabulary for lattice construction. LBERT may have learned this word as an entity for a famous video game from the context in pre-training by explicitly exploiting its representation as a whole. With the coarse-grain text units in pre-training, LBERT directly encodes knowledge about these units to benefit the downstream tasks. (i.e., BERT-our) have the same training epochs when the training steps are equal following previous works (Diao et al., 2020; Zhang and Li, 2020). Thus, comparing with BERT-our, 35% more text units are introduced in the pre-training instances of LBERT, which introduces 48% more computational resources comparing with BERT-our to process the additional word-level tokens (See Appendix C). To illustrate the gains attribute to the incorporation of lattices instead of additional computations, we investigate the lite -size BERT-our with longer input sequences in pre-training, which has the same computational costs as LBERT. We find LBERT still outperforms BERT-our by 2.2% averagely on CLUE classification tasks. More details are elaborated in Appendix D. 4 Related Works Recently, several works utilize the lattice structures to explore multi-granularity information in Chinese NLU tasks. Buckman and Neubig (2018) incorporate lattice into recurrent neural networks based language modeling to capture marginal possibilities across all possible paths. In NER, Lattice-LSTM (Zhang and Yang, 2018), graph neural networks (Gui et al., 2019), and flat-lattice transformers (Li et al., 2020a) are adopted to incorporate words from lattice inputs. Lai et al. (2019) adapt convolutional neural networks to lattice for matching based question answering. Chen et al. (2020) adopt graph matching networks to perform multi-granularity interaction between lattices for text similarity. These works are designed to explore word lattices in specific tasks. We explore the multi-granularity representations with word lattices in PLMs, investigating the previously attempted downstream tasks as well as other tasks, e.g., MRC. We design LPA to meet various interaction needs of the downstream tasks and propose MSP to avoid the leakages. In the field of Chinese PLMs, some efforts incorporate coarse-grained information with the character-level inputs. ERNIE 1.0 (Sun et al., 2019) and BERT-wwm (Cui et al., 2019) propose to mask the words, entities, and phrases as a whole in the MLM task to encourage the modeling of coarse-grained features. ZEN (Diao et al., 2020) adopt auxiliary networks to integrate n-gram representations. BERT-MWA (Li et al., 2020b) propose word-aligned attention to use multiple segmentation boundaries. Different from their methods, we propose Lattice-BERT to consume multi-granularity tokens simultaneously into one PLM via lattice graphs. Thus, Lattice-BERT explicitly exploits the representations of the coarse-grained units, as well as the interactions among wordand character-level tokens. The proposed MSP task can be treated as an extension of the whole word masking (Cui et al., 2019), while considering the span information like Joshi et al. (2020, SpanBERT) according the lattice structures. The concurrent work, Zhang and Li (2020, AMBERT) investigate multi-granularity inputs similarly, but they use two transformer encoders to separately deal the word and character sequences. We treat the words and characters as lattice graphs, which enables thor-ough interactions among multi-granularity tokens and utilizes all potential segmentation results. In this paper, we propose Lattice-BERT to leverage multi-granularity representations of input sentences for Chinese PLMs. Specifically, Lattice-BERT takes a word-lattice as input, modeling the representations of words and characters simultaneously. We design the lattice position attention to embed the multi-granularity structure into transformers and propose the masked segment prediction task to avoid potential leakage in original MLM caused by the redundancy information in lattices. We conduct extensive experiments on 11 Chinese NLU tasks and observe consistent gains over character-level baselines, achieving new sota on CLUE benchmarks. We show that Lattice-BERT can well manage the lattice inputs and utilize multi-granularity representations to augment the character-level inputs. We believe the lattice structure can be adapted to integrate the phrase and word representations into the word-piece based PLMs in other languages, which we leave for future exploration. This work is supported in part by the National Hi-Tech RD Program of China (No. 2020AAA0106600), the NSFC under grant agreements (61672057, 61672058). For any correspondence, please contact Yansong Feng.
abstain objective method method objective objective abstain abstain abstain abstain abstain other abstain abstain abstain other abstain method method objective abstain abstain method abstain abstain abstain abstain objective method objective abstain method abstain abstain other abstain objective method abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method method abstain method method abstain method abstain method abstain abstain method abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain other abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain result abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective objective objective method other other