In this paper, we tackle the task of Definition Generation (DG) in Chinese, which aims at automatically generating a definition for a word. Most existing methods take the source word as an indecomposable semantic unit. However, in parataxis languages like Chinese, word meanings can be composed using the word formation process, where a word ("   ", peachblossom ) is formed by formation components ("  ", peach ; "  ", flower ) using a formation rule ( Modifier-Head ). Inspired by this process, we propose to enhance DG with word formation features. We build a formation-informed dataset and propose a model DeFT , which Decomposes words into formation features, dynamically Fuses different features through a gating mechanism, and generaTes word definitions. Experimental results show that our method is both effective and robust. 1 1 Introduction Definition Generation (DG) aims at automatically generating an explanatory text for a word. This task is of practical importance to assist dictionary construction, especially in highly productive languages like Chinese (Yang et al., 2020). Most existing methods take the source word as an indecomposable lexico-semantic unit, using features like word embedding (Noraset et al., 2017) and context (Gadetsky et al., 2018; Ishiwatari et al., 2019). Recently, Yang et al. (2020) and Li et al. (2020) achieve improvement by decomposing the word meaning into different semantic components. In decomposing the word meaning, the word formation process is an intuitive and informative way that has not been explored in DG by far. For parataxis languages like Chinese, a word is formed by formation components, i.e., morphemes , and  Equal contribution. (cid:20914) 2 : (cid:20912)(cid:23608) ( spend) (cid:17790) 2 : (cid:17788)(cid:17788)(cid:9775) ( vainly) (cid:20914) 1 : (cid:20912)(cid:13876) ( flower) (cid:17790) 1 : (cid:17788)(cid:20849)(cid:17795) ( white) Modifier-Head Adverb-Verb (cid:17790)(cid:20914) (cid:20) (cid:29)(cid:3) (cid:17788)(cid:20849)(cid:17795)(cid:20912)(cid:572) ( White flower.) (cid:17790)(cid:20914) (cid:21) (cid:29) (cid:17788)(cid:17788)(cid:9775)(cid:20912)(cid:23608)(cid:572) ( Vainly spend.) Formation Rule Word Definition (cid:17790)(cid:20914) Morphemes: Definitions Word Figure 1: Word formation process for the polysemous "   ". a formation rule . As shown in Figure 1, the polysemous word "   " holds two meanings "   1 " and "   2 ", which can be distinguished by different morphemes ("  1 ;  1 " vs. "  2 ;  2 ") and different rules ( Modifier-Head vs. Adverb-Verb ). Such intuitive formation process can clearly and unambiguously construct the word meaning. Inspired by the word formation process in Chinese, we propose to enhance DG with formation features. First, we build a formation-informed dataset under expert annotations. Next, we design a DG model DeFT , which Decomposes words into formation features, Fuses different features through a gating mechanism, and generaTes definitions. Our contributions are as follows: (1) We first propose to use word formation features to enhance DG and design a formation-informed model DeFT . (2) We build a new formation-informed DG dataset under expert annotations. (3) Experimental results show that our method brings a substantial performance improvement, and maintains a robust performance even with only word formation features. Definition Generation: Noraset et al. (2017) first propose the DG task and use word embeddings as the main input. The following methods add contexts for disambiguation (Gadetsky et al., 2018; Ishiwatari et al., 2019) or word-pair embeddings to capture lexical relations (Washio et al., 2019). Recent methods attempt to decompose the word meaning by using HowNet sememes (Yang et al., 2020) or modeling latent variables (Li et al., 2020). Semantic Components: To systematically define words, linguists decompose the word meaning into semantic components (Wierzbicka, 1996). Following this idea, HowNet (Dong and Dong, 2006) uses manually-created sememes to describe the semantic aspects of words. Recent studies also show that leveraging subword information produces better embeddings (Park et al., 2018; Lin and Liu, 2019; Zhu et al., 2019), but these methods lack a clear distinction among different formation rules. It is linguistically motivated to explore the word formation process to better understand words. Instead of combining roots and affixes, Chinese words are formed by characters in a parataxis way (Li et al., 2018). Here, we introduce two formation features and construct a formation-informed dataset. Chinese formation components are morphemes , defined as the smallest meaning-bearing units (Zhu, 1982). Morphemes are unambiguous in representing word meanings, since they can distinguish different meanings and uses of each character in a word, like "  1 " and "  2 " in Figure 1. Morphemes are also productive in constructing words, since over 99.48% Chinese words are formed using a small set of nearly 20,000 morphemes (Fu, 1988). These properties make morphemes highly effective as formation components. Formation rules specify how morphemes are combined to form words in a parataxis way. For example, the Modifier-Head rule uses the first morpheme to modify the second morpheme. Following the study of Liu et al. (2018), we adopt 16 Chinese formation rules and show the top 5 in instance percentage in Table 1. Complete descriptions of 16 formation rules are provided in Appendix A. 3.2 Formation-informed dataset We construct a DG dataset under expert annotations, which contains morphemes and formation rules. Each entry consists of (1) source word, (2) morphemes and morpheme definitions, (3) formation rule, (4) context (a sentence containing the source word), (5) source word definition. extract data from the 5 th edition of the Contemporary Chinese Dictionary published by the Commercial Press 2 , one of the most influential Chinese dictionaries. We collect 45,311 Chinese disyllabic word entries with contexts and definitions. To annotate them, we also collect 10,527 Chinese characters and 20,855 morphemes with definitions. Our annotators include two professors and six graduates major in Chinese linguistics. Given the definition, they annotate each word with its formation rule (as shown in Table 1) and morpheme IDs (as shown in Table 2). Each entry is cross-validated by three independent annotators and reviewed by one. The detailed annotation process includes the following three steps: (1) Equipped with the definition, annotators annotate each entry with two morpheme IDs (select from the morphemes of each character) and a formation rule (select from 16 formation rules). Each entry is independently annotated by three annotators, who also note down a confidence score. If three annotations are the same, turn to (3); otherwise, turn to (2). (2) Another annotator reviews the conflicting annotations and confidence scores, and decides the final annotation. Turn to (3). (3) The annotation is collected as an entry into the final dataset. It takes one minute on average for each annotator to annotate an entry. Only 8,193 out of 45,311 entries enter Phase (2) in the whole process. We extend the DG setting in Ishiwatari et al. (2019) to incorporate the word formation features, F = { morph 1 , morph 2 , rule } , where morph i is the i th morpheme definition sentence and rule is the formation rule. The training goal is to maximize the likelihood of the ground-truth definition D = d 1: T given the source word w  , the context sentence C = c 1: n , and the word formation features F : p ( D | w  , C, F ) = T (cid:2) t =1 p ( d t | d i<t , w  , C, F ) . Our optimization objective is to minimize the cross-entropy loss L : L =  T (cid:3) t =1 log (cid:4) p ( d t | d i<t , w  , C, F ) (cid:5) , where d 1: T is the ground-truth definition, w  is the pretrained embedding of the source word, C is the context sentence, F is the formation information. As shown in Figure 2, DeFT first produces a seed vector in a rule-specific manner as global supervision. Then we feed it into the definition generator, which uses a gating mechanism to dynamically fuse different features and generate definitions. We first employ a Bi-LSTM (Graves and Schmidhu-ber, 2005) to encode morph i . Then, we combine morph i into a comprehensive morpheme embedding r m with a rule-specific linear layer, which captures different semantic relations: m i = Bi-LSTM ([ morph i ]) , r m = W ( rule ) m [ m 1 ; m 2 ] + b ( rule ) m . We then use a linear layer to combine r m and the pretrained source word embedding w  to obtain the seed vector r  as the initial generator input: r  = W r [ r m ; w  ] + b r . We employ an LSTM followed by a GRU-like (Cho et al., 2014) gate GRU-GATE (  ) , which dynamically fuses different features, as the generator: h t = LSTM ( d t  1 , h (cid:3) t  1 ) , h (cid:3) t = GRU-GATE ( h t , feat t ) , feat t = [ r m ; w  ; a  ; g t ; c t ] , where h t is the LSTM hidden state at the t th step, h (cid:3) t is the gated hidden state, d t  1 is the embedding of the previous definition word, specially, d 0 (cid:2) r  , and feat t denotes the features that dynamically control the generation process. We explain a  , g t , and c t as follows. a  is the character-level embedding, obtained by combining the embedding ch i of each character in w  with a rule-specific linear layer: a  = W ( rule ) a [ ch 1 ; ch 2 ] + b ( rule ) a . g t is the gated attended morpheme vector that dynamically focuses on the most relevant parts in morphemes during generation. We first calculate attended morpheme vectors g (cid:3) t,i by the attention mechanism (Bahdanau et al., 2015): g (cid:3) t,i = Attention ( h t , morph i ) , where Attention ( h , seq ) denotes the function that uses h to attend over the Bi-LSTM encoded seq . We then design a MorphGATE to compute g t by assigning different weights to two morphemes: z t =  ( W z [ g (cid:3) t, 1 ; g (cid:3) t, 2 ; h t ] + b z ) , g t = ( 1  z t ) (cid:2) g (cid:3) t, 1 + z t (cid:2) g (cid:3) t, 2 , 5527 Split #Words #Entries ContextLength Morph 1 Length Morph 2 Length DefinitionLength Train 29,169 36,248 7.22 7.69 7.29 12.02 Valid 3,673 4,531 7.32 7.45 7.30 11.91 Test 3,666 4,532 7.26 7.51 7.01 12.03 Table 3: Statistics of our formation-informed dataset. The inter-rater kappa (Fleiss and Cohen, 1973) is 0.65 for coverage and 0.66 for overall. We average scores of raters and obtain consistent results with 5528 Point Coverage Overall 1 Nothing is covered. where  (  ) is Sigmoid and (cid:2) is Hadamard product. c t is the attended context vector. Following Ishiwatari et al. (2019), we take c t = Attention ( h t , C ) as a feature since it may assist disambiguation. Finally, GRU-GATE ( h t , feat t ) takes the LSTM hidden state h t and the dynamically controlled features feat t as input, and updates h t to h (cid:3) t by fusing different features: u t =  ( W u [ h t ; feat t ] + b u ) , v t =  ( W r [ h t ; feat t ] + b r ) , h t = tanh ( W h [( v t (cid:2) feat t ); h t ] + b h ) , h (cid:3) t = u t (cid:2) h t + ( 1  u t ) (cid:2) h t , where  denotes the Sigmoid and (cid:2) denotes the Hadamard product. The gate u t controls how much the original state h t is remained, and the gate v t controls the contribution from features feat t . Dataset: We split the dataset described in Section 3 into training, validation and test sets by 8:1:1, as shown in Table 3. Note that we treat polysemous words as different entries, and the words are mutually exclusive across three sets. Hyper-parameters: We tune hyper-parameters to achieve the best BLEU score on the validation set. We use Adam (Kingma and Ba, 2015) with an initial learning rate of 10  3 as the optimizer. We set hidden size to 300, batch size to 64 and dropout rate to 0.2. Word embeddings are 300-dimensional, pretrained by fastText (Bojanowski et al., 2017). We train for up to 50 epochs, and early stop the training process once the performance does not improve for 10 consecutive epochs. We run our experiments on a single NVIDIA GeForce GTX 2080Ti GPU with 11 GB memory. us but using different features, including SG (No-raset et al., 2017) that uses only the word feature, and LOG-CaD (Ishiwatari et al., 2019) that uses both the word and context features. We conduct both automatic and human evaluations to validate our method, and show results in Table 4. For automatic evaluation, we select BLEU-4 (Bahdanau et al., 2015) and ROUGE-L (Lin, 2004) as metrics. We find that (1) our formation-informed DeFT (F+W+C) significantly outperforms baselines and other simplified versions (F, F+W, F+C); (2) based on W or W+C, adding formation features introduces significant improvement; (3) formation features are robust, since using only F can outperform LOG-CaD by 9.8% and 10.45% in BLEU and ROUGE-L, respectively. These findings validate that formation features can effectively enhance DG by assisting word meaning construction. For human evaluation, we measure semantic coverage and overall quality. The coverage metric measures how much ground-truth information is mentioned in the predicted definition. To be specific, the scores are given based on how many semantic aspects in the ground truth definition are covered by the predicted definition. The overall metric measures the overall quality of the predicted definition, referencing the ground-truth definition. We randomly select 100 entries from the test set, and hire three raters to rate the predicted definitions on a scale of 1 to 5, where each entry includes (1) the source word, (2) the ground-truth definition, and (3) the predicted definition to the raters. We show in Table 5 the detailed guideline for raters on each point. the automatic evaluation: formation features are effective and DeFT performs the best. Ablation study: Based on DeFT, we perform ablation study regarding MorphGATE and the formation rule in Table 6. (1) For MorphGATE, we replace it with a simple average function, which leads to a drop in performance. This reveals that different morphemes take effect in different generation phases. (2) For formation rule, we replace the rule-specific layers with a rule-shared layer, leading to a more serious performance drop. This verifies that distinguishing the specific formation rule can assist word meaning construction. Formation features can assist disambiguation: We present the generated definitions for a polysemous word in Figure 3. The example shows that using only the word feature (W) cannot distinguish different meanings. By contrast, using only the formation features (F) can capture the meaning difference and disambiguate the word ( use vs. money ). Further, DeFT (F+W+C) generates the exactly correct definition by fusing different features. Due to space limits, we put two additional interesting analyses on formation rules in Appendix B. Formation features are more feasible and effective compared with sememes: Sememes are expert-crafted words to describe the semantic aspects of words. For annotation cost, annotating sememes is as expensive as writing definitions (Li et al., 2020), whereas annotating formation features is a simple multiple-choice task with 1.98 choices on average. For effectiveness, we conduct experiments using sememe embeddings from Yang et al. (2020) as additional features. Results show that, based on W, adding sememes brings a BLEU improvement of 0.52, lower than that of 2.30 from F+W. Further, based on DeFT, adding sememes even brings noises and decreases BLEU by 0.35. This indicates that, compared with sememes, formation features are more feasible and effective. In this paper, we propose to use formation features to enhance DG. We build a formation-informed dataset and design a model DeFT, which decomposes words into formation features and fuses features via a gating mechanism. Experimental results show that our method is both effective and robust. We would like to thank all the anonymous reviewers and annotators for their helpful advice on various aspects of this work, including Fuqian Wu, Ming Liu, Yaqi Yin, Yue Wang, etc. This paper is supported by the National Natural Science Foundation of China (No. 62036001, U19A2065) and the National Social Science Foundation of China (No. 16YY137, 18ZDA295). 5529 References Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 4. Figure 4 (a) shows that Overlapping is most similar to Suffixation, Prefixation, and Single Morpheme. Interestingly, word meanings constructed 5530 Word Formation Rule Explanation Use Case %   dng  zhong (Modifier-Head) morph 1 modifies morph 2 (noun).
objective abstain abstain objective objective result abstain abstain abstain abstain abstain abstain other abstain abstain abstain objective objective method objective objective result other other other other other other other other method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other method abstain method method method method method abstain abstain method abstain abstain method method abstain result abstain abstain method other abstain result abstain result abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method result other other other other other other