Constructing Semantic Hierarchies via Fusion Learning Architecture
Semantic hierarchies construction means to build structure of concepts linked by hypernym-hyponym (“is-a”) relations. A major challenge for this task is the automatic discovery of hypernym-hyponym (“is-a”) relations. We propose a fusion learning architecture based on word embeddings for constructing semantic hierarchies, composed of discriminative generative fusion architecture and a very simple lexical structure rule for assisting, getting an F1-score of 74.20% with 91.60% precision-value, outperforming the stateof-the-art methods on a manually labeled test dataset. Subsequently, combining our method with manually-built hierarchies can further improve F1-score to 82.01%. Besides, the fusion learning architecture is language-independent.
1 Introduction
Ontologies and semantic thesauri (Miller, 1995; Suchanek et al., 2007) are significant for many natural language processing applications. The main components of ontologies and semantic thesauri are semantic hierarchies (see in Figure 1). In the WordNet, semantic hierarchies are organized in the form of “is-a” relations. For instance, the words “dog” and “canine” have such relation, and we call “canine” is a hypernym of “dog”. Conversely, “dog” is a hyponym of “canine”. The hypernym-hyponym (“is-a”) relation is the main relationship in semantic hierarchies. However, such manual semantic hierarchies construction as WordNet (Miller, 1995) and YAGO (Suchanek et al., 2007), the primary problem is the tradeoff between coverage scope and human labor. A number of papers have proposed some approach
to extract semantic hierarchies automatically.
hypernym-hyponym relation discovery is the key point of semantic hierarchies construction, also the major challenge. The usage of the context is a bottleneck in improving performance of hypernym-hyponym relation discovery. Several works focus on designing or learning lexical patterns (Hearst, 1992; Snow et al., 2004) via observing context of hypernym-hyponym relation, which suffer from covering a small proportion of complex linguistic circumstances. Besides, distributional inclusion hypothesis, which states that hypernyms tend to occur in a superset of contexts in which their hyponyms are found. In other words, hypernyms are semantically broader terms than their hyponyms (Kotlerman et al., 2010; Lenci and Benotto, 2012). However, it is not always rational. To acquire more contexts of words, Fu (2013) applies a web mining method to discover the hypernyms of Chinese entities from multiple sources, assuming that the hypernyms of an entity co-occur with it frequently. The method works well for named entities. But for class names with wider range of meanings, this assumption may fail.
Word embedding is a kind of low-dimensional
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
and dense real-valued vector encoding context information. Inspired by Mikolov (2013b) who founded that word embeddings can capture a considerable amount of syntactic/semantic relations, and found that hypernym-hyponym relations are complicated and a uniform linear projection cannot fit all of the hypernym-hyponym word pairs, Fu (2014) proposed an architecture for learning semantic hierarchies via word embeddings with clustered hypernym-hyponym relation word pairs in advanced. But the method just focuses on linear transformation of word embeddings, using shallow level semantic of the representation. Besides, the method need clustering for hypernymhyponym relation word pairs in advanced.
Since word embeddings can capture a considerable amount of syntactic/semantic relations (Mikolov et al., 2013b), we considered constructing a uniform architecture for semantic hierarchies learning based on nonlinear transformation of word embeddings. Inspired by advantages of discriminative model and generative model (see in Section 3), we fuse the two kind of models into one architecture. Considering word embeddings encode context information but ignore the lexical structures which contain some degree of semantic information, we integrate a very simple lexical structure rule into the previous fusion architecture aiming at building semantic hierarchies construction (see in Section 3.4).
For evaluation, the experimental results show that our method achieves an F1-score of 74.20% which outperforms the previous state-of-the-art methods. Moreover, and gets a much higher precision-value of 91.60%. Combining our method with the manually-built hierarchy can further improve F-score to 82.01%. The main contributions of our work are as follows:
• We present a uniform fusion architecture which can learn semantic hierarchies via word embeddings without any background knowledge.
• The method we proposed outperforms the state-of-the-art methods on a manually labeled test dataset especially with a good enough precision-value for application.
• The fusion learning architecture is languageindependent which can be easily expanded to be suitable for other languages.
2 Related Work
During the early phase of semantic hierarchies study, some focused on building manually-built semantic resources, WordNet (Miller, 1995) is a representative thesauri among them. Such manuallybuilt hierarchies have exact structure and high accuracy, but their coverage is limited, especially for fine-grained concepts and entities. Some researchers presented automatic approaches for supplementing manually-built semantic resources. Suchanek et al. (2007) linked the categories in Wikipedia onto WordNet in construction of YAGO. However, the coverage is still limited by the scope of Wikipedia.
The major challenge for building semantic hierarchies is the discovery of hypernym-hyponym relations automatically. The usage of the context is a bottleneck in improving performance of discovery of hypernym-hyponym relations. Some researchers proposed method based on lexical pattern abstracted from context manually or automatically to mine hypernym-hyponym relations. Hearst (1992) pointed out that certain lexical constructions linking two different noun phases (NPs) often imply hypernym-hyponym relation. A representative example is “such NP1 as NP2”. Considering time-consuming of manuallybuilt lexical patterns, Snow et al. (2004) proposed a automatic method extracting large numbers of lexico-syntactic patterns to detect hypernym relations from a large newswire corpus. But the method suffers from semantic drift of auto-extracted patterns. Generally speaking, these pattern-based methods often suffer from low recall-value or precision-value because of the coverage and quality of extracted patterns.
Some measures rely on the assumption that hypernyms are semantically broader terms than their hyponyms. The assumption is a variation of the Distributional Inclusion Hypothesis (Geffet and Dagan, 2005; Zhitomirsky-Geffet and Dagan, 2009). The pioneer work by Kotlerman et al. (2010) designed a directional distributional measure to infer hypernym–hyponym relations. Differently from Kotlerman et al. (2010), Lenci and Benotto (2012) focus on applying directional, asymmetric similarity measures to identify hypernyms. However the broader semantics hypothesis may not always infer broader contexts (Fu et al., 2014).
Considering the behavior of a person explor-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
ing the meaning of an unknown entity, Fu (2013) applies a web mining method to discover the hypernyms of Chinese entities from multiple sources. The assumption is that the hypernyms will co-occur with its hyponym entity frequently. But the assumption maybe failed when involved with concept words which have boarder semantic compared with entities. Inspired by the fact (Mikolov et al., 2013b) that word embeddings can capture a considerable amount of syntactic/semantic relations(e.g. v(king) - v(queen) ≈ v(man) - v(woman), where v(w) is the word embedding of the word w), Fu (2014) present an approach to learn semantic hierarchies with clustered hypernym-hyponym relation word embedding pairs. However the method just focuses on linear transformation of word embeddings, using shallow level semantic of the representation. Besides, the method need clustering for hypernym-hyponym relation word pairs in advanced and the precision-value on test data is not good enough for practical application. Shwartz et al. (2016) included additional linguistic information for LSTM-based learning, but the method has co-occurrence requirements for hyponym-hypernym pairs in corpus.
Enlightened from good properties of word embeddings for capturing semantic relationship between words in work of Fu (2014), we further explore capacity of word embedding for semantic hierarchies using neural networks based on a fusion learning architecture. Above all, the method we proposed do not need clustering for hypernymhyponym relation word pairs in advanced.
3 Method
Given the hypernyms list of a word, our goal is building a semantic hierarchies construction of these hypernyms and the given word(Fu et al., 2014), the process is presented in Figure 2.
The whole information all we have to figure out whether there exists a hypernym-hyponym relation for the word pair is word embeddings representation. Two major kind of architectures for such problem in general, one is discriminative and another is generative.
Discriminative Architecture: Discriminative architecture regards the discovery of hypernymhypernym relation as a classification task, the process of learning semantic hierarchies equal to classify word pair into yes or no for whether exists
乌头 aconite
药品 medicine
植物 plant
毛茛科 Ranunculaceae
植物药 medicine plant
乌头属 Aconitum
生物 organism
乌头 aconite
植物药 medicine plant 乌头属
Aconitum
药品 medicine 毛茛科
Ranunculaceae
植物 plant
生物 organism
Figure 2: An example of learning semantic hierarchies
hypernym-hypernym relation. Generative Architecture: Generative architecture focus on generating the hypernym of a given hyponym directly. Direct generation is usual impractical, so generative method produces a fake target which is very similar with the true one.
We consider fusing these two models to discovery hypernym-hyponym relation much more precisely. We use Multilayer Perceptron (MLP) (Rosenblatt, 1961) to achieve a generative model and Recurrent Neural Network (RNN) (Mikolov et al., 2010) to implement a discriminative model.
3.1 Word Embedding Training
Various methods for learning word embeddings have been proposed in the recent years, such as neural net language models (Bengio et al., 2003; Mnih and Hinton, 2009; Mikolov et al., 2013b) and spectral models (Dhillon et al., 2011). More recently, Mikolov et al. (2013a) propose two loglinear models, namely the Skip-gram and CBOW model, for inducing word embeddings efficiently on a large-scale corpus because of their low time complexity. Additionally, their experiment results have shown that the Skip-gram model performs best in identifying semantic relationship among words. For this reason, we employ the Skip-gram model for estimating word embeddings in this study.
3.2 Generative Architecture Based on MLP
Multilayer Perceptron (MLP) (Rosenblatt, 1961) is a feedforward artificial neural network which maps inputs onto a set of appropriate outputs. An MLP consists of multiple layers connecting each layer fully connected to the next one. Except for the input nodes, each node is a neuron with a nonlinear activation function(e.g. sigmoid). MLP is a modification of the standard linear perceptron and
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
can distinguish data that are not linearly separable (Cybenko, 1989).
For single hidden layer MLP, there are input layer x, hidden layer h and output layer y range from bottom to top. The value of neurons in each layer is a nonlinear projection of the previous layer. Formalization denotation in formulas are as follows:
h = fh(Whx+ bh) (1)
y = fy(Wyh+ by) (2)
Where Wh,Wy are matrices for linear projection. And fh, fy are nonlinear activation functions for nonlinear transformation.
In our work, we use Multilayer Perceptron as the main component for generative architecture. The inputs of MLP is word embedding representation of hyponym and outputs a fake hypernym embedding which is very similar with the true hypernym vector. The model produces final result by calculating the distance between the fake hypernym and the candidate hypernym word in continuous space, subsequently, comparing the distance and a predefined threshold to give a judgment. By adjusting the threshold value of similarity, we expect MLP model obtain much higher precision compared with the discriminative one.
3.3 Discriminative Architecture Based on RNN
Recurrent Neural Network (RNN) (Mikolov et al., 2010) is a kind of artificial neural network in which connections between neuron form a directed cycle, creating an internal state of the network which allows it to exhibit dynamic temporal behavior according to the history information. Unlike feedforward neural networks (e.g. MLP), RNNs can use their internal memory to process future inputs. The features of RNN makes them applicable to tasks such as unsegmented connected handwriting or speech recognition. There are some variants of original RNN, the most representative one of them is Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) which is capable of learning long-term dependencies. In this paper we use the original simple recurrent networks (SRN). There are two major classes of SRN, know as Elman networks (Elman, 1990) and Jordan networks (Jordan, 1997). In this paper, we use Elman networks as the main component for discriminative architecture (see in Figure 3).
𝑥
𝑦
ℎ
𝑉
𝑈
𝑊
Unfold
𝑥𝑡−1
𝑦𝑡−1
ℎ𝑡−1 𝑉
𝑊 𝑊
𝑥𝑡
𝑦𝑡
ℎ𝑡 𝑉
𝑊
𝑥𝑡+1
𝑦𝑡+1
ℎ𝑡+1 𝑉
𝑊
Figure 3: Recurrent Neural Network (Elman networks)
In Elman networks, the main architecture is composed of three classes layers, namely, input layer x, hidden layer h, and output layer y. Different from feedforward neural networks, the hidden layer ht is depended on the previous time step hidden layer ht−1 and the current time step input layer xt−1. And the output layer yt is update by the current time step hidden layer ht . Formalization denotation in formulas are as follows:
ht = fh(Uxt +Wht−1 + bh) (3)
yt = fy(V ht + by) (4)
Where U, V,W are matrices for linear projection. And fh, fy are nonlinear activation functions for nonlinear transformation.
The inputs of RNN is word embeddings representation of hyponym and candidate hypernym sequence. We regard the hyponym and candidate hypernym as a sequence, because that the judgment of candidate hypernym is depended on hyponym in discrimination process. Ignoring the outputs during recurrent process, we take the final output of the last input in the sequence as the result of discrimination.
3.4 Fusion Learning Architecture Combined with a Simple Lexical Structure Rule
Generative architecture can get a very high precision by adjusting the threshold value of similarity, but will pay a high price for low recall-value. Compared with generative method, discriminative architecture can obtain a higher recall-value with low guarantee for precision-value.
The feature of discriminative architecture indicates that if it determines a candidate hypernymhyponym relation word pair as negative, then the word pair will have high probability for negative. We can use discriminative architecture to help the generative one to get rid of some false positive instance. For this reason, we fuse the generative and
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Discriminative Architecture ( RNN )
Generative Architecture ( based on MLP )
distance ∧ ∨
洲际 弹道 导弹 Intercontinental Ballistic Missile
Lexical Structure Rule
=
: hyponym
: candidate hypernym
: similar with true hypernym
: Boolean value. 0 for unrelated; 1 for hypernym-hyponym relation.
Figure 4: Fusion Learning Architecture.
discriminative architectures together by applying Boolean operator “AND” to the results outputted by the two architectures. Excepting a much higher precision-value than the precious two models and almost the same recall-value as the generative one.
By combining discriminative and generative architectures, the fusion architecture can discovery hypernym-hyponym much more precisely but becomes only focusing deep level semantic and ignoring the lexical stricture information which is very useful for discovery of hypernym-hyponym relationship, especially for compound nouns (CNs), for instance, “洲际 弹道导弹(Intercontinental Ballistic Missile)”. The root word of CN often indicates a hypernym relation, like the word “导 弹(Missile)” is the hypernyms of the precious CN. Root word of a CN can be obtained via using syntax dependency parsing or semantic dependency parsing of CN. Due to the word formation rule of Chinese, the root word is usually the last word in CN segmentation result. To supplement the capacity of learning semantic hierarchy from lexical structure, we use the simple lexical structure rule to assist previous fusion model.
The final fusion learning architecture (showed in Figure 4) is composed of three parts, namely generative architecture, discriminative architecture and lexical structure rule module.
4 Experiments
In the experimental stage, we implement our fusion architecture for learning semantic hierarchies. To the end of this, we first introduce the preparation of experimental setup. Next, we report the
performance of fusion architecture and its components. Subsequently, we compare the performance of our method to those of several previous methods in different aspects and give an example for construction of semantic hierarchies.
4.1 Experimental Setup
Pre-trained Word Embeddings
We use a Chinese encyclopedia corpus named Baidubaike1 to learn word embeddings, which contains about 30 million sentences (about 780 million words). The Chinese segmentation technology is provided by the open-source Chinese language processing platform LTP2 (Che et al., 2010). Then, we employ the Skip-gram method (Section 3.1) to train word embeddings for the further experiment. We obtain the embedding vectors of 0.56 million words in total.
Dataset and Evaluation Metrics
The training data for learning semantic hierarchies is collected from CilinE3 which contains 100,093 Chinese words and organized as a hierarchy of five levels, in which the words are linked by hypernym–hyponym relations. Finally, we obtain 15,242 word pairs of hypernym–hyponym relation for positive instances and constructed 15,242 negative instances for training.
For comparability we use the same test dataset as Fu et al. (2014) in evaluation stage. They obtain the hypernyms for 418 entities, which are se-
1Baidubaike (https://baike.baidu.com/) is one of the largest Chinese encyclopedias.
2http://www.ltp-cloud.com/demo/ 3http://www.ltp-cloud.com/download/
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Word Dimension
# of neutrons in hidden layer(RNN)
# of neutrons in hidden layer(MLP)
Batch size Adadelta parameter
dw = 300 nh = 800 nh = 500 b = 20 ρ = 0.95, ε = 1e −6
Table 1: Parameters used in our experiments.
lected randomly from Baidubaike, following their previous work (Fu et al., 2013). The final data set was manually labeled and measured the interannotator agreement by using the kappa coefficient (Siegel and Castellan, 1981). The kappa value is 0.96, which indicates a good strength of agreement. Training data and test data are showed in Table 2.
Relation # of word pairsTraining Test hypernym-hyponym 15,242 1,079 hyponym-hypernym 7,621 1,079 unrelated 7,621 3,250 Total 30,484 5,408
Table 2: The Experimental Data.
We use precision-value, recall-value, and F1score as metrics to evaluate the performances of the methods. Since the discovery of hypernymhyponym relation is a binary classification task, we only report the performance of the positive instances recognition in the experiments.
Parameter Settings and Training
In our fusion architecture, there are MLP for generation (see in Section 3.2) and RNN for discrimination (see in Section 3.3) need to be trained. We experimentally study the effects of several hyperparameters on this two neural networks: the number of neutrons in hidden layer, the selection of activation function. Table 1 shows all parameters used in the experiments. We use Adadelta (Zeiler, 2012) in the update procedure, which relies on two main parameters, ρ and ε, which do not significantly affect the performance. Following Zeiler (2012), we choose 0.95 and 1e−6, respectively, as the values of these two parameters.
In the training stage, we train the discriminative architecture and generative architecture respectively. For training discriminative architecture based on RNN, we use the whole training data. But for training generative architecture based on MLP, we only use the positive instances in train-
ing data as Fu (2014) for generating positive hypernym vectors.
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 Distance Threshold
63.20%
63.60%
64.00%
64.40%
64.80%
65.20%
65.60%
0.5 0.6 0.7 0.8 0.9 1
Threshold for Probability
(a) generative architecture (b) discriminative architecture
Figure 5: The effects on F1-score of threshold for generative and discriminative architecture.
70.00%
70.50%
71.00%
71.50%
72.00%
72.50%
73.00%
73.50%
74.00%
74.50%
0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1
F1
Threshold for Probability
Figure 6: The effects on F1-score of threshold for fusion architecture.
4.2 Performance of Fusion Architecture
In the test stage, we tune the performance of architecture by changing the thresholds for components of fusion architecture. There two kinds of thresholds need to be tuned: the similar distance of generative architecture and the probability for positive instances of discriminative architecture.
We first experimentally study the effects of semantic distance threshold on generative architecture based on MLP neural network. Selecting the semantic distance threshold of best F1-score for generative architecture and fixing it, we tun the probability threshold for positive instances of discriminative architecture, observing the F1-score
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
curve of discriminative architecture and the fusion learning architecture (see in Figure 5, Figure 6).
4.3 Comparison with Previous Work
In this section, we compare the proposed method with previous methods, including pairwise hypernym-hyponym relation extraction based on patterns, word distributions, web mining, and based on word embeddings(see in Section 2). Results are shown in Table 3.
Method P(%) R(%) F1(%) MPttern 97.47 21.41 35.11 MSnow 60.88 25.67 36.11 MbalApinc 54.96 53.38 54.16 MinvCL 49.63 62.84 55.46 MWeb 87.40 48.19 62.13 MEmb 80.54 67.99 73.74 MlexicalRule 100.0 16.88 28.88 MMLPgen. 77.96 53.51 63.46 MRNNdis. 55.97 77.40 64.96 MMLP+RNN 90.00 51.48 65.50 MFusion 91.60 62.36 74.20
Table 3: Comparison of the proposed method with existing methods in the test set.
Overall Comparison
MPattern refers to the pattern-based method (Hearst, 1992). The method uses the Chinese Hearst-style patterns (Fu et al., 2013). The result shows that only a small part of the hypernyms can be extracted based on these patterns because only a few hypernym relations are expressed in the fixed patterns, and most of them are expressed in highly flexible manners. MSnow originally proposed by Snow et al. (2004). This method relies on an accurate syntactic parser, and the quality of the automatically extracted patterns is difficult to guarantee.
There are two previous distributional methods MbalApinc (Kotlerman et al., 2010) and MinvCL (Lenci and Benotto, 2012). Each word is represented as a feature vector in which each dimension is the point-wise mutual information (PMI) value (Geffet and Dagan, 2005; ZhitomirskyGeffet and Dagan, 2009) of the word and its context words (see in Section 2). MFu refers to a web mining method proposed by Fu et al. (2013) which mines hypernyms of a given word w from multiple sources returning a
ranked list of the hypernyms. MEmb refers to a novel method based on word embeddings achieving the best F1-value among previous methods. MlexicalRule refers to using lexical structure rule to discover hypernym-hyponym relations (see in Section 3.4). The 100% precision-value on test data indicates the lexical rule is correct for most compound nouns (CNs). However the rule only takes effect for CNs which are minority. MMLPgen. represents the generative architecture based on MLP neural network (see in Section 3.2), the method get a higher F1-score than most of previous semantic hierarchy discovery method except MEmb. MRNNdis. represents the discriminative architecture based on RNN neural network (see in Section 3.3) which obtains the highest recall-value in comparison of the proposed methods. MMLP+RNN combines these two architectures based on MLP and RNN (see in Section 3.4), getting a much higher precision-value than any components and a comparable recall-value with MMLPgen. . MFusion refers to a fusion learning architecture composed of discriminative and generative architectures and assisted with lexical structure rule (see in Section 3.4). Assisted with the simple lexical structure rule, the fusion learning architecture get a better F1-score than all of the previous methods do and significantly improve the precisionvalue over the state-of-the-art method MEmb.
Further, we experimentally study the effects of lexical rule for MEmb. The results show that the method MEmb+lexicalRule does not improve the F1-score compared with the original MEmb. The reason maybe that MEmb already recall the compound nouns (CNs) hypernym-hyponym relations (see in Table 4). The results also indicate that the improvement is mainly caused by discriminative generative architecture.
P(%) R(%) F1(%) MEmb+lexicalRule 80.54 67.99 73.74 MFusion 91.60 62.36 74.20
Table 4: Comparison of the MEmb assisted with lexical structure rule.
Comparison on the Out-of-CilinE Data
Since the training data is extracted from CilinE, we are greatly interested in the performance of our method on the hypernym-hyponym relations out-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
side of CilinE. We assume that as long as there is one word in the pair not existing in CilinE, the word pair is outside of CilinE. In our test data, about 61% word pairs are outside of CilinE.
P(%) R(%) F1(%) MWiki+CilinE 80.39 19.29 31.12 MEmb 65.85 44.47 53.09 MFusion 79.92 44.94 57.53
Table 5: Comparison on the Out-of-CilinE Data.
Table 5 shows the performances of the baseline method MWiki+CilinE , previous state-of-theart methodMEmb and our methodMFusion on the out-of-CilinE data. In comparison, MWiki+CilinE has the highest precision-value but has a lowest recall-value, MEmb significantly improve recallvalue and F1-score. By contrast, our method MFusion can discover a little bit more hypernym–hyponym relations than MEmb with achieving a more than 14% precision-value improvement. And our method can get an F1-score of 57.53%, which is a new state-of-the-art result on the Outof-CilinE Data.
Combined with Manually-Built Hierarchies
For further exploration, we combine our method MFusion with the existing manually-built hierarchies in Wikipedia and CilinE. The combination strategy is to simply merge all positive results from the two methods together, and then to infer new relations based on the transitivity of hypernym–hyponym relations. The same manner allied to precious method MEmb to be compared. The comparison is showed in Table 6. Combining our fusion method MFusion with manually-built hierarchies Wikipedia and CilinE can further improve F1-score to 82.01%, gets an about 1.7% improvement compared with the same manners on MEmb.
P(%) R(%) F1(%) MEmb 80.54 67.99 73.74 MFusion 91.60 62.36 74.20 MEmd+CilinE 80.59 72.42 76.29 MFusion+CilinE 91.64 70.76 79.85 MEmd+Wiki+CilinE 79.78 80.81 80.29 MFusion+Wiki+CilinE 91.00 74.63 82.01
Table 6: Comparison ofMFusion andMEmb combined with manually-built hierarchies.
4.4 Example of Learning Semantic Hierarchies
In Figure 7, there is an example of learning semantic hierarchies based on our fusion architecture (MFusion) and combined method using manuallybuilt hierarchies (MFusion+Wiki+CilinE). From the results, we can see that our method can actually learn the semantic hierarchies for a given word and its hypernyms list relatively precisely. NBA球员 NBA player 篮球运动员 basketball player 球员 ball player 运动员 athlete 体育人物 sportsman 人物 person
(b) 𝑀𝐹𝑢𝑠𝑖𝑜𝑛+𝑊𝐼𝑘𝑖+𝐶𝑖𝑙𝑖𝑛𝐸
NBA球员 NBA player
篮球运动员 basketball play r
球员 ball player
运动员 athlete
体育人物 sportsman
人物 person
(a) 𝑀𝐹𝑢𝑠𝑖𝑜𝑛
乌头 aconite
药品 medicine
植物 plant
毛茛科 Ranunculaceae
植物药 medicine plant 乌头属
Aconitum
生物 organism
(b) 𝑀𝐹𝑢𝑠𝑖𝑜𝑛+𝑊𝐼𝑘𝑖+𝐶𝑖𝑙𝑖𝑛𝐸
乌头 aconite
药品 medicine
植物 plant
毛茛科 Ranunculaceae
植物药 medicine plant 乌头属
Aconitum
生物 organism
(a) 𝑀𝐹𝑢𝑠𝑖𝑜𝑛
Hyponym List of Hypernyms
NBA球员 (NBA player) 球员 (ball player) 篮球运动员 (basketball player) 运动员 (athlete) 体育人物 (sportsman) 人物 (person)
Hyponym List of Hypernyms
乌头 (conite) 植物药 (medicine plant) 乌头属 (Aconitum) 药品 (medicine) 毛茛科 (Ranunculaceae) 植物 (plant) 生物 (organism)
Figure 7: Example of Learning Semantic Hierarchies.
The dashed line frames in Figure 7(a) refers to the losing hypernym-hyponym relations words. For instance, our method fail to learn the semantic hierarchies between “乌头属(aconite)” and “毛茛科(Ranunculaceae)”. The reason maybe that their semantic similarity effects representations close to each other in the embedding space and our method can not find suitable projection for these pairs. By combining our method with manually-built hierarchies, we can improve the capacity of learning semantic hierarchies. In this case, the combined method can build the semantic hierarchies correctly (see in Figure 7(b)).
5 Conclusion
This paper proposes a novel method for learning semantic hierarchies based on discriminative generative fusion architecture combined with a very simple lexical structure rule. The fusion architecture method can be easily expanded to be suitable for other languages. In experiments, the proposed method achieves the best F1-score of 74.20% on a manually labeled test dataset outperforming stateof-the-art methods with a much higher precisionvalue of 91.60% for application. Further experiments show that our method is complementary with some manually-built hierarchies to learn semantic hierarchy construction more precisely.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
- Strengths: - The paper tackles an important issue, that is building ontologies or thesauri - The methods make sense and seem well chosen - Methods and setups are well detailed - It looks like the authors outperform the state-of-the-art approach (but see below for my concerns)  - Weaknesses: The main weaknesses for me are evaluation and overall presentation writing. - The list of baselines is hard to understand. Some methods are really old and it doesn't seem justified to show them here (e.g., Mpttern). - Memb is apparently the previous state-of-the-art, but there is no mention to any reference. - While it looks like the method outperforms the previous best performing approach, the paper is not convincing enough. Especially, on the first dataset, the difference between the new system and the previous state-of-the-art one is pretty small. - The paper seriously lacks proofreading, and could not be published until this is fixed   for instance, u00a0I noted 11 errors in the first column of page 2. - The CilinE hierarchy is very shallow (5 levels only). However apparently, it has been used in the past by other authors. I would expect that the deeper the more difficult it is to branch new hyponym-hypernyms. This can explain the very high results obtained (even by previous studies)...  - General Discussion: The approach itself is not really original or novel, but it is applied to a problem that has not been addressed with deep learning yet. For this reason, I think this paper is interesting, but there are two main flaws. The first and easiest to fix is the presentation. There are many errors typos that need to be corrected. I started listing them to help, but there are just too many of them. The second issue is the evaluation, in my opinion. Technically, the performances are better, but it does not feel convincing as explained above. What is Memb, is it the method from Shwartz et al 2016, maybe? If not, what performance did this recent approach have? I think the authors need to reorganize the evaluation section, in order to properly list the baseline systems, clearly show the benefit of their approach and where the others fail. Significance tests  also seem necessary given the slight improvement on one dataset.