Constructing Semantic Hierarchies via Fusion Learning Architecture
Semantic hierarchies construction means to build structure of concepts linked by hypernym-hyponym (“is-a”) relations. A major challenge for this task is the automatic discovery of hypernym-hyponym (“is-a”) relations. We propose a fusion learning architecture based on word embeddings for constructing semantic hierarchies, composed of discriminative generative fusion architecture and a very simple lexical structure rule for assisting, getting an F1-score of 74.20% with 91.60% precision-value, outperforming the stateof-the-art methods on a manually labeled test dataset. Subsequently, combining our method with manually-built hierarchies can further improve F1-score to 82.01%. Besides, the fusion learning architecture is language-independent.
1 Introduction
Ontologies and semantic thesauri (Miller, 1995; Suchanek et al., 2007) are significant for many natural language processing applications. The main components of ontologies and semantic thesauri are semantic hierarchies (see in Figure 1). In the WordNet, semantic hierarchies are organized in the form of “is-a” relations. For instance, the words “dog” and “canine” have such relation, and we call “canine” is a hypernym of “dog”. Conversely, “dog” is a hyponym of “canine”. The hypernym-hyponym (“is-a”) relation is the main relationship in semantic hierarchies. However, such manual semantic hierarchies construction as WordNet (Miller, 1995) and YAGO (Suchanek et al., 2007), the primary problem is the tradeoff between coverage scope and human labor. A number of papers have proposed some approach
to extract semantic hierarchies automatically.
hypernym-hyponym relation discovery is the key point of semantic hierarchies construction, also the major challenge. The usage of the context is a bottleneck in improving performance of hypernym-hyponym relation discovery. Several works focus on designing or learning lexical patterns (Hearst, 1992; Snow et al., 2004) via observing context of hypernym-hyponym relation, which suffer from covering a small proportion of complex linguistic circumstances. Besides, distributional inclusion hypothesis, which states that hypernyms tend to occur in a superset of contexts in which their hyponyms are found. In other words, hypernyms are semantically broader terms than their hyponyms (Kotlerman et al., 2010; Lenci and Benotto, 2012). However, it is not always rational. To acquire more contexts of words, Fu (2013) applies a web mining method to discover the hypernyms of Chinese entities from multiple sources, assuming that the hypernyms of an entity co-occur with it frequently. The method works well for named entities. But for class names with wider range of meanings, this assumption may fail.
Word embedding is a kind of low-dimensional
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
and dense real-valued vector encoding context information. Inspired by Mikolov (2013b) who founded that word embeddings can capture a considerable amount of syntactic/semantic relations, and found that hypernym-hyponym relations are complicated and a uniform linear projection cannot fit all of the hypernym-hyponym word pairs, Fu (2014) proposed an architecture for learning semantic hierarchies via word embeddings with clustered hypernym-hyponym relation word pairs in advanced. But the method just focuses on linear transformation of word embeddings, using shallow level semantic of the representation. Besides, the method need clustering for hypernymhyponym relation word pairs in advanced.
Since word embeddings can capture a considerable amount of syntactic/semantic relations (Mikolov et al., 2013b), we considered constructing a uniform architecture for semantic hierarchies learning based on nonlinear transformation of word embeddings. Inspired by advantages of discriminative model and generative model (see in Section 3), we fuse the two kind of models into one architecture. Considering word embeddings encode context information but ignore the lexical structures which contain some degree of semantic information, we integrate a very simple lexical structure rule into the previous fusion architecture aiming at building semantic hierarchies construction (see in Section 3.4).
For evaluation, the experimental results show that our method achieves an F1-score of 74.20% which outperforms the previous state-of-the-art methods. Moreover, and gets a much higher precision-value of 91.60%. Combining our method with the manually-built hierarchy can further improve F-score to 82.01%. The main contributions of our work are as follows:
• We present a uniform fusion architecture which can learn semantic hierarchies via word embeddings without any background knowledge.
• The method we proposed outperforms the state-of-the-art methods on a manually labeled test dataset especially with a good enough precision-value for application.
• The fusion learning architecture is languageindependent which can be easily expanded to be suitable for other languages.
2 Related Work
During the early phase of semantic hierarchies study, some focused on building manually-built semantic resources, WordNet (Miller, 1995) is a representative thesauri among them. Such manuallybuilt hierarchies have exact structure and high accuracy, but their coverage is limited, especially for fine-grained concepts and entities. Some researchers presented automatic approaches for supplementing manually-built semantic resources. Suchanek et al. (2007) linked the categories in Wikipedia onto WordNet in construction of YAGO. However, the coverage is still limited by the scope of Wikipedia.
The major challenge for building semantic hierarchies is the discovery of hypernym-hyponym relations automatically. The usage of the context is a bottleneck in improving performance of discovery of hypernym-hyponym relations. Some researchers proposed method based on lexical pattern abstracted from context manually or automatically to mine hypernym-hyponym relations. Hearst (1992) pointed out that certain lexical constructions linking two different noun phases (NPs) often imply hypernym-hyponym relation. A representative example is “such NP1 as NP2”. Considering time-consuming of manuallybuilt lexical patterns, Snow et al. (2004) proposed a automatic method extracting large numbers of lexico-syntactic patterns to detect hypernym relations from a large newswire corpus. But the method suffers from semantic drift of auto-extracted patterns. Generally speaking, these pattern-based methods often suffer from low recall-value or precision-value because of the coverage and quality of extracted patterns.
Some measures rely on the assumption that hypernyms are semantically broader terms than their hyponyms. The assumption is a variation of the Distributional Inclusion Hypothesis (Geffet and Dagan, 2005; Zhitomirsky-Geffet and Dagan, 2009). The pioneer work by Kotlerman et al. (2010) designed a directional distributional measure to infer hypernym–hyponym relations. Differently from Kotlerman et al. (2010), Lenci and Benotto (2012) focus on applying directional, asymmetric similarity measures to identify hypernyms. However the broader semantics hypothesis may not always infer broader contexts (Fu et al., 2014).
Considering the behavior of a person explor-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
ing the meaning of an unknown entity, Fu (2013) applies a web mining method to discover the hypernyms of Chinese entities from multiple sources. The assumption is that the hypernyms will co-occur with its hyponym entity frequently. But the assumption maybe failed when involved with concept words which have boarder semantic compared with entities. Inspired by the fact (Mikolov et al., 2013b) that word embeddings can capture a considerable amount of syntactic/semantic relations(e.g. v(king) - v(queen) ≈ v(man) - v(woman), where v(w) is the word embedding of the word w), Fu (2014) present an approach to learn semantic hierarchies with clustered hypernym-hyponym relation word embedding pairs. However the method just focuses on linear transformation of word embeddings, using shallow level semantic of the representation. Besides, the method need clustering for hypernym-hyponym relation word pairs in advanced and the precision-value on test data is not good enough for practical application. Shwartz et al. (2016) included additional linguistic information for LSTM-based learning, but the method has co-occurrence requirements for hyponym-hypernym pairs in corpus.
Enlightened from good properties of word embeddings for capturing semantic relationship between words in work of Fu (2014), we further explore capacity of word embedding for semantic hierarchies using neural networks based on a fusion learning architecture. Above all, the method we proposed do not need clustering for hypernymhyponym relation word pairs in advanced.
3 Method
Given the hypernyms list of a word, our goal is building a semantic hierarchies construction of these hypernyms and the given word(Fu et al., 2014), the process is presented in Figure 2.
The whole information all we have to figure out whether there exists a hypernym-hyponym relation for the word pair is word embeddings representation. Two major kind of architectures for such problem in general, one is discriminative and another is generative.
Discriminative Architecture: Discriminative architecture regards the discovery of hypernymhypernym relation as a classification task, the process of learning semantic hierarchies equal to classify word pair into yes or no for whether exists
乌头 aconite
药品 medicine
植物 plant
毛茛科 Ranunculaceae
植物药 medicine plant
乌头属 Aconitum
生物 organism
乌头 aconite
植物药 medicine plant 乌头属
Aconitum
药品 medicine 毛茛科
Ranunculaceae
植物 plant
生物 organism
Figure 2: An example of learning semantic hierarchies
hypernym-hypernym relation. Generative Architecture: Generative architecture focus on generating the hypernym of a given hyponym directly. Direct generation is usual impractical, so generative method produces a fake target which is very similar with the true one.
We consider fusing these two models to discovery hypernym-hyponym relation much more precisely. We use Multilayer Perceptron (MLP) (Rosenblatt, 1961) to achieve a generative model and Recurrent Neural Network (RNN) (Mikolov et al., 2010) to implement a discriminative model.
3.1 Word Embedding Training
Various methods for learning word embeddings have been proposed in the recent years, such as neural net language models (Bengio et al., 2003; Mnih and Hinton, 2009; Mikolov et al., 2013b) and spectral models (Dhillon et al., 2011). More recently, Mikolov et al. (2013a) propose two loglinear models, namely the Skip-gram and CBOW model, for inducing word embeddings efficiently on a large-scale corpus because of their low time complexity. Additionally, their experiment results have shown that the Skip-gram model performs best in identifying semantic relationship among words. For this reason, we employ the Skip-gram model for estimating word embeddings in this study.
3.2 Generative Architecture Based on MLP
Multilayer Perceptron (MLP) (Rosenblatt, 1961) is a feedforward artificial neural network which maps inputs onto a set of appropriate outputs. An MLP consists of multiple layers connecting each layer fully connected to the next one. Except for the input nodes, each node is a neuron with a nonlinear activation function(e.g. sigmoid). MLP is a modification of the standard linear perceptron and
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
can distinguish data that are not linearly separable (Cybenko, 1989).
For single hidden layer MLP, there are input layer x, hidden layer h and output layer y range from bottom to top. The value of neurons in each layer is a nonlinear projection of the previous layer. Formalization denotation in formulas are as follows:
h = fh(Whx+ bh) (1)
y = fy(Wyh+ by) (2)
Where Wh,Wy are matrices for linear projection. And fh, fy are nonlinear activation functions for nonlinear transformation.
In our work, we use Multilayer Perceptron as the main component for generative architecture. The inputs of MLP is word embedding representation of hyponym and outputs a fake hypernym embedding which is very similar with the true hypernym vector. The model produces final result by calculating the distance between the fake hypernym and the candidate hypernym word in continuous space, subsequently, comparing the distance and a predefined threshold to give a judgment. By adjusting the threshold value of similarity, we expect MLP model obtain much higher precision compared with the discriminative one.
3.3 Discriminative Architecture Based on RNN
Recurrent Neural Network (RNN) (Mikolov et al., 2010) is a kind of artificial neural network in which connections between neuron form a directed cycle, creating an internal state of the network which allows it to exhibit dynamic temporal behavior according to the history information. Unlike feedforward neural networks (e.g. MLP), RNNs can use their internal memory to process future inputs. The features of RNN makes them applicable to tasks such as unsegmented connected handwriting or speech recognition. There are some variants of original RNN, the most representative one of them is Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) which is capable of learning long-term dependencies. In this paper we use the original simple recurrent networks (SRN). There are two major classes of SRN, know as Elman networks (Elman, 1990) and Jordan networks (Jordan, 1997). In this paper, we use Elman networks as the main component for discriminative architecture (see in Figure 3).
𝑥
𝑦
ℎ
𝑉
𝑈
𝑊
Unfold
𝑥𝑡−1
𝑦𝑡−1
ℎ𝑡−1 𝑉
𝑊 𝑊
𝑥𝑡
𝑦𝑡
ℎ𝑡 𝑉
𝑊
𝑥𝑡+1
𝑦𝑡+1
ℎ𝑡+1 𝑉
𝑊
Figure 3: Recurrent Neural Network (Elman networks)
In Elman networks, the main architecture is composed of three classes layers, namely, input layer x, hidden layer h, and output layer y. Different from feedforward neural networks, the hidden layer ht is depended on the previous time step hidden layer ht−1 and the current time step input layer xt−1. And the output layer yt is update by the current time step hidden layer ht . Formalization denotation in formulas are as follows:
ht = fh(Uxt +Wht−1 + bh) (3)
yt = fy(V ht + by) (4)
Where U, V,W are matrices for linear projection. And fh, fy are nonlinear activation functions for nonlinear transformation.
The inputs of RNN is word embeddings representation of hyponym and candidate hypernym sequence. We regard the hyponym and candidate hypernym as a sequence, because that the judgment of candidate hypernym is depended on hyponym in discrimination process. Ignoring the outputs during recurrent process, we take the final output of the last input in the sequence as the result of discrimination.
3.4 Fusion Learning Architecture Combined with a Simple Lexical Structure Rule
Generative architecture can get a very high precision by adjusting the threshold value of similarity, but will pay a high price for low recall-value. Compared with generative method, discriminative architecture can obtain a higher recall-value with low guarantee for precision-value.
The feature of discriminative architecture indicates that if it determines a candidate hypernymhyponym relation word pair as negative, then the word pair will have high probability for negative. We can use discriminative architecture to help the generative one to get rid of some false positive instance. For this reason, we fuse the generative and
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Discriminative Architecture ( RNN )
Generative Architecture ( based on MLP )
distance ∧ ∨
洲际 弹道 导弹 Intercontinental Ballistic Missile
Lexical Structure Rule
=
: hyponym
: candidate hypernym
: similar with true hypernym
: Boolean value. 0 for unrelated; 1 for hypernym-hyponym relation.
Figure 4: Fusion Learning Architecture.
discriminative architectures together by applying Boolean operator “AND” to the results outputted by the two architectures. Excepting a much higher precision-value than the precious two models and almost the same recall-value as the generative one.
By combining discriminative and generative architectures, the fusion architecture can discovery hypernym-hyponym much more precisely but becomes only focusing deep level semantic and ignoring the lexical stricture information which is very useful for discovery of hypernym-hyponym relationship, especially for compound nouns (CNs), for instance, “洲际 弹道导弹(Intercontinental Ballistic Missile)”. The root word of CN often indicates a hypernym relation, like the word “导 弹(Missile)” is the hypernyms of the precious CN. Root word of a CN can be obtained via using syntax dependency parsing or semantic dependency parsing of CN. Due to the word formation rule of Chinese, the root word is usually the last word in CN segmentation result. To supplement the capacity of learning semantic hierarchy from lexical structure, we use the simple lexical structure rule to assist previous fusion model.
The final fusion learning architecture (showed in Figure 4) is composed of three parts, namely generative architecture, discriminative architecture and lexical structure rule module.
4 Experiments
In the experimental stage, we implement our fusion architecture for learning semantic hierarchies. To the end of this, we first introduce the preparation of experimental setup. Next, we report the
performance of fusion architecture and its components. Subsequently, we compare the performance of our method to those of several previous methods in different aspects and give an example for construction of semantic hierarchies.
4.1 Experimental Setup
Pre-trained Word Embeddings
We use a Chinese encyclopedia corpus named Baidubaike1 to learn word embeddings, which contains about 30 million sentences (about 780 million words). The Chinese segmentation technology is provided by the open-source Chinese language processing platform LTP2 (Che et al., 2010). Then, we employ the Skip-gram method (Section 3.1) to train word embeddings for the further experiment. We obtain the embedding vectors of 0.56 million words in total.
Dataset and Evaluation Metrics
The training data for learning semantic hierarchies is collected from CilinE3 which contains 100,093 Chinese words and organized as a hierarchy of five levels, in which the words are linked by hypernym–hyponym relations. Finally, we obtain 15,242 word pairs of hypernym–hyponym relation for positive instances and constructed 15,242 negative instances for training.
For comparability we use the same test dataset as Fu et al. (2014) in evaluation stage. They obtain the hypernyms for 418 entities, which are se-
1Baidubaike (https://baike.baidu.com/) is one of the largest Chinese encyclopedias.
2http://www.ltp-cloud.com/demo/ 3http://www.ltp-cloud.com/download/
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Word Dimension
# of neutrons in hidden layer(RNN)
# of neutrons in hidden layer(MLP)
Batch size Adadelta parameter
dw = 300 nh = 800 nh = 500 b = 20 ρ = 0.95, ε = 1e −6
Table 1: Parameters used in our experiments.
lected randomly from Baidubaike, following their previous work (Fu et al., 2013). The final data set was manually labeled and measured the interannotator agreement by using the kappa coefficient (Siegel and Castellan, 1981). The kappa value is 0.96, which indicates a good strength of agreement. Training data and test data are showed in Table 2.
Relation # of word pairsTraining Test hypernym-hyponym 15,242 1,079 hyponym-hypernym 7,621 1,079 unrelated 7,621 3,250 Total 30,484 5,408
Table 2: The Experimental Data.
We use precision-value, recall-value, and F1score as metrics to evaluate the performances of the methods. Since the discovery of hypernymhyponym relation is a binary classification task, we only report the performance of the positive instances recognition in the experiments.
Parameter Settings and Training
In our fusion architecture, there are MLP for generation (see in Section 3.2) and RNN for discrimination (see in Section 3.3) need to be trained. We experimentally study the effects of several hyperparameters on this two neural networks: the number of neutrons in hidden layer, the selection of activation function. Table 1 shows all parameters used in the experiments. We use Adadelta (Zeiler, 2012) in the update procedure, which relies on two main parameters, ρ and ε, which do not significantly affect the performance. Following Zeiler (2012), we choose 0.95 and 1e−6, respectively, as the values of these two parameters.
In the training stage, we train the discriminative architecture and generative architecture respectively. For training discriminative architecture based on RNN, we use the whole training data. But for training generative architecture based on MLP, we only use the positive instances in train-
ing data as Fu (2014) for generating positive hypernym vectors.
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 Distance Threshold
63.20%
63.60%
64.00%
64.40%
64.80%
65.20%
65.60%
0.5 0.6 0.7 0.8 0.9 1
Threshold for Probability
(a) generative architecture (b) discriminative architecture
Figure 5: The effects on F1-score of threshold for generative and discriminative architecture.
70.00%
70.50%
71.00%
71.50%
72.00%
72.50%
73.00%
73.50%
74.00%
74.50%
0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1
F1
Threshold for Probability
Figure 6: The effects on F1-score of threshold for fusion architecture.
4.2 Performance of Fusion Architecture
In the test stage, we tune the performance of architecture by changing the thresholds for components of fusion architecture. There two kinds of thresholds need to be tuned: the similar distance of generative architecture and the probability for positive instances of discriminative architecture.
We first experimentally study the effects of semantic distance threshold on generative architecture based on MLP neural network. Selecting the semantic distance threshold of best F1-score for generative architecture and fixing it, we tun the probability threshold for positive instances of discriminative architecture, observing the F1-score
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
curve of discriminative architecture and the fusion learning architecture (see in Figure 5, Figure 6).
4.3 Comparison with Previous Work
In this section, we compare the proposed method with previous methods, including pairwise hypernym-hyponym relation extraction based on patterns, word distributions, web mining, and based on word embeddings(see in Section 2). Results are shown in Table 3.
Method P(%) R(%) F1(%) MPttern 97.47 21.41 35.11 MSnow 60.88 25.67 36.11 MbalApinc 54.96 53.38 54.16 MinvCL 49.63 62.84 55.46 MWeb 87.40 48.19 62.13 MEmb 80.54 67.99 73.74 MlexicalRule 100.0 16.88 28.88 MMLPgen. 77.96 53.51 63.46 MRNNdis. 55.97 77.40 64.96 MMLP+RNN 90.00 51.48 65.50 MFusion 91.60 62.36 74.20
Table 3: Comparison of the proposed method with existing methods in the test set.
Overall Comparison
MPattern refers to the pattern-based method (Hearst, 1992). The method uses the Chinese Hearst-style patterns (Fu et al., 2013). The result shows that only a small part of the hypernyms can be extracted based on these patterns because only a few hypernym relations are expressed in the fixed patterns, and most of them are expressed in highly flexible manners. MSnow originally proposed by Snow et al. (2004). This method relies on an accurate syntactic parser, and the quality of the automatically extracted patterns is difficult to guarantee.
There are two previous distributional methods MbalApinc (Kotlerman et al., 2010) and MinvCL (Lenci and Benotto, 2012). Each word is represented as a feature vector in which each dimension is the point-wise mutual information (PMI) value (Geffet and Dagan, 2005; ZhitomirskyGeffet and Dagan, 2009) of the word and its context words (see in Section 2). MFu refers to a web mining method proposed by Fu et al. (2013) which mines hypernyms of a given word w from multiple sources returning a
ranked list of the hypernyms. MEmb refers to a novel method based on word embeddings achieving the best F1-value among previous methods. MlexicalRule refers to using lexical structure rule to discover hypernym-hyponym relations (see in Section 3.4). The 100% precision-value on test data indicates the lexical rule is correct for most compound nouns (CNs). However the rule only takes effect for CNs which are minority. MMLPgen. represents the generative architecture based on MLP neural network (see in Section 3.2), the method get a higher F1-score than most of previous semantic hierarchy discovery method except MEmb. MRNNdis. represents the discriminative architecture based on RNN neural network (see in Section 3.3) which obtains the highest recall-value in comparison of the proposed methods. MMLP+RNN combines these two architectures based on MLP and RNN (see in Section 3.4), getting a much higher precision-value than any components and a comparable recall-value with MMLPgen. . MFusion refers to a fusion learning architecture composed of discriminative and generative architectures and assisted with lexical structure rule (see in Section 3.4). Assisted with the simple lexical structure rule, the fusion learning architecture get a better F1-score than all of the previous methods do and significantly improve the precisionvalue over the state-of-the-art method MEmb.
Further, we experimentally study the effects of lexical rule for MEmb. The results show that the method MEmb+lexicalRule does not improve the F1-score compared with the original MEmb. The reason maybe that MEmb already recall the compound nouns (CNs) hypernym-hyponym relations (see in Table 4). The results also indicate that the improvement is mainly caused by discriminative generative architecture.
P(%) R(%) F1(%) MEmb+lexicalRule 80.54 67.99 73.74 MFusion 91.60 62.36 74.20
Table 4: Comparison of the MEmb assisted with lexical structure rule.
Comparison on the Out-of-CilinE Data
Since the training data is extracted from CilinE, we are greatly interested in the performance of our method on the hypernym-hyponym relations out-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
side of CilinE. We assume that as long as there is one word in the pair not existing in CilinE, the word pair is outside of CilinE. In our test data, about 61% word pairs are outside of CilinE.
P(%) R(%) F1(%) MWiki+CilinE 80.39 19.29 31.12 MEmb 65.85 44.47 53.09 MFusion 79.92 44.94 57.53
Table 5: Comparison on the Out-of-CilinE Data.
Table 5 shows the performances of the baseline method MWiki+CilinE , previous state-of-theart methodMEmb and our methodMFusion on the out-of-CilinE data. In comparison, MWiki+CilinE has the highest precision-value but has a lowest recall-value, MEmb significantly improve recallvalue and F1-score. By contrast, our method MFusion can discover a little bit more hypernym–hyponym relations than MEmb with achieving a more than 14% precision-value improvement. And our method can get an F1-score of 57.53%, which is a new state-of-the-art result on the Outof-CilinE Data.
Combined with Manually-Built Hierarchies
For further exploration, we combine our method MFusion with the existing manually-built hierarchies in Wikipedia and CilinE. The combination strategy is to simply merge all positive results from the two methods together, and then to infer new relations based on the transitivity of hypernym–hyponym relations. The same manner allied to precious method MEmb to be compared. The comparison is showed in Table 6. Combining our fusion method MFusion with manually-built hierarchies Wikipedia and CilinE can further improve F1-score to 82.01%, gets an about 1.7% improvement compared with the same manners on MEmb.
P(%) R(%) F1(%) MEmb 80.54 67.99 73.74 MFusion 91.60 62.36 74.20 MEmd+CilinE 80.59 72.42 76.29 MFusion+CilinE 91.64 70.76 79.85 MEmd+Wiki+CilinE 79.78 80.81 80.29 MFusion+Wiki+CilinE 91.00 74.63 82.01
Table 6: Comparison ofMFusion andMEmb combined with manually-built hierarchies.
4.4 Example of Learning Semantic Hierarchies
In Figure 7, there is an example of learning semantic hierarchies based on our fusion architecture (MFusion) and combined method using manuallybuilt hierarchies (MFusion+Wiki+CilinE). From the results, we can see that our method can actually learn the semantic hierarchies for a given word and its hypernyms list relatively precisely. NBA球员 NBA player 篮球运动员 basketball player 球员 ball player 运动员 athlete 体育人物 sportsman 人物 person
(b) 𝑀𝐹𝑢𝑠𝑖𝑜𝑛+𝑊𝐼𝑘𝑖+𝐶𝑖𝑙𝑖𝑛𝐸
NBA球员 NBA player
篮球运动员 basketball play r
球员 ball player
运动员 athlete
体育人物 sportsman
人物 person
(a) 𝑀𝐹𝑢𝑠𝑖𝑜𝑛
乌头 aconite
药品 medicine
植物 plant
毛茛科 Ranunculaceae
植物药 medicine plant 乌头属
Aconitum
生物 organism
(b) 𝑀𝐹𝑢𝑠𝑖𝑜𝑛+𝑊𝐼𝑘𝑖+𝐶𝑖𝑙𝑖𝑛𝐸
乌头 aconite
药品 medicine
植物 plant
毛茛科 Ranunculaceae
植物药 medicine plant 乌头属
Aconitum
生物 organism
(a) 𝑀𝐹𝑢𝑠𝑖𝑜𝑛
Hyponym List of Hypernyms
NBA球员 (NBA player) 球员 (ball player) 篮球运动员 (basketball player) 运动员 (athlete) 体育人物 (sportsman) 人物 (person)
Hyponym List of Hypernyms
乌头 (conite) 植物药 (medicine plant) 乌头属 (Aconitum) 药品 (medicine) 毛茛科 (Ranunculaceae) 植物 (plant) 生物 (organism)
Figure 7: Example of Learning Semantic Hierarchies.
The dashed line frames in Figure 7(a) refers to the losing hypernym-hyponym relations words. For instance, our method fail to learn the semantic hierarchies between “乌头属(aconite)” and “毛茛科(Ranunculaceae)”. The reason maybe that their semantic similarity effects representations close to each other in the embedding space and our method can not find suitable projection for these pairs. By combining our method with manually-built hierarchies, we can improve the capacity of learning semantic hierarchies. In this case, the combined method can build the semantic hierarchies correctly (see in Figure 7(b)).
5 Conclusion
This paper proposes a novel method for learning semantic hierarchies based on discriminative generative fusion architecture combined with a very simple lexical structure rule. The fusion architecture method can be easily expanded to be suitable for other languages. In experiments, the proposed method achieves the best F1-score of 74.20% on a manually labeled test dataset outperforming stateof-the-art methods with a much higher precisionvalue of 91.60% for application. Further experiments show that our method is complementary with some manually-built hierarchies to learn semantic hierarchy construction more precisely.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: - The paper tackles an important issue, that is building ontologies or thesauri - The methods make sense and seem well chosen - Methods and setups are well detailed - It looks like the authors outperform the state-of-the-art approach (but see below for my concerns)  - Weaknesses: The main weaknesses for me are evaluation and overall presentation writing. - The list of baselines is hard to understand. Some methods are really old and it doesn't seem justified to show them here (e.g., Mpttern). - Memb is apparently the previous state-of-the-art, but there is no mention to any reference. - While it looks like the method outperforms the previous best performing approach, the paper is not convincing enough. Especially, on the first dataset, the difference between the new system and the previous state-of-the-art one is pretty small. - The paper seriously lacks proofreading, and could not be published until this is fixed   for instance, u00a0I noted 11 errors in the first column of page 2. - The CilinE hierarchy is very shallow (5 levels only). However apparently, it has been used in the past by other authors. I would expect that the deeper the more difficult it is to branch new hyponym-hypernyms. This can explain the very high results obtained (even by previous studies)...  - General Discussion: The approach itself is not really original or novel, but it is applied to a problem that has not been addressed with deep learning yet. For this reason, I think this paper is interesting, but there are two main flaws. The first and easiest to fix is the presentation. There are many errors typos that need to be corrected. I started listing them to help, but there are just too many of them. The second issue is the evaluation, in my opinion. Technically, the performances are better, but it does not feel convincing as explained above. What is Memb, is it the method from Shwartz et al 2016, maybe? If not, what performance did this recent approach have? I think the authors need to reorganize the evaluation section, in order to properly list the baseline systems, clearly show the benefit of their approach and where the others fail. Significance tests  also seem necessary given the slight improvement on one dataset.


Generating Memorable Mnemonic Encodings of Numbers
The major system is a mnemonic system that can be used to memorize sequences of numbers. In this work, we present a method to automatically generate sentences that encode a given number. We propose several encoding models and compare the most promising ones in a password memorability study. The results of the study show that a model combining part-of-speech sentence templates with an n-gram language model produces the most memorable password representations.
1 Introduction
The major system is a mnemonic device used to help memorize numbers. The system works by mapping each digit of a number to a consonant phoneme and allowing for arbitrary insertion of vowel phonemes to produce words (FauvelGouraud, 1845). For instance, the digit 1 maps to <T>, and the digit 2 maps to <N>. The number 121 can then be encoded as the word tent by replacing both 1s with <T>s, replacing the 2 with <N>, and inserting an <e>. The full major system mapping is shown in Table 1.
The difficulty of generating a memorable sequence of words that encodes a number with the major system stems from the constraint that the sequence of words must encode exactly the given digits. While there are many sequences of words that correctly encode a given number, the vast majority of these sequences are incoherent and thus difficult to remember. So, this task requires the use of a language model that balances the encoding constraints with syntactic plausibility and some notion of memorability.
We have developed a system that automatically produces a sequence of words to encode a se-
quence of digits. Each such encoding is a sequence of sentences that balance memorability and length. We sample from a distribution of partof-speech (POS) templates to produce a syntactically plausible sentence, then use an n-gram language model to fill each POS slot in the selected template to produce an encoding.
A system like ours can be used to memorize fairly short numbers, such as a numeric password, a phone number, or an account number; or to memorize arbitrarily long numbers, like digits of π. One could use our system to encode a smartphone passcode as a short sentence. Thus, our system can help improve the strength of security practices.
To test the effectiveness of our system, we conducted a study on password memorability. Participants were asked to memorize an eight-digit number representing a numeric password and a phrase produced by our system to encode the same number. After seven days, participants remembered the encoding produced by our final model better than the number itself. Participants also stated a
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
strong preference for our final model’s encodings. The rest of this paper is organized as follows. Section 2 describes existing systems for automatically generating encodings with the major system and puts our work in the context of related academic problems. In Section 3, we describe the different encoding models that we studied. Section 4 gives results and analysis of our models. In Section 5, we describe the password memorability study we conducted. We describe possible extensions to our work and conclude in Section 6.
2 Previous Work
Several tools are available online for generating naive encodings of numbers using the major system. In this section, we describe those tools and identify the shortcomings in those implementations that are addressed by our work. We also put our work into the context of previous studies on password memorability.
2.1 Existing Tools
There are a number of existing tools that use the major system to encode sequences of digits as sequences of words. However, all such tools we found have considerable limitations. Most notably, the majority of these tools simply return the entire set of words that can individually encode the given number.
Many mobile applications will generate encodings, but their focus appears to be on helping users learn the major system and not on generating memorable encodings automatically (Shichijo, 2014; Reindl, 2015; Vladislav, 2016; Scott, 2015a; Pfeiffer, 2013; Buder, 2012). None of these tools rank the multiple encodings they produce, and none of them produce sentences. Most of these tools only produce one- or two-word encodings, greatly limiting the length of sequences they can encode (Graaff, 2016; Rememberg, 2010; Jurkowski, 2014).
Other tools produce encodings of longer sequences of digits by breaking the sequence into chunks of a fixed length, often two digits per word, and most do not combine the single-word encodings into one sequence of words (Ströbele, 2013; Scott, 2015b; Parcel, 2016; Got2Know, 2013).
Thus, these existing approaches are ill-suited for the memorization of even moderately long sequences of digits. Since the most sophisticated of these approaches are equivalent to our baseline
models, we do not empirically compare these tools to our models.
2.2 Related Academic Work
We are only aware of two previous corpus-based methods for generating mnemonic encodings. The first presents a method to help remember text passwords by finding a newspaper headline from the Reuters Corpus such that the first letters of each word in the headline match the letters in the password (Jeyaraman and Topkara, 2005). However, the restriction of using only newspaper headlines means that only about 40% of seven-character passwords are covered.
The second corpus-based method addresses the related problem of memorizing random strings of bits. Ghazvininejad and Knight (2015) created a method to encode random 60-bit strings as memorable sequences of words. However, their methods that create the most memorable passwords do not allow the user to mentally convert their memorized sequence of words to the original string. In contrast, our use of the major system allows users to easily convert any sequence of words into the encoded number in their head.
Although these methods encode a sequence of letters and a string of bits while our system encodes a sequence of digits, all aim to create memorable sentences as output. Based on the results of these two previous methods, our system favors unique words and sentences of moderate length. Because our system needs to encode any arbitrary sequence of digits, we use a language model to generate sentences instead of relying on a preexisting set of newspaper headlines.
Substantially more work has been done on the memorability and security of passwords. Forget and Biddle (2008) found that modifying usercreated passwords to increase security had the unintended consequence of reducing memorability. Yan et al. (2004)’s work provides a possible means of dealing with that tension between security and memorability, showing that passwords based on mnemonic phrases were as easy to remember as naively created passwords and as strong as random passwords. Their positive results for mnemonicbased passwords are encouraging for our own mnemonic-based system. Our system is further informed by the work of Leiva and Sanchis-Trilles (2014), who analyzed different methods of sampling memorable sentences from corpora to use as
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
prompts in text entry research. They found that prompts are more memorable when they are complete phrases and have fewer words.
Our user study experiment evaluating the memorability of the phrases generated by our systems is informed by the existing work in this area. The format of our human subjects experiment is largely informed by the work of Vu et al. (2004). They examined the use of passphrases created by taking the first letter of each word in a sentence. Their user study split participants into two passphrase groups, the second of which had to include a number and a special character in the passphrase. The participants were not allowed to write the passwords down. The researchers then tested the participants’ recall after five minutes and after a week. The results showed that the second group produced much more secure passwords but at the cost of memorability.
3 Methodology
In this section, we describe the data we used to train our models and present the six mnemonicgenerating systems we considered. These models include two baseline models, three preliminary models, and the final sentence encoder model. The source code for these models will be made available online 1.
3.1 Data Sets
We use two data sets in our system. The first is the Brown corpus, which contains about 56,000 types and 1.2 million tokens (Francis and Kuera, 1964). We use this corpus to train our n-gram models. The corpus is also tagged with part-of-speech data, which we use to train our part-of-speech n-gram model and our sentence encoder model.
The second data set we use is the CMU Pronouncing Dictionary (Lenzo, 2014). This data set is a file of about 134,000 words, each labeled with its pronunciation in the Arpabet phoneme set. We work with the intersection of these two data sets, which contains about 34,000 words. This ensures that all the words produced by our language model can be pronounced from the CMU Pronouncing Dictionary. We pre-process both data sets to lowercase all words.
1 URL withheld for blind review.
3.2 Baseline Models
We designed two baseline models to compare our results against. Each of these baselines satisfies the requirement that the sequences of words produced encode exactly the input digits. Both baselines are greedy: they generate encodings one word at a time. At each time step, they choose a word from the set of words that encode the maximum number of digits possible. They differ in how a word is chosen from that set:
Random Encoder: At each step, a word is selected at random.
Unigram Encoder: At each step, the word with the highest unigram probability is selected.
3.3 Preliminary Models
We also considered three models that were more sophisticated than our baseline models. Unlike the baseline models that greedily encode as many digits per word as possible, these models consider all words that can encode at least one digit.
n-gram Encoder: Words are generated one at a time. At each step, the next word is chosen using an n-gram language model with Stupid Backoff (Brants et al., 2007). We tested different combinations of hyperparameters and decided on default values of n = 3 and backoff factor α = 0.1. An additional hyperparameter indicates if the model should select the word with the highest n-gram probability or sample from a weighted probability distribution based on n-gram probabilities, with the former option as the default.
Part-of-Speech (POS) Encoder: Words are generated one at a time, but a POS trigram model is used to restrict the set of possible words at each step so that the generated phrases are syntactically motivated. Each word is associated with the POS tag it most often has in the Brown corpus. To select each word in the encoding, the most likely POS tag is identified from the POS trigram model. From all words with that POS, we choose the word that has the greatest likelihood according to a word trigram model.
Chunk Encoder: Instead of generating encodings one word at a time, we generate one sentence at a time. Each sentence must match a
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
fixed phrase template: <noun phrase><verb phrase><noun phrase>. Additionally, each chunk encodes exactly three digits. This encoder breaks the given number into chunks of three digits and encodes each chunk as one or two words that can be parsed as the desired chunk type. For each chunk, the phrase that has the greatest likelihood according to a bigram language model is selected.
3.4 Final Model: Sentence Encoder
After examining the output of the three preliminary models, we combined the best elements of each into a final model, which we call the Sentence Encoder. This model aims to produce a variety of sentence structures that are both adequately long and reasonably likely to occur.
The sentence encoder trains a trigram model on the Brown corpus and stores the 100 most frequent sentence templates found in the corpus. A sentence template is a sequence of part-of-speech tags from the corpus’s simple “universal” tag set. We filter these sentence templates to only contain sentences that have a verb, do not have any numbers or “other”-category words (like foreign words), and are guaranteed to produce at least 5 words.
To encode a sequence of digits, the sentence encoder first samples a sentence template based on the templates’ frequencies in the training corpus. Then, for every part of speech in the template, the encoder selects the word that encodes at least the next digit with the most likely trigram score based on the previous words. If the encoder is unable to find a word that matches the necessary part of speech, it replaces the current sentence with a different, newly sampled sentence template. This process is repeated until all digits are encoded, possibly resulting in the end of the last sentence template being unused.
A few additional changes to this model greatly improve its performance:
• We allow nouns in place of pronouns, since there are more possible nouns than pronouns.
• We allow certain parts of speech - determiners, adjectives, and adverbs - to be skipped if no word is found that matches them.
• We weight the trigram score of each word based on how many digits it encodes. We
do this by multiplying the score by the number of digits the word encodes raised to some power, which is set to 10 by default. We previously found that the most likely sequences of words tend to encode only one or two digits per word, resulting in sentences that are long and thus less memorable.
• We run a post-processing pass over the output sentences. The post-processing pass iterates over each word, calculates the probabilities for all possible words that encode the exact same digits using a bigram language model with the preceding and following words, and replaces the original word with the most likely word.
With these four changes, the sentence encoder is able to encode all numbers as memorable, syntactically plausible sentences of a reasonable length.
As an example, consider the number 86101521. The sentence encoder first samples a sentence template, such as “<verb> <noun> <conj> <verb> <adv>.” Then, the sentence encoder finds a verb that encodes at least the first digit, 8. The sentence encoder selects the verb “officiate,” which has consonant sounds “<F>,<SH>,<T>,” to represent 861. The remaining digits are 01521. The sentence encoder then selects the noun “wasteland,” with consonant sounds “<S>,<T>,<L>,<N>,<D>,” to represent 01521. So, 86101521 is encoded as “Officiate wasteland.”
4 Model Output
For each model, we look at the encoding generated for two numbers. The first number is a random eight-digit number, and the second is the first fifty digits of π.
The eight-digit number demonstrates each model’s ability to encode fairly short sequences of digits, such as a numeric password, a phone number, or an account number. Table 2 shows how each model encoded the number 86101521.
The first fifty digits of π demonstrates each model’s ability to encode an arbitrarily long sequence of digits. Table 3 shows how each model encoded the first fifty digits of π: 31415926535897932384626433832795028841971 693993751.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Encoder Phrase Random Vouching wits widely and Unigram Fish this tell and n-gram Of which the house to all. And POS Wife age at sea with law in the Chunk Half shut settle night. Sentence Officiate wasteland.
Table 2: Encodings generated by each model for the 8-digit code 86101521.
Encoder Phrase Random meeting rawhide yelping hunch alum levy bog boom annum ivory gin sharing meme
femme knock appeal sinning vivo readying bake twitch beaming pub hammock highlighting
Unigram made right help enjoy william life back p.m. name over john sure mama foam neck able seen five right back touch p.m. baby make old n-gram Him to hire youth all. Be in show all my life. Be echoing be my own home. Of our age in which our. Him home of my own. Go up all his own. Of every day by god. Which by him by be. Him go along with. POS Matter with law by an age along mile of hope. Week by man among every age in age. Year among aim of man. Week by law use in favor with pike with age by humming by boy among week along the. Chunk Matter would leap new jail. Home life book by money home. Average enjoy her home movie. Human ego being less won five. Earth by god she poem by. Buy make lead. Sentence Matter tell been shell among life. Pickup man moving or which nature. Mama of many couples in favor. Tobacco touch pump by my cold.
Table 3: Encodings generated by each model for the first 50 digits of π.
4.1 Comparison of Encodings
The random encoder generates mnemonic encodings that use obscure words without any structure. As such, the random encoder does not produce memorable encodings.
The unigram encoder improves on the random encoder by favoring more common words, which tends to result in shorter encodings. However, as neither model considers part-of-speech information, the two baselines produce unrelated sequences of words, which are difficult to chunk and to remember.
The n-gram encoder generates encodings that tend to be long and unmemorable. The encoder often produces incoherent phrases of common words, such as “the of which by his own.” The sentences produced by the other three nonbaseline models tend to be more memorable.
For example, the POS encoder generates longer, more syntactically plausible sentences. The biggest drawback of this model is the relative lack of verbs in its generated sentences.
The chunk encoder produces encodings similar to those produced by the POS encoder but with a noun-verb-noun pattern that results in relatively short, simply structured sentences. However, the chunk encoder has a slow running time due to the relatively expensive process of parsing each possible noun phrase and verb phrase.
The sentence encoder produces words such that each word encodes many digits and tends to be distinctive. This means that fewer words and thus fewer sentences are needed to encode a given sequence of digits, making the mnemonic encodings generated by the sentence encoder easy to chunk and memorable. We hypothesize that the sentence encoder generates more memorable sentences. We tested that hypothesis through a user study.
5 User Study
We conducted a user study to test the memorability of the phrases generated by our models. The study was presented as a study of password mem-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
orability, in which each password was an 8-digit number or its encoding from the n-gram encoder or the sentence encoder. We compared the ngram model (which served as our baseline) to the sentence model (which we hypothesized best balances all aspects of memorability) for four factors:
Short-Term Recall: How well can participants remember the number or its encoding after five minutes?
Long-Term Recall: How well can participants remember the number or its encoding after one week?
Long-Term Recognition: How well can participants identify the number or its encoding from a list of options after one week?
Subjective Comparison: How easy does each number or its encoding seem to users? This comparison may give an indication of how likely users are to consider trying a particular mnemonic.
5.1 Study Overview
The study was comprised of two online surveys, which together take about fifteen minutes to complete. We recruited participants through emails to a computer science summer research program and through social media posts. Participants were compensated with a five-dollar Amazon.com gift card via email after completing the second survey.
While we had 167 participants complete both surveys, we found in the second survey that 101 of the respondents were fraudulent. These responses shared a number of suspicious characteristics:
• They were completed consecutively, usually with only a few seconds from one response to the next, for both the first and the second survey.
• None of them spent as much time on the survey tasks as other participants. The median time spent on our distraction task, for example, was 4 seconds for these participants. For other participants, the median time spent on the same task was 81 seconds.
• They appeared in groups of consecutive email addresses from the same free email provider.
• They all remembered the numeric code and its encoding perfectly after one week.
Removing these participants left 67 responses to the first survey and 66 responses to the second.
5.2 Study Design
Each participant was randomly assigned to one of two groups, n-gram or sentence. Participants were identified by the email address they entered in the first survey, which we then used to send them a link to the second survey and to identify their responses.
The first survey presented each participant with an 8-digit sequence and a corresponding sequence of words from either the n-gram encoder or the sentence encoder that encoded the same 8-digit sequence. The participant was asked to remember both the number and the encoding, without writing them down, and was informed that they would be asked to recall both sequences at the end of the first survey and on the second survey. After entering both sequences to confirm initial memorization, the participant was asked to read a page of baking recipes for approximately five minutes. This page, containing numbers and words, served to clear the participant’s working memory. Next, the participant was asked to enter both sequences, with an “I forgot” button available if necessary. Each correct or incorrect attempt was recorded, as was the time spent on each page of the survey.
The second survey was sent to each participant seven days after their completion of the first survey. The survey again asked each participant to recall both sequences and recorded the same information. The participant was then asked to recognize their 8-digit sequence from a list of five sequences and to recognize their word sequence from a list of five word sequences generated by their group’s encoder. Next, the participant was shown an 8-digit sequence, its n-gram encoding, and its sentence encoding and asked to rank the three passwords from “easiest to remember” to “hardest to remember.” Finally, the participant was asked to share the approach they used to memorize the two sequences.
5.3 Study Results
Our primary goal in performing this user study was to evaluate our hypothesis that the sentence encoder produced more memorable encodings than the n-gram encoder. We also sought
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
to determine whether the sentence encodings are more memorable than the relatively short 8-digit sequences themselves.
Short-Term Recall: On the first survey, more participants correctly confirmed the sentence password than the n-gram password. 29 of the 31 sentence password participants (94%) remembered the password, as opposed to only 26 of the 36 n-gram password participants (72%). This difference is statistically significant at α = 0.05 under a z-test for two proportions with p = 0.02. Participants with the sentence password also spent significantly less time on the confirmation page than did participants with the n-gram password (27 seconds versus 52 seconds, with p = 0.019 under an independent two-tailed t-test). After reading the recipes distraction page, more participants recalled the numeric password than the n-gram password (55/67 versus 23/36, p = 0.0403).
Long-Term Recall: On the second survey, we found no statistically significant differences between the number of participants who recalled the n-gram password, the sentence password, or the numeric password (respectively 9/36, 10/30, 25/66).
Long-Term Recognition: On the second survey, more participants recognized the sentence password than the numeric password when shown five options for each (30/30 versus 58/66, p = 0.05).
Subjective Comparison: More participants rated the sentence password as “easiest to remember” than the numeric password (36/66 versus 24/66, p = 0.04), and more participants rated the numeric password as “easiest to remember” than the n-gram password (24/66 versus 6/66, p < 0.001).
The results of our user study indicate that the sentence encoder produces more memorable encodings than the n-gram encoder does. These results also indicate that n-gram encodings are harder to remember in the short-term than the number itself. While few participants recalled any passwords after seven days, more participants recognized the sentence password than the numeric password, indicating that the sentence password is more memorable. Furthermore, the sentence password was most frequently rated “easiest to remember,” followed by the numeric password and the n-gram password. We conclude that the sentence encoder produces more memorable encod-
ings than the n-gram encoder and effectively aids in the memorization of numbers.
6 Conclusions and Future Work
We have described several systems for generating encodings of numbers using the major system. While n-gram models generate sentences that accurately match their training corpora, the sentences tend to be long and unmemorable. Sentences based on POS tags tend to be more memorable but are still not syntactically reasonable. Forcing the same sentence structure on every sentence by parsing ensures a reasonable structure but at a high computational cost. Ultimately, ensuring that each sentence is of a known but randomly selected syntactic structure that favors short encodings produces a reasonable balance of syntactic correctness, length, and memorability. A user study on password memorability supports our claim that the sentence encoder produces memorable mnemonic encodings of numbers.
Future work could further improve the sentence encoder. We could produce a more interesting variety of sentences by using punctuation from the training corpus besides periods, such as commas, exclamation marks, and question marks. The sentence encoder often produces a fragment as its last sentence because it runs out of digits to encode. This problem could be mitigated by making the encoder select shorter sentence templates when there are few digits remaining. Furthermore, the encoder could use more nuanced sentence templates to enforce subject-verb agreement and grammatical use of auxiliary verbs.
While the sentence encoder takes a greedy approach in an effort to encode digits in as few words as possible, another potential approach would be to use a dynamic programming algorithm to efficiently search through all possible encodings. Given a suitable objective function as a proxy for memorability, this could potentially produce a more memorable encoding than the sentence encoder without excessively increasing the encoding’s length.
One issue in our user study was the unexpectedly short amount of time spent on the distraction page, with an average of 81 seconds instead of the intended five minutes. Future user studies should enforce a set distraction time through the use of a timer or other mechanism when studying shortterm recall.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
A future user study could examine the effectiveness of our system on longer numbers. While our user study did not show that any particular type of password was easiest to recall after seven days, we expect that a study involving longer passwords would show the sentence password to be easiest to recall. This is because the major system is wellsuited for aiding the memorization of long numbers while the eight-digit numeric passwords used in our study are relatively short. It would also be interesting to see how memorable our encodings are in a context where users are prompted to recall the password each day over a long period of time.

- Strengths:  Tackles a not very explored task, with obvious practical application Well written and motivated  - Weaknesses:  The only method of validation is a user study, which has several weaknesses.  - Discussion:  The paper investigates various methods to generate memorable mnemonic encodings of numbers based on the  Major d system. As opposed to other methods that rely on this system to encode sequences, the methods proposed in this work return a single sequence (instead of a set of candidates) which is selected to improve memorability. Since  memorability d is an ambiguous criterion to optimize for, the authors explore various syntactic approaches that aim for short and likely sentences.  Their final model uses a POS template sampled form a set of  nice d structures, and a tri-gram language model to fill in the slots of the template.   The proposed approach is well motivated: the section on existing tools places this approach in the context of previous work on security and memorability. The authors point to results showing that passwords based on mnemonic phrases offer the best of both worlds in terms of security (vs random passwords) and memorability (vs naive passwords). This solid motivation will appease those readers initially skeptical about the importance feasibility of such techniques.   In terms of the proposed methods, the baselines and n-gram models (unsurprisingly) generate bad encodings. The results in table 2 show that indeed Chunk and Sentence produce shorter sentences, but for short digits such as this one, how relevant are the additional characteristics of these methods (eg. POS replacements, templates etc)? It seems that a simple n-gram model with the number-of-digits-per-trigram reweighing could perform well here.   The evaluation is weaker than the rest of the paper. My main concern is that a one-time memorization setting seems inadequate to test this framework. Mnemonic techniques are meant to aid recall after repeated memorization exercises, not just a single  priming d event. Thus, a more informative setting would have had the users be reminded of the number and encoding daily over a period of time, and after a  buffer period d, test their recall. This would also more closely resemble the real-life conditions in which such a technique would be used (e.g. for password memorization).  In terms of the results, the difference between (long term) recall and recognition is interesting. Do the authors have some explanation for why in the former most methods performed similarly, but in the latter  Sentence d performs better? Could it be that the use of not very likely words (e.g.  officiate , in the example provided) make the encodings hard to remember but easy to spot? If this were the case, it would somewhat defeat the purpose of the approach.  Also, it would be useful for the reader if the paper provided  (e.g. in an appendix) some examples of the digits encodings that the users were presented during the study, to get a better sense of the difficulty of recall and the quality of the encodings.   - Suggestions:  It would be nice to provide some background on the Major system for those not familiar with it, which I suspect might be many in the ACL audience, myself included. Where does it come from? What  the logic behind those digit-phoneme maps?