Improved Neural Relation Detection for Knowledge Base Question Answering
Relation detection is a core component for many NLP applications including Knowledge Base Question Answering (KBQA). In this paper, we propose a hierarchical recurrent neural network enhanced by residual learning that detects KB relations given an input question. Our method uses deep residual bidirectional LSTMs to compare questions and relation names via different hierarchies of abstraction. Additionally, we propose a simple KBQA system that integrates entity linking and our proposed relation detector to enable one enhance another. Experimental results evidence that our approach achieves not only outstanding relation detection performance, but more importantly, it helps our KBQA system to achieve state-of-the-art accuracy for both single-relation (SimpleQuestions) and multi-relation (WebQSP) QA benchmarks.
1 Introduction
Knowledge Base Question-Answering (KBQA) systems answer questions by obtaining information from KB tuples (Berant et al., 2013; Yao et al.,
2014; Bordes et al., 2015; Bast and Haussmann,
2015; Yih et al., 2015; Xu et al., 2016). For an input question, these systems typically generate a
KB query, which can be executed to retrieve the answers from a KB. Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single <head-entity, relation, tail-entity> KB tuple (Fader et al., 2013; Yih et al., 2014; Bordes et al., 2015); and (b) a more complex case, where some constraints need to be handled for multiple entities in the question. The KBQA
system in the figure performs two key tasks: (1) entity linking, which links n-grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to.
The main focus of this work is to improve the relation detection subtask and further explore how it can contribute to the KBQA system. Although general relation detection1 methods are well studied in the NLP community, such studies usually do not take the end task of KBQA into consideration. As a result, there is a significant gap between general relation detection studies and KB-specific relation detection. First, in most general relation detection tasks, the number of target relations is limited, normally smaller than 100. In contrast, in KBQA even a small KB, like Freebase2M (Bordes et al., 2015), contains more than 6,000 relation types. Second, relation detection for KBQA often becomes a zero-shot learning task, since some golded test relations may not appear in the training data. For example, the SimpleQuestions (Bordes et al., 2015) data set has 14% of the golden test relations not observed in golden training tuples. Third, as shown in Figure 1(b), for some KBQA tasks like WebQuestions (Berant et al., 2013), we need to predict a chain of relations instead of a single relation. This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB relation detection. Owing to these reasons, KB relation detection is significantly more challenging compared to general relation detection tasks.
This paper improves KB relation detection to cope with the above mentioned problems. First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching. Second, noticing that original relation names can sometimes help
1In the information extraction field such tasks are usually called relation extraction or relation classification.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Question: what episode was mike kelley the writer of
Knowledge Base
Mike Kelley
(American television writer/producer)
Mike Kelley
(American baseball
player)
…
Entity Linking
Love Will Find a Way
USA
…
First baseman …
episodes_written
position_played
Relation Detection
(a) (b)
Question: what tv show did grant show play on in 2008
Mike Kelley ? episodes_written
Entity Linking Relation Detection
Grant Show
? starring_roles series
(date) from 2008
Constraint Detection
Grant Show (American actor) SwingTown
Big Love
episodes
Scoundrels series 2011 from
2010
2008
Figure 1: KBQA examples and its three key components. (a) A single relation example. We first identify the topic entity with entity linking and then detect the relation asked by the question with relation detection (from all relations connecting the topic entity). Based on the detected entity and relation, we form a query to search the KB for the correct answer “Love Will Find a Way”. (b) A more complex question containing two entities. By using “Grant Show” as the topic entity, we could detect a chain of relations “starring roles-series” pointing to the answer. An additional constraint detection takes the other entity “2008” as a constraint, to filter the correct answer “SwingTown” from all candidates found by the topic entity and relation.
to match longer question contexts, we propose to build both relation-level and word-level relation representations. Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels of question representations in order to match the different levels of relation information. Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching.
In order to assess how the proposed improved relation detection could benefit the KBQA end task, we also propose a simple KBQA implementation composed of two-step relation detection. Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a key role in the KBQA process: (1) Reranking the entity candidates according to whether they connect to high confident relations detected from the raw question text by the relation detection model. This step is important to deal with the ambiguities normally present in entity linking results. (2) Finding the core relation (chains) for each topic entity2 selection from a much smaller candidate entity set after re-ranking. The above steps are followed by an optional constraint detection step, when the question cannot be answered by single relations (e.g. containing multiple entities). Finally the highest scored query from the above steps is used to query the KB for answers.
2Following (Yih et al., 2015), here topic entity refers to the node of the (directed) query tree; and core-chain is the directed path of relation from root to the answer node.
Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks.
2 Related Work
Relation Extraction Relation extraction (RE) is an important sub-field of information extraction. General research in this field usually works on a (small) pre-defined relation set, where given a text paragraph and two target entities, the goal is to determine whether the text indicates any types of relations between the entities or not. As a result RE is usually formulated as a classification task. Traditional RE methods rely on large amount of hand-crafted features (Zhou et al., 2005; Rink and Harabagiu, 2010; Sun et al., 2011). Recent research benefits a lot from the advancement of deep learning: from word embeddings (Nguyen and Grishman, 2014; Gormley et al., 2015) to deep networks like CNNs and LSTMs (Zeng et al., 2014; dos Santos et al., 2015; Vu et al., 2016) and attention models (Zhou et al., 2016; Wang et al., 2016).
The above research assumes there is a fixed (closed) set of relation types, thus no zero-shot learning capability is required. The number of relations is usually not large: The widely used ACE2005 has 11/32 coarse/fine-grained relations; SemEval2010 Task8 has 19 relations; TACKBP2015 has 74 relations although it considers
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
open-domain Wikipedia relations. All are much fewer than thousands of relations in KBQA. As a result, few work in this field focuses on dealing with large number of relations or unseen relations.
Relation Detection in KBQA Systems Relation detection for KBQA also starts with featurerich approaches (Yao and Van Durme, 2014; Bast and Haussmann, 2015) towards usages of deep networks (Yih et al., 2015; Xu et al., 2016; Dai et al., 2016) and attention models (Yin et al., 2016; Golub and He, 2016). Many of the above relation detection research could naturally support large relation vocabulary and open relation sets (especially for QA with OpenIE KB like ParaLex (Fader et al., 2013)), in order to fit the goal of open-domain question answering.
Different KBQA data sets have different levels of requirement about the above open-domain capacity. For example, most of the gold test relations in WebQuestions can be observed during training, thus some prior work on this task adopted the close domain assumption like in the general RE research. While for data sets like SimpleQuestions and ParaLex, the capacity to support large relation sets and unseen relations becomes more necessary. To the end, there are two main solutions: (1) use pre-trained relation embeddings (e.g. from TransE (Bordes et al., 2013)), like (Dai et al., 2016); (2) factorize the relation names to sequences and formulate relation detection as a sequence matching and ranking task. Such factorization works because that the relation names usually comprise meaningful word sequences. For example, Yin et al. (2016) split relations to word sequences for single-relation detection. Liang et al. (2016) also achieve good performance on WebQSP with wordlevel relation representation in an end-to-end neural programmer model. Yih et al. (2015) use character tri-grams as inputs on both question and relation sides. Golub and He (2016) propose a generative framework for single-relation KBQA which predicts relation with a character-level sequenceto-sequence model.
Another difference between relation detection in KBQA and general RE is that general RE research assumes that the two argument entities are both available. Thus it usually benefits from features (Nguyen and Grishman, 2014; Gormley et al., 2015) or attention mechanisms (Wang et al., 2016) based on the entity information (e.g. entity types or entity embeddings). For relation detec-
tion in KBQA, such information is mostly missing because: (1) one question usually contains single argument (the topic entity) and (2) one KB entity could have multiple types (type vocabulary size larger than 1,500). This makes KB entity typing itself a difficult problem so no previous used entity information in the relation detection model.3
3 Improved KB Relation Detection
3.1 Different Granularity in KB Relations
Previous research formulates KB relation detection as a sequence matching problem. However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem. Here we give an overview of two types of relation sequence representations commonly used in previous work:
(1) Relation Name as a Single Token (relationlevel). In this case, each relation name is treated as a unique token. The problem with this approach is that it suffers from the low relation coverage due to limited amount of training data, thus cannot generalize well to large number of opendomain relations. For example in Figure 1, when treating relation names as single tokens, it will be difficult to match the questions to relation names “episodes written” and “starring roles” if these names do not appear in training data – their relation embeddings hrs will be random vectors thus are not comparable to question embeddings hqs.
(2) Relation as Word Sequence (word-level). In this case, the relation is treated as a sequence of words from the tokenized relation name. It has better generalization, but suffers from the lack of global information from the original relation names. For example in Figure 1(b), when doing only word-level matching, it is difficult to rank the target relation “starring roles” higher compared to the incorrect relation “plays produced”. This is because the incorrect relation contains word “plays”, which is more similar to the question (containing word “play”) in the embedding space. On the other hand, if the target relation co-occurs with questions related to “tv appearance” in training, by treating the whole relation as a token (i.e. relation id), we could better learn the correspondence between this token and phrases like “tv show” and “play on”.
3Such entity information has been used in KBQA systems as features for the final answer re-rankers.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Relation Token Question 1 Question 2what tv episodes were <e> the writer of what episode was written by <e> relation-level episodes written tv episodes were <e> the writer of episode was written by <e>
word-level episodes tv episodes episodewritten the writer of written
Table 1: An example of KB relation (episodes written) with two types of relation tokens (relation names and words), and two questions asking this relation. The topic entity is replaced with token <e> which could give the position information to the deep networks. The italics show the evidence phrase for each relation token in the question.
…
what tv show did <e> …
Bi-LSTM 1
Question Relation
…
Relation Representation
… Bi-LSTM 2
!"#$%&
…
…
…
(max-pooling) (max-pooling) Question Representation
Shortcut connections
(cosine similarity)
!'(
$%)
starring role seriesstarring_role series
Relation-Level Word-Level
Figure 2: The proposed Hierarchical Residual BiLSTM (HR-BiLSTM) model for relation detection. Note that without the dotted arrows of shortcut connections between two layers, the model will only compute the similarity between the second-layer of questions representations and the relation, thus is not doing hierarchical matching.
The two types of relation representation contain different levels of abstraction. As shown in Table 1, the word-level focus more on local information (words and short phrases), and the relation-level focus more on global information (long phrases and skip-grams) but suffer from data sparsity. Since the two levels of granularity both have their own pros and cons, we propose a hierarchical matching approach for KB relation detection: for a candidate relation, our approach matches the input question to both word-level and relation-level representations to get the final ranking score. Section 3.2 gives the details of our proposed approach.
3.2 Hierarchical Sequence Matching with Residual Learning for Relation Detection
This section describes our hierarchical matching approach. In order to match the question to different aspects of a relation (with different abstraction levels), we need to deal with the following problems on learning question/relation representations. Relation Representations from Different Granularity. We provide our model with both types of relation representation: word-level and relationlevel. Therefore, the input relation becomes r = {rword1 , · · · , rword M1 } [ {rrel1 , · · · , rrel M2 }, where the first M1 tokens are words (e.g. {episode, written}), and the last M2 tokens are relation names, e.g., {episode written} or {starring roles, series}
(when the target is a chain like in Figure 1(b)). We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations [Bword1:M1 : Brel1:M2 ] (each row vector i is the concatenation between forward/backward representations at i). We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations. We apply one max-pooling on these two sets of vectors and get the final relation representation hr.
Different Abstractions of Questions Representations. From Table 1, we can see that different parts of a relation could match different contexts of question texts. Usually relation names could match longer phrases in the question and relation words could match short phrases. Yet different words might match phrases of different lengths.
As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity. We deal with this problem by applying deep BiLSTMs on questions. The first-layer of BiLSTM works on the word embeddings of question words q = {q1, · · · , qN} and gets hidden representations (1)1:N = [ (1) 1 ; · · · ; (1) N
]. The second-layer BiLSTM works on (1)1:N to get the second set of hid-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
den representations (2)1:N . Since the second BiLSTM starts with the hidden vectors from the first layer, intuitively it could learn more general and abstract information compared to the first layer.
Hierarchical Matching between Relation and Question Now we have question contexts of different lengths encoded in (1)1:N and (2) 1:N . Unlike the standard usage of deep BiLSTMs that employs the representations in the final layer for prediction, here we expect that two layers of question representations can be complementary to each other and both should be compared to the relation representation space (Hierarchical Matching). This is important for our task since each relation token can correspond to phrases of different lengths, mainly because of syntactic variations. For example in Table 1, the relation word written could be matched to either the same single word in the question or a much longer phrase be the writer of.
We could perform the above hierarchical matching by computing the similarity between each layer of and hr separately and doing the (weighted) sum between the two scores. However this does not give significant improvement (see Table 2). Our analysis shows that this naive method suffers from the training difficulty, evidenced by that the converged training loss of this model is much higher than that of a single-layer baseline model. This is mainly because (1) Deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable, the training usually falls to a local optima where one layer has good matching scores and the other always has weight close to 0. (2) The training of deeper architectures itself is more difficult.
To overcome the above difficulties, we adopt the idea from Residual Networks (He et al., 2016) for hierarchical matching by adding shortcut connections between two BiLSTM layers. We proposed two ways of such Hierarchical Residual Matching: (1) Connecting each (1)
i and (2) i
, resulting in a 0
i = (1) i + (2) i
for each position i. Then the final question representation hq becomes a maxpooling over all 0
i s, 1iN . (2) Applying maxpooling on (1)1:N and (2) 1:N to get h (1) max and h(2) max
, respectively, then setting hq = h(1)
max + h(2) max
. Finally we compute the matching score of r given q as s
rel (r;q) = cos(hr,hq). Intuitively, the proposed method should benefit from hierarchical training since the second layer is
fitting the residues from the first layer of matching, so the two layers of representations are more likely to be complementary to each other. This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier.
During training we adopt a ranking loss to maximizing the margin between the gold relation r+ and other relations r in the candidate pool R.
lrel = max{0, srel(r+;q) + srel(r ;q)} where is a constant parameter. Fig 2 summarizes the above Hierarchical Residual BiLSTM (HR-BiLSTM) model. Remark: Another way of hierarchical matching consists in relying on attention mechanism, e.g. (Parikh et al., 2016), to find the correspondence between different levels of representations. This performs below the HR-BiLSTM (see Table 2).
4 KBQA Enhanced by Relation Detection
This section describes our KBQA pipeline system. We make minimal efforts beyond the training of the relation detection model, making the whole system easy to build.
Following previous work (Yih et al., 2015; Xu et al., 2016), our KBQA system takes an existing entity linker to produce the top-K linked entities, EL
K (q), for a question q (“initial entity linking”). Then we generate the KB queries for q following the four steps illustrated in Algorithm 1.
Algorithm 1: KBQA with two-step relation detection Input : Question q, Knowledge Base KB, the initial
top-K entity candidates ELK(q) Output: Top query tuple (ê, r̂, {(c, rc)})
1 Entity Re-Ranking (first-step relation detection): Use the raw question text as input for a relation detector to score all relations in the KB that are associated to the entities in ELK(q); use the relation scores to re-rank ELK(q) and generate a shorter list EL0K0(q) containing the top-K0 entity candidates (Section 4.1) 2 Relation Detection: Detect relation(s) using the reformatted question text in which the topic entity is replaced by a special token <e> (Section 4.2) 3 Query Generation: Combine the scores from step 1 and 2, and select the top pair (ê, r̂) (Section 4.3) 4 Constraint Detection (optional): Compute similarity between q and any neighbor entity c of the entities along r̂ (connecting by a relation rc) , add the high scoring c and rc to the query (Section 4.4).
Compared to previous approaches, the main difference is that we have an additional entity reranking step after the initial entity linking. We have this step because we have observed that entity
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
linking sometimes becomes a bottleneck in KBQA systems. For example, on SimpleQuestions the best reported linker could only get 72.7% top-1 accuracy on identifying topic entities. This is usually due to the ambiguities of entity names, e.g. in Fig 1(a), there are TV writer and baseball player “Mike Kelley”, which is impossible to distinguish with only entity name matching.
Having observed that different entity candidates usually connect to different relations, here we propose to help entity disambiguation in the initial entity linking with relations detected in questions.
Sections 4.1 and 4.2 elaborate how our relation detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection. The KBQA end task, as a result, benefits from this process.
4.1 Entity Re-Ranking
In this step, we use the raw question text as input for a relation detector to score all relations in the KB with connections to at least one of the entity candidates in EL
K (q). We call this step relation detection on entity set since it does not work on a single topic entity as the usual settings. We use the HR-BiLSTM as described in Sec. 3. For each question q, after generating a score s
rel (r; q) for each relation using HR-BiLSTM, we use the top l best scoring relations (Rl
q ) to re-rank the original entity candidates. Concretely, for each entity e and its associated relations R
e , given the original entity linker score s
linker , and the score of the most confident relation r 2 Rl
q
\R e , we sum these two scores to re-rank the entities:
srerank(e; q) =↵ · slinker(e; q) +(1 ↵) · max
r2Rlq\Re srel(r; q).
Finally, we select top K 0 <K entities according to score s
rerank to form the re-ranked list EL0 K
0(q). We use the same example in Fig 1(a) to illustrate the idea. Given the input question in the example, a relation detector is very likely to assign high scores to relations such as “episodes written”, “author of ” and “profession”. Then, according to the connections of entity candidates in KB, we find that the TV writer “Mike Kelley” will be scored higher than the baseball player “Mike Kelley”, because the former has the relations “episodes written” and “profession”. This method can be viewed as exploiting entity-relation collocation for entity linking.
4.2 Relation Detection
In this step, for each candidate entity e 2
EL
0 K
(q), we use the question text as the input to a relation detector to score all the relations r 2 R
e
that are associated to the entity e in the KB.4 Because we have a single topic entity input in this step, we do the following question reformatting: we replace the the candidate e’s entity mention in q with a token “<e>”. This helps the model better distinguish the relative position of each word compared to the entity. We use the HR-BiLSTM model to predict the score of each relation r 2 R
e : s
rel
(r; e, q).
4.3 Query Generation
Finally, the system outputs the <entity, relation (or core-chain)> pair (ê, r̂) according to:
s(ê, r̂; q) = max
e2EL0 K0 (q),r2Re
( · srerank(e; q)
+(1 ) · srel(r; e, q)) , where is a hyperparameter to be tuned.
4.4 Constraint Detection
Similar to (Yih et al., 2015), we adopt an additional constraint detection step based on text matching. Our method can be viewed as entitylinking on a KB sub-graph. It contains two steps: (1) Sub-graph generation: given the top scored query generated by the previous 3 steps5, for each node v (answer node or the CVT node like in Figure 1(b)), we collect all the nodes c connecting to v (with relation r
c ) with any relation, and generate a sub-graph associated to the original query. (2) Entity-linking on sub-graph nodes: we compute a matching score between each n-gram in the input question (without overlapping the topic entity) and entity name of c (except for the node in the original query) by taking into account the maximum overlapping sequence of characters between them (see Appendix A for details and B for special rules dealing with date/answer type constraints). If the matching score is larger than a threshold ✓ (tuned on training set), we will add the constraint entity c (and r
c ) to the query by attaching it to the corresponding node v on the core-chain.
4Note that the number of entities and the number of relation candidates will be much smaller than those in the previous step.
5Starting with the top-1 query suffers more from error propagation. However we still achieve state-of-the-art on WebQSP in Sec.5, showing the advantage of our relation detection model. We leave in future work beam-search and feature extraction on beam for final answer re-ranking like in previous research.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Accuracy Model Relation Input Views SimpleQuestions WebQSP AMPCNN (Yin et al., 2016) words 91.3 - BiCNN (Yih et al., 2015) char-3-gram 90.0 77.74 BiLSTM w/ words words 91.2 79.32 BiLSTM w/ relation names rel names 88.9 78.96 Hier-Res-BiLSTM (HR-BiLSTM) words + rel names 93.3 82.53
w/o rel name words 91.3 81.69 w/o rel words rel names 88.8 79.68 w/o residual learning (weighted sum on two layers) words + rel names 92.5 80.65 replacing residual with attention (Parikh et al., 2016) words + rel names 92.6 81.38 single-layer BiLSTM question encoder words + rel names 92.8 78.41 replacing BiLSTM with CNN (HR-CNN) words + rel names 92.9 79.08
Table 2: Accuracy on the SimpleQuestions and WebQSP relation detection tasks (test sets). The top shows performance of baselines. On the bottom we give the results of our proposed model together with the ablation tests.
5 Experiments
5.1 Task Introduction & Settings
We use the SimpleQuestions (Bordes et al., 2015) and WebQSP (Yih et al., 2016) datasets. Each question in these datasets is labeled with the gold semantic parse. Hence we can directly evaluate the relation detection performance independently as well as evaluate on the KBQA end task. SimpleQuestions (SQ): It is a single-relation KBQA task. The KB we use consists of a Freebase subset with 2M entities (FB2M) (Bordes et al., 2015), in order to compare with previous research. Yin et al. (2016) also evaluated their relation extractor on this data set and released their proposed question-relation pairs, so we run our relation detection model on their data set. For the KBQA evaluation, we also start with their entity linking results6. Therefore, our results can be compared with their reported results on both tasks. WebQSP (WQ): A multi-relation KBQA task. For this evaluation we use the full Freebase KB. We downloaded the S-MART (Yang and Chang, 2015) entity-linking results from (Yih et al., 2016)7. In order to evaluate the relation detection models, we create a new relation detection task from the WebQSP data set 8. For each question and its labeled semantic parse: (1) we first select the topic entity from the parse; and then (2) select all the relations and relation chains (length  2) connecting to the topic entity, and set the corechain labeled in the parse as the positive label and all the others as the negative examples.
We tune the following hyper-parameters on development sets: (1) the size of hidden state for LSTMs ({50, 100, 200, 400})9; (2) learning rate
6The two resources have been downloaded from https://github.com/Gorov/SimpleQuestions-EntityLinking
7 https://github.com/scottyih/STAGG 8We will make the datasets publicly available soon. 9For CNNs we double the size for fair comparison.
({0.1, 0.5, 1.0, 2.0}); (3) whether the shortcut connections are between hidden states or between max-pooling results (see Section 3.2); (4) The number of training epochs.
For both the relation detection experiments and the second-step relation detection in KBQA, we have entity replacement first (see Section 4.2 and Figure 1). All words are initialized by 300-d pretrained word embeddings (Mikolov et al., 2013).
5.2 Relation Detection Results
Table 2 shows the results on two relation detection tasks. The AMPCNN result is from (Yin et al., 2016), which got the previous state-ofthe-art by outperforming several attention-based methods. We re-implemented the BiCNN from (Yih et al., 2015), where both questions and relations are represented with the word hash trick on character tri-grams. The baseline BiLSTM with relation word sequence appears to be the best baseline on WebQSP and is close to the previous best result of AMPCNN on SimpleQuestions. Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p < 0.001 and 0.01 compared to the best baseline BiLSTM w/ words on SQ and WQ respectively).
Note that in the BiLSTM baselines, using only relation-names causes a great performance drop on SimpleQuestions compared to using only relation-words (91.2 to 88.9), while the gap is small on WebQSP. This shows that the SimpleQuestions suffers more from unseen relations while WebQSP questions can usually be answered by a fixed small set of relations and relation chains. Ablation Test: The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM. First, both tasks benefit from hierarchical matching between questions and both relation names and relations words, especially for SimpleQuestions (93.3% vs 91.2/88.8%). Second, residual
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
learning helps hierarchical matching compared to weighted-sum and attention baselines (see Section 3.2). For the attention baseline we tried the model from (Parikh et al., 2016) and its one-way variations, where the one-way model gives better results10. Note that residual learning significantly helps on WebQSP (80.65% to 82.53%), while it does not help as much on SimpleQuestions. On SimpleQuestions, even removing the deep layers does not change much the performance. WebQSP benefits more from residual and deeper architecture, possibly because in this dataset it is more important to handle larger scope of context matching.
Finally, on WebQSP, replacing BiLSTM with CNN in our hierarchical matching framework results in a large performance drop. Yet on SimpleQuestions the gap is small. We believe this is because the LSTM relation-encoder can better learn the composition of chains of relations in WebQSP, as it is better at dealing with longer dependencies. Remark: To verify that residual learning helps WebQSP because it helps the hierarchical architecture learn different levels of abstraction instead of only benefiting from combination of two BiLSTMs, we replace the deep BiLSTM question encoder with two single-layer BiLSTMs (both on words), with shortcut connections between their hidden states. This decreases accuracy to 76.11%, probably because of over-fitting (similar training accuracy compared to HR-BiLSTM). This proves that the residual and deep structures both contribute to the good performance of HR-BiLSTM.
5.3 KBQA End-Task Results
Table 3 compares our system with two published baselines (1) STAGG (Yih et al., 2015), the stateof-the-art on WebQSP11 and (2) AMPCNN (Yin et al., 2016), the state-of-the-art on SimpleQuestions. Since these two baselines are specially designed/tuned for one particular dataset, they lead to limited results when applied to the other dataset. In order to highlight the effect of different relation detection models on the KBQA end-task, we also implemented another baseline that uses our KBQA system but replaces HR-BiLSTM with our implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP) relation detectors (second block in Table 3).
Compared to the baseline relation detector (3rd 10We think the idea of hierarchical matching with attention mechanism may
work better for long sequences, and leave this for future work. 11The STAGG score on SQ is from (Bao et al., 2016).
row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves from 2 to 3% the KBQA end task (4th row). Note that in contrast to previous KBQA systems, our system does not use joint-inference or featurebased re-ranking step, nevertheless it still achieves better or comparable results to the state-of-the-art.
The third block of the table details two ablation tests for the proposed components in our KBQA systems: (1) Removing the entity re-ranking step significantly decreases the scores. Since the reranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways. Appendix C gives the detailed performance of the re-ranking step. (2) In contrast to the conclusion in (Yih et al., 2015), constraint detection is crucial for our system. This is probably because our joint performance on topic entity and core-chain detection is more accurate (77.5% top-1 accuracy), leaving a huge potential (77.5% vs. 58.0%) for the constraint detection module to improve.
Finally, like STAGG that uses multiple relation detectors (see (Yih et al., 2015) for the three models used), we also try to use the top-3 relation detectors from Section 5.2. As shown on the last row of Table 3, this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable to the state-of-the-art on WebQSP.
6 Conclusion
KB relation detection is a key step in KBQA and has significant differences from general relation extraction tasks. We propose a novel KB relation detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations. Our model outperforms the previous methods on KB relation detection tasks and allows our KBQA system to achieve state-of-the-arts.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and
Tiejun Zhao. 2016. Constraint-based question answering with knowledge graph. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, pages 2503–2514.
Hannah Bast and Elmar Haussmann. 2015. More accurate question answering on freebase. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. ACM, pages 1431–1440.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy
Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 1533– 1544.
Antoine Bordes, Nicolas Usunier, Sumit Chopra, and
Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075 .
Antoine Bordes, Nicolas Usunier, Alberto GarciaDuran, Jason Weston, and Oksana Yakhnenko.
2013. Translating embeddings for modeling multirelational data. In Advances in Neural Information Processing Systems. pages 2787–2795.
Zihang Dai, Lei Li, and Wei Xu. 2016. Cfo: Conditional focused neural question answering with largescale knowledge bases. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 800–810.
Cicero dos Santos, Bing Xiang, and Bowen Zhou.
2015. Classifying relations by ranking with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, pages 626–634.
Anthony Fader, Luke S Zettlemoyer, and Oren Etzioni.
2013. Paraphrase-driven learning for open question answering. In ACL (1). Citeseer, pages 1608–1618.
David Golub and Xiaodong He. 2016. Character-level
question answering with attention. arXiv preprint arXiv:1604.00727 .
Matthew R. Gormley, Mo Yu, and Mark Dredze. 2015.
Improved relation extraction with feature-rich compositional embedding models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1774– 1784.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 770–778.
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D
Forbus, and Ni Lao. 2016. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. arXiv preprint arXiv:1611.00020 .
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. pages 3111–3119.
Thien Huu Nguyen and Ralph Grishman. 2014. Employing word representations and regularization for domain adaptation of relation extraction. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Baltimore, Maryland, pages 68–74.
Ankur Parikh, Oscar Täckström, Dipanjan Das, and
Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2249–2255.
Bryan Rink and Sanda Harabagiu. 2010. Utd: Classifying semantic relations by combining lexical and semantic resources. In Proceedings of the 5th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Uppsala, Sweden, pages 256–259.
Ang Sun, Ralph Grishman, and Satoshi Sekine. 2011.
Semi-supervised relation extraction with large-scale word clustering. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, pages 521–529.
Ngoc Thang Vu, Heike Adel, Pankaj Gupta, and Hinrich Schütze. 2016. Combining recurrent and convolutional neural networks for relation classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 534–539.
Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan
Liu. 2016. Relation classification via multi-level attention cnns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1298–1307.
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. Question answering on freebase via relation extraction and textual evidence. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 2326–2336.
Yi Yang and Ming-Wei Chang. 2015. S-mart: Novel tree-based structured learning algorithms applied to
tweet entity linking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, pages 504–513.
Xuchen Yao, Jonathan Berant, and Benjamin
Van Durme. 2014. Freebase qa: Information extraction or semantic parsing? ACL 2014 page 82.
Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In ACL (1). Citeseer, pages 956–966.
Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and
Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Association for Computational Linguistics (ACL).
Wen-tau Yih, Xiaodong He, and Christopher Meek.
2014. Semantic parsing for single-relation question answering. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Baltimore, Maryland, pages 643–648.
Wen-tau Yih, Matthew Richardson, Chris Meek, MingWei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Berlin, Germany, pages 201–206.
Wenpeng Yin, Mo Yu, Bing Xiang, Bowen Zhou, and
Hinrich Schütze. 2016. Simple question answering by attentive convolutional neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1746–1756.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou,
and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. Dublin City University and Association for Computational Linguistics, Dublin, Ireland, pages 2335– 2344.
GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang.
2005. Exploring various knowledge in relation extraction. In Association for Computational Linguistics. pages 427–434.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen
Li, Hongwei Hao, and Bo Xu. 2016. Attentionbased bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Berlin, Germany, pages 207–212.

- Strengths: The paper addresses a relevant topic: learning the mapping between natural language and KB relations, in the context of QA (where we have only partial information for one of the arguments), and in the case of having a very large number of possible target relations.  The proposal consists in a new method to combine two different representations of the input text: a word level representation (i.e. with segmentation of the target relation names and also the input text), and relations as a single token (i.e without segmentation of relation names nor input text).   It seems, that the main contribution in QA is the ability to re-rank entities after the Entity Linking step.  Results show an improvement compared with the state of the art.   - Weaknesses: The approach has been evaluated in a limited dataset.   - General Discussion:  I think, section 3.1 fits better inside related work, so the 3.2 can become section 3 with the proposal. Thus, new section 3 can be splitted more properly.


From Characters to Words to in Between: Do We Capture Morphology?
Words can be represented by composing the representations of subword units such as word segments, characters, and/or character n-grams. While such representations are effective and may capture the morphological regularities of words, they have not been systematically compared, and it is not understood how they interact with different morphological typologies. On a language modeling task, we present experiments that systematically vary (1) the basic unit of representation, (2) the composition of these representations, and (3) the morphological typology of the language modeled. Our results largely confirm previous findings that character representations are effective across many languages, though we find that a previously unstudied combination of character trigram representations composed with bi-LSTMs outperforms most other settings. However, we also find room for improvement: character models do not match the predictive accuracy of a model with access to explicit morphological analyses.
1 Introduction
Continuous representations of words learned by neural networks are central to many NLP tasks (Cho et al., 2014; Chen and Manning, 2014; Dyer et al., 2015). However, directly mapping a finite set of word types to a continuous representation has well-known limitations. First, it makes a closed vocabulary assumption, enabling only generic out-of-vocabulary handling. Second, it cannot exploit systematic functional relationships in learning. For example, cat and cats stand in the same relationship as dog and dogs. While this re-
lationship might be discovered for these specific frequent words, it does not help us learn that the same relationship also holds for the much rarer words tarsier and tarsiers.
These functional relationships reflect the fact that words are composed from smaller units of meaning, or morphemes. For instance, cats consists of two morphemes, cat and -s, with the latter shared by the words dogs and tarsiers. Modeling this effect is crucial for languages with rich morphology, where vocabulary sizes are larger, many more words are rare, and many more such functional relationships exist. Hence, some models produce word representations as a function of subword units obtained from morphological segmentation or analysis (Luong et al., 2013; Botha and Blunsom, 2014; Cotterell and Schütze, 2015). However, a downside of these models is that they introduce a dependence on morphological segmentation or analysis.
But morphemes typically have similar orthographic representations across words. For example, the morpheme -s is realized as -es in finches, but this variation is limited, and the general relationship between morphology and orthography can be exploited by composing the representations of characters (Ling et al., 2015; Kim et al., 2016), character n-grams (Sperr et al., 2013; ?; Bojanowski et al., 2016; Botha and Blunsom, 2014), bytes (Plank et al., 2016; Gillick et al., 2016), or combinations thereof (dos Santos and Zadrozny, 2014; Qiu et al., 2014). These models can also represent rare and unknown words, and they produce compact parameterizations. They have the added appeal that they do not depend on morphological analysis, and they raise a provocative question: does NLP benefit from explicit modeling of morphology, or can this be replaced entirely by modeling of characters?
Our understanding of these models is incom-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
plete because they have been tested on different tasks, often compared only with direct word embeddings. A number of questions remain open:
1. How do representations based on morphemes compare with those based on characters? 2. What is the best way to compose subword representations? 3. Do character representations adequately substitute for morphological analysis? 4. How do different representations interact with languages of different morphological typologies?
The last question is relevant, since, as Bender (2013) note, languages are typologically diverse, and behaviors exhibited by a model on one language may differ radically on others. Most models implicitly assume concatenative morphology, but typology of many widely-spoken languages is primarily non-concatenative, and it is unclear such models will behave on these languages.
To answer these questions, we performed a systematic comparison across different models for the simple and ubiquitous task of language modeling. We present experiments that vary (1) the type of subword unit; (2) the composition function; and (3) morphological typology. To understand the extent to which character-level models capture true morphological regularities, we present oracle experiments using human morphological annotations instead of automatic morphological segments. Our results show that:
1. For most languages, character-level representations outperform the standard word representations. Most interestingly, a previously unstudied combination of character trigrams composed with bi-LSTMs performs best on the majority of languages. 2. Bi-LSTMs and CNNs are more effective composition functions than addition. 3. Character representations learn functional relationships between orthographically similar words, but are not as accurate as models with explicit knowledge of morphology. 4. Character-level models are effective across a range of morphological typologies, but orthography influences their effectiveness.
2 Morphological Typology
A morpheme is the smallest unit of meaning in a word. Some morphemes express core meaning (roots), while others express one or more depen-
dent features of the core meaning, such as person, gender, or aspect. A morphological analysis identifies the lemma and features of a word. A morph is the surface realization of a morpheme (Morley, 2000), which may vary from word to word. These distinctions are shown in Table 1.
Morphological typology classifies languages based on the processes by which morphemes are composed to form words. While most languages will exhibit a variety of such processes, for any given language, some processes are much more frequent than others, and we will broadly identify our experimental languages with these processes.
When morphemes are combined sequentially, the morphology is concatenative. However, morphemes can also be composed by nonconcatenative processes. We consider four broad categories of both concatenative and nonconcatenative processes in our experiments.
Fusional languages realize multiple features in a single concatenated morpheme. For example, English verbs can express number, person, and tense in a single morpheme:
wanted (English) want + ed
want + VB+1st+SG+Past Agglutinative languages assign one feature per morpheme. Morphemes are concatenated to form a word and the morpheme boundaries are clear. For example (Haspelmath, 2010):
okursam (Turkish) oku+r+sa+m
“read”+AOR+COND+1SG Root and Pattern Morphology forms words by inserting consonants and vowels of dependent morphemes into a consonantal root based on a given pattern. For example, the Arabic root ktb (“write”) produces (Roark and Sproat, 2007):
katab “wrote” (Arabic) takaatab “wrote to each other” (Arabic)
Reduplication is a process where a word form is produced by repeating part or all of the root to express new features. For example:
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
anak “child” (Indonesian) anak-anak “children” (Indonesian)
buah “fruit” (Indonesian) buah-buahan “various fruits” (Indonesian)
3 Representation Models
We compare ten different models, varying subword units and composition functions commonly used in recent work (Table 2). Given word w, we compute its representation w as:
w = f(Ws, σ(w)) (1)
where σ is a deterministic function that returns a sequence of subword units; Ws is a parameter matrix of representations for the vocabulary of subword units; and f is a composition function which takes σ(w) and Ws as input and returns w. All of the representations that we consider take this form, varying only in f and σ.
3.1 Subword Units
We consider four variants of σ in Equation 1, each returning a different type of subword unit: character, character trigram, or morph. Morphs are obtained from Morfessor (Smit et al., 2014) or a word segmentation based on Byte Pair Encoding (BPE; Gage (1994)), which has been shown to be effective for handling rare words in neural machine translation (Sennrich et al., 2016). BPE works by iteratively replacing frequent pairs of bytes with a single unused byte. For Morfessor, we use default parameters while for BPE we set the number of merge operations to 10,000.1 When we segment into character trigrams, we consider all trigrams in the word, including those covering notional beginning and end of word characters, as in Sperr et al. (2013). Example output of σ is shown in Table 3.
3.2 Composition Functions
We use three variants of f in Eq. 1. The first constructs the representation w of word w by adding the representations of subwords s1, . . . , sn = σ(w), where the representation of si is vector si.
w = n∑
i=1
si (2)
1BPE takes a single parameter: the number of merge operations. We tried different parameter values (1k, 10k, 100k) and manually examined the resulting segmentation on the English dataset. Qualitatively, 10k gave the most plausible segmentation and we used this setting across all languages.
The only subword unit that we don’t compose by addition is characters, since this will produce the same representation for many different words.
Our second composition function is a bidirectional long-short-term memory (bi-LSTM), which we adapt based on its use in the characterlevel model of Ling et al. (2015) and its widespread use in NLP generally. Given si and the previous LSTM hidden state hi−1, an LSTM (Hochreiter and Schmidhuber, 1997) computes the following outputs for the subword at position i:
hi = LSTM(si,hi−1) (3)
ŝi+1 = g(VT · hi) (4)
where ŝi+1 is the predicted target subword, g is the softmax function and V is a weight matrix.
A bi-LSTM (Graves et al., 2005) combines the final state of an LSTM over the input sequence with one over the reversed input sequence. Given the hidden state produced from the final input of the forward LSTM, hfwn and the hidden state produced from the final input of the backward LSTM hbw0 , we compute the word representation as:
wt = Wf · hfwn + Wb · hbw0 + b (5)
where Wf and Wb are parameter matrices and hfwn and hbw0 are forward and backward LSTM states.
The third composition function is a convolutional neural network (CNN) with highway layers, as in Kim et al. (2016). Let c1, . . . , ck be the sequence of characters of word w. The character embedding matrix is C ∈ Rd×k, where the i-th column corresponds to the embeddings of ci. We first apply a narrow convolution between C and a filter F ∈ Rd×n of width n to obtain a feature map f ∈ Rk−n+1. In particular, the computation of the j-th element of f is defined as
f [j] = tanh(〈C[∗, j : j + n− 1],F〉+ b) (6)
where 〈A,B〉 = Tr(ABT ) is the Frobenius inner product and b is a bias. The CNN model applies filters of varying width, representing features of character n-grams. We then calculate the maxover-time of each feature map.
yj = max j f [j] (7)
and concatenate them to derive the word representation wt = [y1, . . . , ym], where m is the number of filters applied. Highway layers allow some dimensions of wt to be carried or transformed. Since it can learn character n-grams directly, we only use the CNN with character input.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Models Subword Unit(s) Composition Function Sperr et al. (2013) words, character n-grams addition Luong et al. (2013) morphs (Morfessor) recursive NN Botha and Blunsom (2014) words, morphs (Morfessor) addition Qiu et al. (2014) words, morphs (Morfessor) addition dos Santos and Zadrozny (2014) words, characters CNN Cotterell and Schütze (2015) words, morphological analyses addition Sennrich et al. (2016) morphs (BPE) none Kim et al. (2016) characters CNN Ling et al. (2015) characters bi-LSTM Wieting et al. (2016) character n-grams addition Bojanowski et al. (2016) character n-grams addition Vylomova et al. (2016) characters, morphs (Morfessor) bi-LSTM, CNN Miyamoto and Cho (2016) words, characters bi-LSTM Lee et al. (2016) characters CNN
Table 2: Summary of previous work on representing words through compositions of subword units.
Unit Output of σ(wants) Morfessor ˆwant, s$ BPE ˆw, ants$ char-trigram ˆwa, wan, ant, ts$ character ˆ, w, a, n, t, s, $ analysis want+VB, +3rd, +SG, +Pres
Table 3: Input representations for wants
Figure 1: Our LSTM-LM architecture.
3.3 Language Model
We use language models (LM) because they are simple and fundamental to many NLP applications. Given a sequence of text s = w1, . . . , wT , our LM computes the probability of s as:
P (w1, . . . , wT ) = T∏ t=1 P (yt|w1, . . . , wt−1) (8)
where yt = wt if wt is in the output vocabulary and yt = UNK otherwise.
Our language model is an LSTM variant of recurrent neural network language (RNN) LM (Mikolov et al., 2010). At time step t, it receives input wt and predicts yt+1. Using Eq. 1, it first computes representation wt of wt. Given this representation and previous state ht−1, it produces a new state ht and predicts yt+1:
ht = LSTM(wt,ht−1) (9)
ŷt+1 = g(VT · ht) (10)
where g is a softmax function over the vocabulary yielding the probability in Equation 8. Note that this design means that we can predict only words from a fixed output vocabulary, so our models differ only in their representation of context words. This makes design it possible to compare language models using perplexity on a fixed output space, though open vocabulary word prediction is an interesting direction for future work.
The complete architecture of our system is shown in Figure 1, showing segmentation function σ and composition function f from Equation 1.
4 Experiments
We perform experiments on ten languages (Table 4). We use datasets from Ling et al. (2015) for English and Turkish. For Czech and Russian we use Universal Dependencies (UD) v1.3 (Nivre et al., 2015). For other languages, we use prepro-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Typology Languages #tokens #types Fusional Czech 1.2M 125.4K English 1.2M 81.1K Russian 0.8M 103.5K Agglutinative Finnish 1.2M 188.4K Japanese 1.2M 59.2K Turkish 0.6M 126.2K Root&Pattern Arabic 1.4M 137.5K Hebrew 1.1M 104.9K Reduplication Indonesian 1.2M 76.5K Malaysian 1.2M 77.7K
Table 4: Statistics of our datasets.
cessed Wikipedia data (Al-Rfou et al., 2013).2 For each dataset, we use approximately 1.2M tokens to train, and approximately 150K tokens each for development and testing. Preprocessing involves lowercasing (except for character models) and removing hyperlinks.
To ensure that we compared models and not implementations, we reimplemented all models in a single framework using Tensorflow (Abadi et al., 2015). We use a common setup for all experiments based on that of (Ling et al., 2015; Kim et al., 2016; Miyamoto and Cho, 2016). In preliminary experiments, we confirmed that our models produced similar patterns of perplexities for the reimplemented word and character LSTM models of Ling et al. (2015). Even following detailed discussion with Ling (p.c.), we were unable to reproduce their perplexities exactly—our English reimplementation gives lower perplexities; our Turkish higher—but we do reproduce their general result that character bi-LSTMs outperform word models. We suspect that different preprocessing and the stochastic learning explains differences in perplexities. Our final model with bi-LSTMs composition follows Miyamoto and Cho (2016) as it gives us the same results reported in their paper.
4.1 Training and Evaluation
Our LSTM-LM uses two hidden layers with 200 hidden units and representation vectors for words, characters, and morphs all have dimension 200. All parameters are initialized uniformly at random from -0.1 to 0.1, trained by stochastic gradient de-
2The Arabic and Hebrew dataset are unvocalized. Japanese mixes Kanji, Katakana, Hiragana, and Latin characters (for foreign words). Hence, a Japanese character can correspond to a character, syllable, or word. The preprocessed dataset is already word-segmented.
scent with mini-batch size of 32, time steps of 20, for 50 epochs. To avoid overfitting, we apply dropout with probability 0.5. For all models which do not use bi-LSTM composition, we start with a learning rate of 1.0 and decrease it by half if the validation perplexity does not decrease by 0.1 after 3 epochs. For models with bi-LSTMs composition, we use a constant learning rate of 0.2 and stop training when validation perplexity does not improve after 3 epochs. For the character CNN model, we use the same settings as the small model of Kim et al. (2016).
To make our results comparable to Ling et al. (2015), for each language we limit the output vocabulary to the most frequent 5,000 training words plus an unknown word token. To learn to predict unknown words, we follow Ling et al. (2015): in training, words that occur only once are stochastically replaced with the unknown token with probability 0.5. To evaluate the models, we compute perplexity on the test data.
5 Results and Analysis
Table 5 presents our main results. In six of ten languages, character-trigram representations composed with bi-LSTMs achieve the lowest perplexities. As far as we know, this particular model has not been tested before, though it is similar (but more general) than the model of Sperr et al. (2013). We can see that the performance of character, character trigrams, and BPE are very competitive. Composition by bi-LSTMs or CNN is more effective than addition, except for Turkish. We also observe that BPE always outperforms Morfessor, even for the agglutinative languages. We now turn to a more detailed analysis by morphological typology.
Fusional languages. For these languages, character trigrams composed with bi-LSTMs outperformed all other models, particularly for Czech and Russian (up to 20%), which is unsurprising since both are morphologically richer than English.
Agglutinative languages. We observe different results for each language. For Finnish, character trigrams composed with bi-LSTMs achieves the best perplexity. Surprisingly, for Turkish character trigrams composed via addition is best and addition also performs quite well for other representations, potentially useful since the addition function is simpler and faster than bi-LSTMs. We
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Language word character char trigrams BPE Morfessor
%imp bi-lstm CNN add bi-lstm add bi-lstm add bi-lstm
Czech 41.46 34.25 36.60 42.73 33.59 49.96 33.74 47.74 36.87 18.98 English 46.40 43.53 44.67 45.41 42.97 47.51 43.30 49.72 49.72 7.39 Russian 34.93 28.44 29.47 35.15 27.72 40.10 28.52 39.60 31.31 20.64 Finnish 24.21 20.05 20.29 24.89 18.62 26.77 19.08 27.79 22.45 23.09 Japanese 98.14 98.14 91.63 101.99 101.09 126.53 96.80 111.97 99.23 6.63 Turkish 66.97 54.46 55.07 50.07 54.23 59.49 57.32 62.20 62.70 25.24 Arabic 48.20 42.02 43.17 50.85 39.87 50.85 42.79 52.88 45.46 17.28 Hebrew 38.23 31.63 33.19 39.67 30.40 44.15 32.91 44.94 34.28 20.48 Indonesian 46.07 45.47 46.60 58.51 45.96 59.17 43.37 59.33 44.86 5.86 Malay 54.67 53.01 50.56 68.51 50.74 68.99 51.21 68.20 52.50 7.52
Table 5: Language model perplexities on test, showing improvement of the best system over words.
suspect that this is due to the fact that Turkish morphemes are reasonably short, hence wellapproximated by character trigrams. For Japanese, we observe that the improvements from character representation more modest than in other language.
Root and Pattern. For these languages, character trigrams composed with bi-LSTMs also achieve the best perplexity. We had wondered with CNNs would be more effective for root-andpatter morphology, but since these data are unvocalized, it is more likely that non-concatenative effects are minimized, though we do still find morphological variants with consonantal inflections that behave more like concatenation. For example, maktab (root:ktb) is written as mktb. We suspect this makes character trigrams quite effective since they match the tri-consonantal root patterns among words which share the same root.
Reduplication. For Indonesian, BPE morphs composed with bi-LSTMs model obtain the best perplexity. For Malay, the character CNN outperforms other models. However, these improvements are small compared to other languages. This likely reflects that Indonesian and Malay are only moderately inflected, where inflection involves both concatenative and non-concatenative processes.
5.1 Effects of Morphological Analysis
In the experiments above, we used unsupervised morphological segmentation as a proxy for morphological analysis (Table 3). However, as discussed in Section 2, this is quite approximate, so it is natural to wonder what would happen if we had an ideal morphological analysis. If character-
Languages Addition bi-LSTM Czech 51.8 30.07
Russian 41.82 26.44
Table 6: Perplexity results using hand-annotated morphological analyses (cf. Table 5).
level models were adequate to model the effects of morphology, then they would have similar predictive accuracy. To answer this question, we used the human-annotated morphological analyses provided in the UD datasets for Czech and Russian, the only languages in our set for which these analyses were available. In these experiments we treat the lemma and each morphological feature as a subword unit.
The results (Table 6) show that bi-LSTM composition of these representations outperforms all other models for both languages. These results demonstrate that neither character representations nor unsupervised segmentation is a perfect replacement for manual morphological analysis, at least in terms of predictive accuracy. Especially in light of character-level results, they also imply that current methods of unsupervised morphological analysis are inadequate substitutes for morphological analysis.
5.2 Automatic Morphological Analysis
The oracle experiments show promising results if we have annotated data. But these annotations are expensive, so investigated the effects of automatic morphological analysis. We obtained analyses for Arabic with the MADAMIRA (Pasha et al., 2014). As in the experiment using annotations, we treated each morphological feature as a sub-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
666
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
word unit. The resulting perplexities of 71.94 and 42.85 for addition and bi-LSTMs, respectively, are worse than those obtained with character trigrams (39.87), though it approaches the best perplexities.
5.3 Targeted Perplexity Results
A difficulty in interpreting the results of Table 5 with respect to specific morphological processes is that perplexity is measured for all words. But these processes do not apply to all words, so it may be that the effects of specific morphological processes are washed out. To get a clearer picture, we measured perplexity for only specific subsets of words in our test data: specifically, given target word wi, we measure perplexity of word wi+1. In other words, we analyze the perplexities when the inflected words of interest are in the most recent history, exploiting the recency bias of our LSTM-LM. This is the perplexity most likely to be strongly affected by different representations, since we do not vary representations of the predicted word itself.
We look at several cases: nouns and verbs in Czech and Russian, where word classes can be identified from annotations, and reduplication in Indonesian, which we can identify mostly automatically. For each analysis, we also distinguish between frequent cases, where the inflected word occurs more than ten times in the training data, and rare cases, where it occurs fewer than ten times. We compare only bi-LSTM models.
For Czech and Russian, we again use the UD annotation to identify words of interest. The results (Table 7), show that manual morphological analysis uniformly outperforms other subword models, with an especially strong effect for Czech nouns, suggesting that other models do not capture useful predictive properties of a morphological analysis. We do however note that character trigrams achieve low perplexities in most cases, similar to overall results (Table 5). We also observe that the subword models are more effective for rare verbs.
For Indonesian, we exploit the fact that the hyphen symbol ‘-’ typically separates the first and second occurrence of a reduplicated morpheme, as in the examples of Section 2. We use the presence of word tokens containing hyphens to estimate the percentage of those exhibiting reduplication. As shown in Table 8, the numbers are quite low.
Table 9 shows results for reduplication. In
Model all frequent rare word 101.71 91.71 156.98 characters 99.21 91.35 137.42 BPE 117.2 108.86 156.81
Table 9: Average perplexities of words that occur after reduplicated words in the test set.
contrast with the overall results, BPE bi-LSTMs model produce the worse perplexities, while character bi-LSTMs produce the best, suggesting that these models are more effective for reduplication.
Looking more closely at BPE segmentation of reduplicated words, we found that only 6 of 252 reduplicated words have a correct word segmentation, with the reduplicated morpheme often being combined differently with the notional startof-word or hyphen character. One the other hand BPE correctly learns 8 out of 9 Indonesian prefixes and 4 out of 7 Indonesian suffixes.3 This analysis supports our intuition that the improvement from BPE might come from its modeling of concatenative morphology.
3We use Indonesian preprefixes, prefixes and suffixes listed in Larasati et al. (2011)
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Model Frequent Words Rare Words OOV wordsman including relatively unconditional hydroplane uploading foodism
word
person like extremely nazi molybdenum - - anyone featuring making fairly your - - children include very joints imperial - -
men includes quite supreme intervene - -
BPE ii called newly unintentional emphasize upbeat vigilantism LSTM hill involve never ungenerous heartbeat uprising pyrethrumtext like essentially unanimous hybridized handling pausanias netherlands creating least unpalatable unplatable hand-colored footway char- mak include resolutely unconstitutional selenocysteine drifted tuaregs trigrams vill includes regeneratively constitutional guerrillas affected quft LSTM cow undermining reproductively unimolecular scrofula conflicted subjectivism
maga under commonly medicinal seleucia convicted tune-up
char- mayr inclusion relates undamaged hydrolyzed musagte formulas LSTM many insularity replicate unmyelinated hydraulics mutualism formallymary includes relativity unconditionally hysterotomy mutualists fecal may include gravestones uncoordinated hydraulic meursault foreland char- mtn include legislatively unconventional hydroxyproline unloading fordism CNN mann includes lovely unintentional hydrate loading dadaismjan excluding creatively unconstitutional hydrangea upgrading popism nun included negatively untraditional hyena upholding endemism
Table 10: Top nearest neighbours (cosine similarities) of semantically and syntactically similar words.
Query Top nearest neighbours kota-kota wilayah-wilayah (areas), pulau-pulau (islands), negara-negara (countries), (cities) bahasa-bahasa (languages), koloni-koloni (colonies)
berlembah-lembah berargumentasi (argue), bercakap-cakap (converse), berkemauan (will), (have many valleys) berimplikasi (imply), berketebalan (have a thickness)
Table 11: Top nearest neighbours (cosine similarities) of Indonesian reduplicated words.
5.4 Qualitative Analysis
Table 10 presents nearest neighbors under cosine similarity for in-vocabulary, rare, and out-ofvocabulary (OOV) words.4 For frequent words, standard word embeddings are clearly superior for lexical meaning. Character and morph representations tend to find words that are orthographically similar, suggesting that they better at modeling dependent than root morphemes. The same pattern holds for rare and OOV words. We suspect that the subword models outperform words on language modeling because they exploit affixes to signal word class. We also noticed similar patterns in Japanese.
We analyze reduplication by querying reduplicated words to find their nearest neighbors using the BPE bi-LSTM model. If the model were sensitive to reduplication, we would expect to see the morphological variants word in the top-n nearest neighbors. However, from Table 11, this is not so. With the partially reduplicated query berlembahlembah, we do not find the lemma lembah.
4https://radimrehurek.com/gensim/
6 Conclusion
We presented a systematic comparison of word representation models with different levels of morphological awareness, across languages with different morphological typologies. Our results confirm previous findings that character-level models are effective for many languages, but these models do not match the predictive accuracy of model with explicit knowledge of morphology, and our qualitative analysis suggests that they learn orthographic similarity of affixes, and lose the meaning of root morphemes.
Although morphological analyses are available in limited quantities, our results suggest that there might be utility in semi-supervised learning from partially annotated data. Across languages with different typologies, our experiments show that the subword unit models are most effective on agglutinative languages. However, these results do not generalize to all languages, since factors such as morphology and orthography affect the utility of these representations. We plan to explore these effects in future work.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

tldr: The authors compare a wide variety of approaches towards sub-word modelling in language modelling, and show that modelling morphology gives the best results over modelling pure characters. Further, the authors do some precision experiments to show that the biggest benefit towards sub-word modelling is gained after words typically exhibiting rich morphology (nouns and verbs). The paper is comprehensive and the experiments justify the core claims of the paper.   - Strengths:  1) A comprehensive overview of different approaches and architectures towards sub-word level modelling, with numerous experiments designed to support the core claim that the best results come from modelling morphemes.  2) The authors introduce a novel form of sub-word modelling based on character tri-grams and show it outperforms traditional approaches on a wide variety of languages.  3) Splitting the languages examined by typology and examining the effects of the models on various typologies is a welcome introduction of linguistics into the world of language modelling.  4) The analysis of perplexity reduction after various classes of words in Russian and Czech is particularly illuminating, showing how character-level and morpheme-level models handle rare words much more gracefully. In light of these results, could the authors say something about how much language modelling requires understanding of semantics, and how much it requires just knowing various morphosyntactic effects?  - Weaknesses:  1) The character tri-gram LSTM seems a little unmotivated. Did the authors try other character n-grams as well? As a reviewer, I can guess that character tri-grams roughly correspond to morphemes, especially in Semitic languages, but what made the authors report results for 3-grams as opposed to 2- or 4-? In addition, there are roughly 26^3=17576 possible distinct trigrams in the Latin lower-case alphabet, which is enough to almost constitute a word embedding table. Did the authors only consider observed trigrams? How many distinct observed trigrams were there?  2) I don't think you can meaningfully claim to be examining the effectiveness of character-level models on root-and-pattern morphology if your dataset is unvocalised and thus doesn't have the 'pattern' bit of 'root-and-pattern'. I appreciate that finding transcribed Arabic and Hebrew with vowels may be challenging, but it's half of the typology.  3) Reduplication seems to be a different kind of phenomenon to the other three, which are more strictly morphological typologies. Indonesian and Malay also exhibit various word affixes, which can be used on top of reduplication, which is a more lexical process. I'm not sure splitting it out from the other linguistic typologies is justified.  - General Discussion:  1) The paper was structured very clearly and was very easy to read.  2) I'm a bit puzzled about why the authors chose to use 200 dimensional character embeddings. Once the dimensionality of the embedding is greater than the size of the vocabulary (here the number of characters in the alphabet), surely you're not getting anything extra?  -------------------------------  Having read the author response, my opinions have altered little. I still think the same strengths and weakness that I have already discussed hold.