Time Expression Analysis and Recognition Using Syntactic Types and Simple Heuristic Rules
Extracting time expressions from free text is a fundamental task for many applications. We analyze the time expressions from four datasets and observe that only a small group of words are used to express time information, and the words in time expressions demonstrate similar syntactic behaviour. Based on the observations, we propose a type-based approach, named SynTime, to recognize time expressions. Specifically, we define three main syntactic types, namely time token, modifier, and numeral, to group time-related regular expressions over tokens. On the types we design simple heuristic rules to recognize time expressions. In recognition, SynTime first identifies the time tokens from raw text, then searches their surroundings for modifiers and numerals to form time segments, and finally merges the time segments to time expressions. As a light-weight rule-based tagger, SynTime runs in real time, and can be easily expanded by simply adding keywords for the text of different types and of different domains. Experiment results show that SynTime outperforms state-of-the-art methods on benchmark datasets and tweets data.
1 Introduction
Time expression plays an important role in information retrieval and many applications in natural language processing (Alonso et al., 2011). Recognizing time expressions from free text has attracted considerable attention since last decade (Verhagen et al., 2007, 2010; UzZaman et al., 2013).
Time expression recognition main involves two
kinds of methods, rule-based method (Strotgen and Gertz, 2013; Strötgen and Gertz, 2010; Chang and Manning, 2012) and machine learning based method (Bethard, 2013; Lee et al., 2014). Rulebased time expression taggers could recognize most time expressions with carefully designed rules, while they could not recognize the time expressions that are not matched in any explicit rule. Machine learning based methods require training data for good performance, and may not recognize less frequent time expressions.
In our study, we analyze the time expressions in four datasets: TimeBank (Pustejovsky et al., 2003b), Gigaword (Parker et al., 2011), WikiWars (Mazur and Dale, 2010), and Tweets. From the analysis, we make four observations. First, most time expressions are very short, with 80% of time expressions containing no more than three tokens. Second, the vocabulary used to express time information is very small, with a small group of keywords. Third, at least 93% of time expressions contain at least one time token. The last observation is that words in time expressions demonstrate similar syntactic behaviors. All the observations relate to the principle of least effort (Zipf, 1949). That is, people will act under the least effort in order to minimize the cost of energy at both individual level and collective level to language usage (Zipf, 1949). Time expression is part of language and acts as an interface of communication. Short expressions, small vocabulary, occurrence, and similar syntactic behaviors all reduce the cost of energy required to communicate.
Based on the observations, we propose a typebased approach, named SynTime (‘Syn’ is from Syntactic), to recognize time expressions. Specifically, we define 3 main types, namely time token, modifier, and numeral, to group time-related regular expressions over tokens. Time tokens are the words that explicitly express time information,
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
such as time units (e.g., ‘month’). Modifiers modify time tokens; they may appear before or after time tokens, e.g., ‘several’ and ‘ago’ in ‘several months ago.’ Numeral are ordinals and numbers. From free text SynTime first identifies time tokens, and then recognizes modifiers and numerals.
Naturally, SynTime is a rule-based tagger. The key difference between SynTime and other rulebased taggers lies in the way of defining types and the way of designing rules. The type definition of SynTime is inspired by part-of-speech in which “linguists group some words of language into classes (sets) which show similar syntactic behaviour.” (Manning and Schutze, 1999) SynTime defines types for tokens according to their syntactic behaviors. Other rule-based taggers define types for tokens based on their semantic meaning. For example, SUTime defines 5 semantic modifier types, such as frequency modifiers; 1 while SynTime defines 5 syntactic modifier types, such as modifiers that appear before time tokens. (see Section 4.1 for details.) Accordingly, other rule-based taggers design rules for each type based on their meanings and deal with each type separately. SynTime designs rules based on the token types and their relative positions in time expressions. That is why we call SynTime a type-based approach. More importantly, other rule-based taggers design rules in a fixed way, including fixed length and fixed position. In contrast, SynTime designs rules in a heuristic method based on the idea of boundary expansion. The heuristic rules are quite simple that it makes SynTime much more flexible, expansible, and extremely light-weight, leading SynTime to run in real time.
We evaluate SynTime against three state-of-theart baselines, namely, HeidelTime, SUTime, and UWTime, on three datasets, namely, TimeBank, WikiWars, and Tweets. TimeBank and WikiWars are benchmark datasets for time expression extraction. 2 Experiment results show that SynTime significantly outperforms the three state-of-the-art methods on TimeBank and Tweets datasets. On WikiWars, SynTime achieves comparable results. More importantly, SynTime achieves the best recalls on all three datasets and exceptionally good results on Tweets dataset. To sum up, we make the following contributions.
1 https://github.com/stanfordnlp/CoreNLP/tree/
master/src/edu/stanford/nlp/time/rules 2Gigaword dataset was not used in our experiments because the labels in the dataset was automatically generated by other taggers which may not be the ground truth labels.
• We analyze the time expressions from four datasets and make four observations. The observations provide evidence in terms of time expression for the principle of least effort (Zipf, 1949). We design SynTime based on the observations.
• We propose a type-based time expression tagger, SynTime, that defines syntactic types and designs simple heuristic rules for time expression recognition. SynTime provides an idea to simplify rule-based time tagger.
• We conduct experiments on three datasets, and the results demonstrate the effectiveness of SynTime against state-of-the-art baselines.
2 Related Work
Many research on time expression identification are reported in TempEval exercises (Verhagen et al., 2007, 2010; UzZaman et al., 2013). The task is divided into two subtasks: Recognition and normalization.
Rule-based Time Expression Recognition. Rule-based time taggers like GUTime, HeidelTime, and SUTime, predefine time-related words and rules (Verhagen et al., 2005; Strötgen and Gertz, 2010; Chang and Manning, 2012). HeidelTime (Strötgen and Gertz, 2010) hand-crafts rules with time resources like weekdays and months, and leverages language clues like part-of-speech to identify time expression. SUTime (Chang and Manning, 2012) designs fixed rules using a cascade finite automata (Hobbs et al., 1997) on regular expressions over tokens (Chang and Manning, 2014). It first identifies individual words, then expands them to chunks, and finally to time expressions. Rule-based taggers achieve very good results in TempEval exercises.
SynTime is also a rule-based tagger while the key differences between SynTime and other rulebased taggers are the way of defining types and of designing rules. SynTime defines types for tokens according to their syntactic behaviors and designs rules in a heuristic way.
Machine Learning based Method. Machine learning based methods extract features from the text and apply statistical models on the features for recognizing time expressions. Example features include character features, word features, syntactic features, semantic features, and gazetteer
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
features (Llorens et al., 2010; Filannino et al., 2013; Bethard, 2013). The statistical models include Markov Logic Network, Logistic Regression, Support Vector Machines, Maximum Entropy, and Conditional Random Fields (Llorens et al., 2010; UzZaman and Allen, 2010; Filannino et al., 2013; Bethard, 2013). Some models obtain good performance, and even achieve the highest F1 of 82.71% on strict match in TempEval3 (Bethard, 2013).
Outside TempEval exercises, Angeli et al. leverage compositional grammar and employ a EM-style approach to learn a latent parser for time expression recognition (Angeli et al., 2012). In the method named UWTime, Lee et al. handcraft a Combinatory Categorial Grammar (CCG) (Steedman, 1996) to define a set of lexicon with rules and use L1-regularization to learn linguistic context (Lee et al., 2014). The two methods explicitly use linguistic information. Particulaly in (Lee et al., 2014), CCG could capture rich structure information of language, similar to the rule-based methods. Tabassum et al. focus on resolving the dates in tweets, and use distant supervision to recognize time expressions (Tabassum et al., 2016). They use five time types and assign one of them to each word, which is similar to SynTime in the way of defining types over tokens. However, they focus only on the type of date, while SynTime can recoginize all the time expressions and does not involve learning and runs in real time.
Time Expression Normalization. Methods in TempEval exercises design rules for time expression normalization (Verhagen et al., 2005; Strötgen and Gertz, 2010; Llorens et al., 2010; UzZaman and Allen, 2010; Filannino et al., 2013; Bethard, 2013). Because the rule systems has high similarity, Llorens et al. suggest to construct a large knowledge base as a public resource for the task (Llorens et al., 2012). Some researchers treat the normalization process as a learning task and use machine learning methods (Lee et al., 2014; Tabassum et al., 2016). Lee et al. (Lee et al., 2014) use AdaGrad algorithm (Duchi et al., 2011) and Tabassum et al. (Tabassum et al., 2016) use a loglinear algorithm to normalize the time expressions.
SynTime focuses only on time expression recognition, and the normalization could be achieved by using methods similar to the existing rule systems, because they are highly similar (Llorens et al., 2012).
3 Time Expression Analysis
3.1 Dataset
We conduct analysis on four datasets: TimeBank, Gigaword, WikiWars, and Tweets. TimeBank (Pustejovsky et al., 2003b) is a benchmark dataset in TempEval series (Verhagen et al., 2007, 2010; UzZaman et al., 2013), consisting of 183 news articles. Gigaword (Parker et al., 2011) is a large automatically labeled dataset with 2,452 news articles and is used in TempEval-3. WikiWars dataset is derived from Wikipedia articles about wars (Mazur and Dale, 2010). Tweets is our manually annotated dataset with 942 tweets of which each contains at least one time expression. Table 1 summarizes the datasets.
3.2 Observation
From the four datasets, we analyze their time expressions and make four observations.
Observation 1 Time expressions are very short. More than 80% of time expressions contain no more than three words and more than 90% contain no more than four words.
Figure 1 plots the length distribution of time expressions. Although the texts are collected from different sources (i.e., news articles, Wikipedia articles, and tweets) and vary in sizes, the length of time expressions follow a similar distribution. In particular, the one-word time expressions range
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Table 2: The percentage of time expressions that contain at least one time token, and the average length of time expressions.
Dataset Percent Average Length TimeBank 95.15 2.00 Gigaword 97.69 1.70 WikiWars 93.18 2.38 Tweets 97.81 1.51
Table 3: Number of distinct words and number of distinct time tokens in time expressions.
Dataset #Words #Time Tokens TimeBank 130 64 Gigaword 214 80 WikiWars 224 74 Tweets 107 64
from 36.23% in WikiWars to 62.91% in Tweets. In informal communication people tend to use words in minimum length to express time information. The third column in Table 2 reports the average length of time expressions. On average, time expressions contain about two words.
Observation 2 Only a small group of time-related keywords are used to express time information.
From the time expressions in all four datasets, we observe that the group of keywords used to express time information is small.
Table 3 reports the number of distinct words and of distinct time tokens. The words/tokens are manually normalized before counting and their variants are ignored. For example, ‘month’ and ‘5mons’ are counted as one token ‘month.’ Numerals in the counting are ignored. Despite the different sizes in the four datasets, the numbers of distinct time tokens are comparable.
Observation 3 More than 93% of time expressions contain at least one time token.
The second column in Table 2 reports the percentage of time expressions that contain at least one time token. Observe that at least 93.18% of time expressions contain time token(s), which suggests that to recognize the time expressions, it is essential to recognize their time tokens.
Observation 4 Part-of-speech (POS) could not distinguish time expressions from common words, but within time expressions, POS can help distinguish their constituents.
For each dataset we list the top 10 POS tags that appear in time expressions, and their percentages over whole text. Among the 40 tags (10 × 4 datasets), 36 have percentage lower than 10%; other 4 are CD. This indicates that POS could not provide enough information to distinguish time expressions from common words. However, the most common POS tags in time expressions are NN*, JJ, RB, CD, and DT. Within time expressions, the time tokens usually have NN* and RB, the modifiers have JJ and RB, and the numerals have CD. This observation indicates that for the time expressions, their similar constituents behave in similar syntactic way. When seeing this, we realize that this is exactly how linguists define part-of-speech for language. 3 The definition of POS for language inspires us to define a syntactic type system for the time expression, part of language.
The four observations all relate to the principle of least effort (Zipf, 1949). That is, people will act under the least effort so as to minimize the cost of energy at both individual and collective level to the language usage (Zipf, 1949). Time expression is part of language and acts as an interface of communication. Short expressions, small vocabulary, occurrence, and similar syntactic behaviors all reduce the cost of energy required to communicate.
To summarize: On average, a time expression contains two tokens of which one is time token and the other is modifier/numeral, and the size of time tokens is small. To recognize a time expression, therefore, we first recognize the time token, then recognize the modifier/numeral.
4 SynTime: Syntactic Types and Simple Heuristic Rules
Figure 2 shows the overview of SynTime. Shown in the left-hand side of the figure, SynTime is initialized with regular expressions over tokens. After initialization SynTime can be directly applied on text, without training. On the other hand, SynTime can be easily expanded by adding timerelated token regular expressions from training text. The expansion enables SynTime to recognize time expressions in text from different types and from different domains.
Shown in the right-hand side of Figure 2, SynTime recognizing time expression includes three main steps. In the first step, SynTime identifies
3“linguists group some words of language into classes (sets) which show similar syntactic behaviour.” (Manning and Schutze, 1999)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 2: Overview of SynTime. Left-hand side shows the construction of SynTime, with initialization using token regular expressions, and optional expansion using training text. Right-hand side shows the main steps that SynTime recognizes time expressions.
the time tokens from POS-tagged raw text. Then around the time tokens SynTime searches for modifiers and numerals to form time segments. In the last step, SynTime transforms the time segments to time expressions.
4.1 SynTime Construction
We define a syntactic type system for time expression, specifically, 15 types for time tokens, 5 types for modifiers, and 1 type for numeral.
Time Token. We define 15 types for the time tokens and use their names similar to JodaTime classes: 4 DECADE (-), YEAR (-), SEASON (5), MONTH (12), WEEK (7), DATE (-), TIME (-), DAY TIME (27), TIMELINE (12), HOLIDAY (20), PERIOD (9), DURATION (-), TIME UNIT (15), TIME ZONE (6), and ERA (2). Number in ‘()’ indicates the number of distinct tokens in this type, without counting variants. ‘-’ indicates the type involves changing digits and cannot be counted.
Modifier. We define 3 types for the modifiers according to their possible positions relative to time tokens. Modifiers that appear before time tokens are PREFIX (48); modifiers after time tokens are SUFFIX (2). LINKAGE (4) link two time tokens. Besides, we define 2 special modifier types, COMMA (1) for comma ‘,’ and IN ARTICLE (2) for indefinite articles ‘a’ and ‘an.’
4 http://www.joda.org/joda-time/
TimeML (Pustejovsky et al., 2003a) and TimeBank (Pustejovsky et al., 2003b) do not treat most prepositions like ‘on’ as a part of time expressions. Thus SynTime does not collect those prepositions.
Numeral. Number in time expressions can be a time token e.g., ‘10’ in ‘October 10, 2016,’ or a modifier e.g., ‘10’ in ‘10 days.’ We define NUMERAL (-) for the ordinals and numbers.
SynTime Initialization. The token regular expressions for initializing SynTime are collected from SUTime,5 a state-of-the-art rule-based tagger that achieved the highest recall in TempEval3 (Chang and Manning, 2012, 2013). Specifically, we collect from SUTime only the tokens and the regular expressions over tokens, and discard its other rules of recognizing full time expressions.
4.2 Time Expression Recognition
On the types, SynTime designs simple heuristic rules to recognize time expressions. The recognition process includes three main steps: (1) Time token identification, (2) time segment identification, and (3) time expression extraction.
4.2.1 Time Token Identification
Identifying Time tokens is simple and straightforward, through matching of string and regular expression. Some words might cause ambiguity. For example ‘May’ could be a modal verb, or the fifth month of year. To filter out the ambiguous words, we use POS information. In implementation, we use Stanford POS Tagger, 6 and the POS tags for matching the instances of SynTime types are based on our Observation 4 in Section 3.2.
4.2.2 Time Segment Identification
The task of time segment identification is to search the surrounding of each time token identified in previous step for the modifiers and numerals, then gather the time token with its modifiers and numerals to form a time segment. The searching is under simple heuristic rules in which the key idea is to expand the time token’s boundaries.
At first, each time token is a time segment. If it is either a PERIOD or DURATION, then no need to further search. Otherwise, search its left and its right for modifiers and numerals. For the left searching, if encounter a PREFIX or NUMERAL or IN ARTICLE, then continue searching. For the right
5 https://github.com/stanfordnlp/CoreNLP/tree/
master/src/edu/stanford/nlp/time/rules 6 http://nlp.stanford.edu/software/tagger.shtml
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
the/PREFIX last/PREFIX week/TIME_UNIT … said Friday/WEEK s1 s2
e1 s1
(a) Stand-alone time segment to time expression.
s1 s2
s1
the/PREFIX third/NUMERAL quarter/TIME_UNIT of/PREFIXT 1984/YEAR
(b) Merge adjacent time segments.
s1 s2
s1
January/MONTH 13/NUMERAL 1951/YEAR
(c) Merge overlapping time segments.
s1 s2
s1
June/MONTH 30/NUMERAL,/COMMA 1990/YEAR
(d) Merge overlapping time segments.
s1 s2
e1 s1
8/NUMERAL to/LINKAGE 20/NUMERAL days/TIME_UNIT
(e) Dependent time segment and time segment.
Figure 3: Example time segments and time expressions. The labels above are from time segment identification; the labels below are for time expression extraction.
searching, if encounter a SUFFIX or NUMERAL, then continue searching. Both the left and the right searching will stop when reaching a COMMA or LINKAGE or non-modifier/numeral word. The left searching will not exceed the previous time token; the right searching will not exceed the next time token. A time segment consists of exactly one time token, and zero or more modifiers/numerals.
A special kind of time segments do not contain any time token; they depend on other time segments next to them. For example, in ‘8 to 20 days,’ ‘to 20 days’ is a time segment, and ‘8 to’ forms a dependent time segment. (see Figure 3(e).)
4.2.3 Time Expression Extraction
The task of time expression extraction is to extract time expressions from the identified time segments in which the core step is to determine whether to merge two adjacent or overlapping time segments into a new time segment.
We scan the time segments in a sentence from beginning to the end. A stand-alone time segment is a time expression. (see Figure 3(a).) The main focus is to deal with two or more time segments that are adjacent and overlapping. If two time segments s1 and s2 are adjacent, merge them to form
a new time segment s1. (see Figure 3(b).) Consider that s1 and s2 overlap at a shared boundary. According to our time segment identification, the shared boundary could be a modifier or a numeral. If the word at the shared boundary is neither a COMMA nor a LINKAGE, then merge s1 and s2. (see Figure 3(c).) If the word is a LINKAGE, then extract s1 as a time expression and continue scanning. When the shared boundary is a COMMA, merge s1 and s2 only if the COMMA’s previous token and its next token satisfy the three conditions: (1) The previous token is a time token or a NUMERAL; (2) the next token is a time token; and (3) the types of the previous token and of the next token are not the same. (see Figure 3(d).)
4.3 SynTime Expansion
SynTime expansion requires the words to be added to be annotated manually. We apply the initial SynTime on the time expressions from training text and list the words that are not covered. Whether the uncovered words are added to SynTime is manually determined. The rule for determination is that the added words can not cause ambiguity and should be generic. WikiWars dataset contains a few examples like this: ‘The time Arnold reached Quebec City.’ Words in this example are extremely descriptive, and we do not collect them. In tweets, on the other hand, people may use words’ abbreviations and informal variants, for example, ‘2day’ and ‘tday’ are popular spellings of ‘today.’ Such kind of abbreviations and informal variants will be collected.
According to our observations, not many words are used to express time information, the manual addition of keywords thus will not cost much. In addition, we find that even in tweets people tend to use formal words. In the Twitter word clusters trained from 56 million English tweets, 7 the most often used words are the formal words, and their frequencies are much greater than informal words’. The cluster of ‘today,’ 8 for example, its most often use is the formal one, ‘today,’ which appears 1,220,829 times; while its second most often use ‘2day’ appears only 34,827 times. The low rate of informal words (e.g., about 3% in ‘today’ cluster) suggests that even in informal environment the manual keyword addition costs little.
7 http://www.cs.cmu.edu/˜ark/TweetNLP/cluster_
viewer.html 8 http://www.cs.cmu.edu/˜ark/TweetNLP/paths/ 01111110010.html
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
5 Experiments
We conduct experiments on three datasets and compare SynTime with three state-of-the-art baselines: HeidelTime, SUTime, and UWTime. For SynTime we report the results of its two versions: SynTime-I and SynTime-E. SynTime-I is the initial version, and SynTime-E is the expanded version by adding keywords to SynTime-I.
5.1 Experiment Setting
Datasets. We use two benchmark datasets, TimeBank and WikiWars, and one manually labeled Tweets dataset. Section 3.1 shows details of TimeBank and WikiWars datasets. The Tweets dataset is collected from Twitter. We randomly sample 4000 tweets and use SUTime to tag them. 942 tweets of which each contains at least one time expression. From the remaining 3,058 tweets, we randomly sample 500 and manually annotate them, and find that only 15 tweets contain time expressions. Thus we roughly consider that SUTime misses about 3% time expressions in tweets. We then manually annotate the 942 tweets, according to the standards of TimeML and TimeBank, and get 1,127 manually labeled time expressions. For the 942 tweets, we randomly sample 200 tweets as test set, and the rest 742 as training set, because a baseline UWTime requires training.
Baseline Methods. We compare SynTime with methods: HeidelTime (Strötgen and Gertz, 2010), SUTime (Chang and Manning, 2012), and UWTime (Lee et al., 2014). HeidelTime and SUTime both are rule-based methods, and UWTime is a learning method. When training UWTime on Tweets, we try two settings: (1) Train with only Tweets training set; (2) train with TimeBank and Tweets training set. The second setting achieves slightly better result and we report that result.
Evaluation Metrics. We follow TempEval-3 and use their evaluation toolkit 9 to report Precision, Recall, and F1 in terms of strict match and relaxed match (UzZaman et al., 2013).
5.2 Experiment Result
Table 4 reports the overall perfermance. Among the 18 measures, SynTime-I and SynTime-E achieve 12 best results and 13 second best results. Except the strict match on WikiWars dataset,
9 http://www.cs.rochester.edu/˜naushad/tempeval3/
tools.zip
both SynTime-I and SynTime-E achieve F1 above 91%. For the relaxed match on all three datasets, SynTime-I and SynTime-E achieve recalls above 92%. The high recalls are consistent with our observation that at least 93.18% of time expressions contain time token(s). (see Table 2.) This indicates that SynTime covers most of time tokens. On Tweets dataset, SynTime-I and SynTime-E achieve exceptionally good performance. Their F1 reach 91.74% with 11.37% improvement in strict match and 95.87% with 6.33% improvement in relaxed match. The reasons are that in informal environment people tend to use time expressions in minimum length, (62.91% of one-word time expressions in Tweets; see Figure 1.) the size of time keywords is small, (only 60 distinct time tokens; see Table 3.) and even in tweets people tend to use formal words. (see Section 4.3 for our finding from Twitter word clusters.) For precision, SynTime achieves comparable results in strict match and performs slightly poorer in relaxed match.
5.2.1 SynTime-I v.s. Baseline Methods
On TimeBank dataset, SynTime-I achieves F1 of 92.09% in strict match and of 94.96% in relaxed match. On Tweets, SynTime-I achieves 91.74% and 95.87%, respectively. It outperforms all the baseline methods. The reason is that for the rulebased time taggers, their rules are designed in a fixed way, lacking flexibility. For example, SUTime could recognize ‘1 year’ but not ‘year 1.’ For the machine learning based methods, some of the features they used actually hurt the modelling. Time expressions involve quite many changing numbers which in themselves affect the pattern recognition. For example, it is difficult to build connection between ‘June 30, 1990’ and ‘October 12, 2008’ at the level of word or of character. One suggestion is to consider a type-based learning method that could use type information. For example, the above two time expressions refer to the same pattern of ‘MONTH NUMERAL COMMA YEAR’ at the level of type. POS is a kind of type information. But according to our analysis, POS could not distinguish time expressions from common words. Features need carefully designing. On WikiWars, SynTime-I achieves competitive results in both matches. Time expressions in WikiWars include lots of prepositions and quite a few descriptive time expressions. SynTime could not fully recognize such kinds of time expressions because it follows TimeML and TimeBank.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Table 4: Overall performance. The best results are in bold face and the second best are underlined. Some results are borrowed from their original papers and the papers are indicated by the references.
Dataset Method Strict Match Relaxed Match Pr. Re. F1 Pr. Re. F1
TimeBank
HeidelTime(Strotgen et al., 2013) 83.85 78.99 81.34 93.08 87.68 90.30 SUTime(Chang and Manning, 2013) 78.72 80.43 79.57 89.36 91.30 90.32
UWTime(Lee et al., 2014) 86.10 80.40 83.10 94.60 88.40 91.40 SynTime-I 91.43 92.75 92.09 94.29 95.65 94.96 SynTime-E 91.49 93.48 92.47 93.62 95.65 94.62
WikiWars
HeidelTime(Lee et al., 2014) 85.20 79.30 82.10 92.60 86.20 89.30 SUTime 78.61 76.69 76.64 95.74 89.57 92.55
UWTime(Lee et al., 2014) 87.70 78.80 83.00 97.60 87.60 92.30 SynTime-I 80.00 80.22 80.11 92.16 92.41 92.29 SynTime-E 79.18 83.47 81.27 90.49 95.39 92.88
Tweets
HeidelTime 89.58 72.88 80.37 95.83 77.97 85.98 SUTime 76.03 77.97 76.99 88.43 90.68 89.54 UWTime 88.54 72.03 79.44 96.88 78.81 86.92 SynTime-I 89.52 94.07 91.74 93.55 98.31 95.87 SynTime-E 89.20 94.49 91.77 93.20 98.78 95.88
Table 5: Number of time tokens and modifiers for expansion
Dataset #Time Tokens #Modifiers TimeBank 4 5 WikiWars 16 19 Tweets 3 2
5.2.2 SynTime-E v.s. SynTime-I
Table 5 lists the number of time tokens and modifiers added to SynTime-I to get SynTime-E.
On TimeBank and Tweets datasets, only a few tokens are added, the corresponding results are affected slightly. This confirms the small size of time words and the high coverage of SynTime. On WikiWars, relatively more tokens are added, SynTime-E performs much better than SynTimeI, especially in recall. It improves the recall by 3.25% in strict match and by 2.98% in relaxed match. This indicates that with more words added from specific domains (e.g., WikiWars about war), SynTime can improve the performance.
5.3 Limitations
SynTime assumes that words are tokenized and POS tagged correctly. In reality, however, the tokenized and tagged words are not that perfect, due to the limit of used tools. For example, Stanford POS Tagger assigns VBD to the word ‘sat’ in ‘friday or sat’ while whose tag should be NNP. The incorrect tokens and POS tags affect the result.
6 Conclusion and future work
We conduct an analysis on the time expressions from four datasets, and observe that time expressions in general are very short and expressed by a small vocabulary, and words in time expressions demonstrate similar syntactic behavior. Our observations provide evidence in terms of time expression for the principle of least effort (Zipf, 1949). Inspired by part-of-speech, based on the observations, we define a syntactic type system for the time expression, and propose a type-based time expression tagger, named by SynTime. SynTime defines syntactic types for tokens and on the types it designs simple heuristic rules based on the idea of boundary expansion. Experiments on three datasets show that SynTime outperforms the state-of-the-art baselines, including rule-based time taggers and machine learning based time tagger. As an extremely light-weight rule-based tagger, SynTime runs in real time. SynTime provides an idea to simplify the rule-based time tagger.
Time expression is part of language and follows the principle of least effort (Zipf, 1949). Since language usage relates to human habits (Chomsky, 1986; Pinker, 1995), we might expect that humans would share some common habits, and therefore expect that other languages and other parts of language would more or less follow the same principle. In the future we will try our analytical method on other languages and other parts of language.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper describes a rule based approach to time expression extraction. Its key insights are time expressions typically are short and contain at least 1 time token. It first recognizes the time token through a combination of dictionary lookup, regular expression match with POS tagging information. It then expands the time segment from either direction of the time token until it reaches based on a set of heuristic rules. Finally, it merges the time segments into a single time expression based on another set of rules. Evaluation of this approach with both rule based & ML based systems on 3 data sets show significant improvements.  - Strengths:  It's well written and clearly presented. The rules are motivated by empirical observations of the data, and seems to be well justified as evidenced by the evaluation.   - Weaknesses:   There are some underspecification in the paper that makes it difficult to reproduce the results. See below for details.  - General Discussion:  * Section 4.1: what are there 5 seasons? What about things such as Ramadan month or Holiday Season? * Section 5.1:  two benchmark datasets  =>  three datasets ? * Section 5.2: an example without time token will be helpful. * Section 5.2: given this approach is close to the ceiling of performance since 93% expressions contain time token, and the system has achieved 92% recall, how do you plan to improve further? * Is there any plan to release the full set of rules software used?


An End-to-End Model for Question Answering over Knowledge Base with Cross-Attention Combining Global Knowledge
With the rapid growth of knowledge bases (KBs) on the web, how to take full advantage of them becomes increasingly important. Knowledge base-based question answering (KB-QA) is one of the most promising approaches to access the substantial knowledge. Meantime, as the neural network-based (NN-based) methods develop, NN-based KB-QA has already achieved impressive results. However, previous work did not put emphasis on question representation, and the question is converted into a fixed vector regardless of its candidate answers. This simple representation strategy is unable to express the proper information of the question. Hence, we present an end-to-end neural network model to represent the questions and their corresponding scores dynamically according to the various candidate answer aspects via cross-attention mechanism. In addition, we leverage the global knowledge inside the underlying KB, aiming at integrating the rich KB information into the representation of the answers. Moreover, it also alleviates the out-ofvocabulary (OOV) problem, which helps the cross-attention model to represent the question more precisely. The experimental results on WebQuestions demonstrate the effectiveness of the proposed approach.
1 Introduction
As the amount of the knowledge bases (KBs) grows, people are paying more attention to seeking effective methods for accessing these precious intellectual resources. There are several tailor-made languages designed for querying KBs, such as
SPARQL (Prudhommeaux and Seaborne, 2008). However, to handle such query languages, users are required to not only be familiar with the particular language grammars, but also be aware of the architectures of the KBs. By contrast, knowledge base-based question answering (KB-QA) (Unger et al., 2014), which takes natural language as query language, is a more user-friendly solution, and has become a research focus in recent years.
Given natural language questions, the goal of KB-QA is to automatically return answers from the KB. There are two mainstream research directions for this task: semantic parsing-based (SPbased) (Zettlemoyer and Collins, 2009, 2012; Kwiatkowski et al., 2013; Cai and Yates, 2013; Berant et al., 2013; Yih et al., 2015, 2016; Reddy et al., 2016) and information retrieval-based (IR-based) (Yao and Van Durme, 2014; Bordes et al., 2014a,b, 2015; Dong et al., 2015; Xu et al., 2016a,b) methods. SP-based methods usually focus on constructing a semantic parser that could convert natural language questions into structured expressions like logical forms. IR-based methods usually search answers from the KB based on the information conveyed in questions, where ranking techniques are often adopted to make correct selections from candidate answers.
Recently, with the progress of deep learning, neural network-based (NN-based) methods have been introduced to the KB-QA task (Bordes et al., 2014b). They belong to IR-based methods. Different from previous methods, NN-based methods represent both of the questions and the answers as semantic vectors. Then the complex process of KB-QA could be converted into a similarity matching process between an input question and its candidate answers in a semantic space. The candidates with the highest similarity score will be selected as the final answers. Because they are adaptive and robust, NN-based methods have at-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
tracted more and more attention, and this paper also focuses on using end-to-end neural networks to answer questions over knowledge base.
In NN-based methods, the crucial step is to compute the similarity score between a question and a candidate answer, where the key is to learn their representations. Previous methods put more emphasis on learning representation of the answer end. For example, Bordes et al. (2014a) consider the importance of the subgraph of the candidate answer. Dong et al. (2015) make use of the context and the type of the answer. However, the representation of the question end is oligotrophic. Existing approaches often represent a question into a single vector using simple bag-of-words (BOW) model (Bordes et al., 2014a,b), whereas the relatedness to the answer end is neglected. We argue that a question should be represented differently according to the different focuses of various answer aspects1.
Take the question “Who is the president of France?” and one of its candidate answers “Francois Hollande” as an example. When dealing with the answer entity Francois Holland, “president” and “France” in the question is more focused, and the question representation should bias towards the two words. While facing the answer type /business/board member, “Who” should be the most prominent word. Meantime, some questions may value answer type more than other answer aspects. While in some other questions, answer relation may be the most important information we should consider, which is dynamic and flexible corresponding to different questions and answers. Obviously, this is an attention mechanism, which reveals the mutual influences between the representation of questions and the corresponding answer aspects.
We believe that such kind of representation is more expressive. Dong et al. (2015) represents questions using three CNNs with different parameters when dealing with different answer aspects including answer path, answer context and answer type. We argue that simply selecting three independent CNNs is mechanical and inflexible. Thus, we go one step further, and propose a crossattention based neural network to perform KBQA. The cross-attention model, which stands for the mutual attention between the question and the
1An answer aspect could be the answer entity itself, the answer type, the answer context, etc.
answer aspects, contains two parts: the answertowards-question attention part and the questiontowards-answer attention part. The former help learn flexible and adequate question representation, and the latter help adjust the question-answer weight, getting the final score. We illustrate in section 3.2 for more details. In this way, we formulate the cross-attention mechanism to model the question answering procedure. Note that our proposed model is an entire end-to-end approach which only depends on training data. Some integrated systems which use extra patterns and resources are not directly comparable to ours. Our target is to explore a better solution following the end-to-end KB-QA technical path.
Moreover, we notice that the representations of the KB resources (entities and relations) are also limited in previous work. specifically, they are often learned barely on the QA training data, which results in two limitations. 1) The global information of the KB is deficient. For example, if question-answer pair (q, a) appears in the training data, and the global KB information implies us that a′ is similar to a2, denoted by (a∼ a′), then (q, a′) is more probable to be right. However, current QA training mechanism cannot guarantee (a ∼ a′) could be learned. 2) The problem of out-ofvocabulary (OOV) stands out. Due to the limited coverage of the training data, the OOV problem is common while testing, and many answer entities in testing candidate set have never been seen before. The attention of these resources become the same because they shared the same OOV embedding, and this will do harm to the proposed attention model. To tackle these two problems, we additionally incorporates KB itself as training data for training embeddings besides original questionanswer pairs. In this way, the global structure of the whole knowledge could be captured, and the OOV problem could be alleviated naturally.
In summary, the contributions are as follows. 1) We present a novel cross-attention based NN model tailored to KB-QA task, which considers the mutual influence between the representation of questions and the corresponding answer aspects. 2) We leverage the global KB information, aiming at represent the answers more precisely. It also alleviates the OOV problem, which is very helpful to the cross-attention model.
2The complete KB is able to offer this kind of information, e.g., a and a′ share massive context.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
3) The experimental results on the open dataset WebQuestions demonstrate the effectiveness of the proposed approach.
2 Overview
The goal of KB-QA task could be formulated as follows. Given a natural language question q, the system returns an entity set A as answers. The architecture of our proposed KB-QA system is shown in Figure 1, which illustrates the basic flow of our approach. First, we identify the topic entity of the question, and generate candidate answers from Freebase. Then, a cross-attention based neural network is employed to represent the question under the influence of the candidate answer aspects. Finally, the similarity score between the question and each corresponding candidate answer is calculated, and the candidates with highest score will be selected as the final answers3.
Cross-Attention based Neural Network
q: Who is the president of France?
Topic entity France Candidate
generation
Candidate Answers
Paris 𝑎1
French 𝑎2
Semi-presidential system
⋮
Answer Aspects
answer entity:/m/05qtj answer relation: capital
answer type: /location/city town answer context:/m/0276jx2, /m
/0jd4j, /m/0f3vz, …
⋮
S(𝑞, 𝑎1) S(𝑞, 𝑎2 )
⋮ ⋮
⋯
A
Ranking
Freebase
Figure 1: The overview of the proposed KB-QA system.
We utilize Freebase (Bollacker et al., 2008) as our knowledge base. It has more than 3 billion facts, and is used as the supporting KB for many QA tasks. In Freebase, the facts are represented by subject-predicate-object triples (s, p, o). For clarity, we call each basic element a resource, which could be either an entity or a relation. For example, (/m/0f8l9c, location.country.capital,/m/05qtj)4 describes the fact that the capital of France is Paris, where /m/0f8l9c and /m/05qtj are entities denoting France and Paris respectively, and location.country.capital is a relation.
3We also adopt a margin strategy to obtain multiple answers for a question and this will be explained in the next section.
4Note that the Freebase prefixes are omitted for neatness.
3 Our Approach
3.1 Candidate Generation
All the entities in Freebase should be candidate answers ideally, but in practice, this is time consuming and not really necessary. For each question q, we use Freebase API (Bollacker et al., 2008) to identify a topic entity, which could be simply understood as the main entity of the question. For example, France is the topic entity of question “Who is the president of France?”. Freebase API method is able to resolve as many as 86% questions if we use the top1 result (Yao and Van Durme, 2014). After getting the topic entity, we collect all the entities directly connected to it and the ones connected with 2-hop5. These entities constitute a candidate set Cq .
3.2 The Neural Cross-Attention Model
We present a cross-attention based neural network, which represents the question dynamically according to different answer aspects, also considering their connections. Concretely, each aspect of the answer focuses on different words of the question and thus decides how the question is represented. Then the question pays different attention to each answer aspect to decide their weights. Figure 2 is the architecture of our model. We will illustrate how the system works as follows.
3.2.1 Question Representation
First of all, we have to obtain the representation of each word in the question. These representations retain all the information of the question, and could serve the following steps. Suppose question q is expressed as q = (x1, x2, ..., xn), where xi denotes the ith word. As shown in Figure 2, we first look up a word embedding matrix Ew ∈ Rd×vw to get the word embeddings, which is randomly initialized, and updated during the training process. Here, d means the dimension of the embeddings and vw denotes the vocabulary size of natural language words.
Then, the embeddings are fed into a long shortterm memory (LSTM) (Hochreiter and Schmidhuber, 1997) networks. LSTM has been proven to be effective in many natural language processing (NLP) tasks such as machine translation (Sutskever et al., 2014) and dependency parsing (Dyer et al., 2015), and it is adept in harnessing long
5For example, (/m/0f8l9c, governing officials, government.position held.office holder, /m/02qg4z) is a 2-top connection.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
𝑥1𝑞 𝑥2 𝑥6𝑥5𝑥3 𝑥4 𝑎𝑡𝑎𝑟𝑎𝑒 𝑎𝑐
Word Embedding Matrix 𝐸𝑤 KB Embedding Matrix 𝐸𝑘
ℎ1 ℎ2 ℎ3 ℎ4 ℎ5 ℎ6 𝑞
𝛼1 𝛼2 𝛼3 𝛼4 𝛼5 𝛼6
𝑞1A-Q Attention Model
𝑒𝑐
+ + + = 𝑆(𝑞, 𝑎)
𝛽1 𝛽2 𝛽3 𝛽4
Q-A Attention Model
Mean
over time
Bidirectional LSTM
𝑒𝑡𝑒𝑟𝑒𝑒
Figure 2: The architecture of the proposed attention-based neural network. Note that only one aspect(in orange color) is depicted for clarity. The other three aspects follow the same way.
sentences. Note that if we use unidirectional LSTM, the outcome of a specific word contains only the information of the words before it, whereas the words after it are not taken into account. To avoid this, we employ bidirectional LSTM as Bahdanau (2015) does, which consists of both forward and backward networks. The forward LSTM handles the question from left to right, and the backward LSTM processes in the reverse order. Thus, we could acquire two hidden state sequences, one from the forward one ( −→ h1, −→ h2, ..., −→ hn) and the other from the backward one ( ←− h1, ←− h2, ..., ←− hn). We concatenate the forward hidden state and the backward hidden state of each word, resulting in hj = [ −→ hj ; ←− hj ]. The hidden unit of forward and backward LSTM is d2 , so the concatenated vector is of dimension d. In this way, we obtain the representation of each word in the question.
3.2.2 Answer aspect representation
We directly use the embedding for each answer aspect through the KB embedding matrix Ek ∈ Rd×vk . Here, vk means the vocabulary size of the KB resources. The embedding matrix is randomly initialized and learned during training, and could be further enhanced with the help of global information as described in Section 3.3. Concretely, we employ four kinds of answer aspects: answer entity ae, answer relation ar, answer type at and answer context ac. Their embeddings are denoted as ee, er, et and ec, respectively. It is worth noting that the answer context consists of multiple KB resources, and we denote it as (c1, c2, ..., cm). We first acquire their KB embeddings (ec1 , ec2 , ..., ecm) throughEk, then calculate
an average embedding by ec = 1m ∑m
i=1 eci . 3.2.3 Cross-Attention model The most crucial part of the proposed approach is the cross-attention mechanism. The crossattention mechanism is composed of two parts: the answer-towards-question attention part and the question-towards-answer attention part. • Answer-towards-question(A-Q) attention
Based on our assumption, each answer aspect should focus on different words of the same question. The extent of attention can be measured by the relatedness between each word representation hj and an answer aspect embedding ei. We propose the following formulas to calculate the weights.
αij = exp(ωij) n∑ k=1 exp(ωik) (1)
ωij = f(W T [hj ; ei] + b) (2)
Here, αij denotes the weight of attention from answer aspect ei to the jth word in the question, where ei ∈ {ee, er, et, ec}. f(·) is a non-linear activation function, such as hyperbolic tangent transformation here. Let n be the length of the question. W ∈ R2d×d is an intermediate matrix and b is the offset. Both of them are randomly initialized and updated during training. Subsequently, according to the specific answer aspect ei, the attention weights are employed to calculate a weighted sum of the hidden representations, resulting in a semantic vector that represent the question.
qi = n∑ j=1 αijhj (3)
The similarity score of the question q and this particular candidate answer aspect ei (ei ∈ {ee, er, et, ec}) could be defined as follows.
S (q, ei) = h(qi, ei) (4)
The scoring function h(·) is computed as the inner product between the sentence representation qi, which has already carried the attention from the answer aspect part, and the corresponding answer aspect ei, and is parametrized into the network and updated during the training process. • Question-towards-answer(Q-A) attention
Intuitively, different question should value the four answer aspect differently. Since we have already calculated the scores of (q, ei), we define the final similarity score of the question q and each candidate answer a as follows.
S (q, a) = ∑
ei∈{ee,er,et,ec}
βei · S (q, ei) (5)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
βei = exp (ωei)∑
ek∈{ee,er,et,ec} exp (ωek)
(6)
ωei = f ( WT · [q; ei] + b ) (7)
q = 1
n ∑n j hj (8)
Here βei denotes the attention of question towards answer aspects, indicating which answer aspect should be more focused in one (q, a) pair. W ∈ R2d×d is also a intermediate matrix as in the answer-towards-question attention part, and b is an offset value.6 q is calculated by averagely pooling all the bi-directional LSTM hidden state sequences, resulting a vector which represents the question to determine which answer aspect should be more focused.
The proposed cross-attention model could also be intuitively interpreted as a re-reading mechanism (Hermann et al., 2015). Our aim is to select correct answers from a candidate set. When we judge a candidate answer, suppose we first look at its type, and we will re-read the question to find out which part of the question should be more focused (handling attention). Then we go to next aspect and re-read the question again, until the all the aspects are utilized. After we read all the answer aspects and get all the scores, the final similarity score between question and answer should be a weighted sum of all the scores. We believe that this mechanism is beneficial for the system to better understand the question with the help of the answer aspects, and it may lead to a performance promotion.
3.2.4 Training
We first construct the training data. Since we have (q, a) pairs as supervision data, candidate set Cq can be divided into two subsets, namely, correct answer set Pq and wrong answer set Nq. For each correct answer a ∈ Pq, we randomly select k wrong answers a′ ∈ Nq as negative examples. For some topic entities, there may be not enough wrong answers to acquire k wrong answers. Under this circumstance, we extend Nq from other randomly selected candidate set C ′q. With the generated training data, we are able to make use of pairwise training. The training loss is given as follows, which is a hinge loss.
Lq,a,a′ = [γ + S (q, a ′)− S (q, a)]+ (9)
6Note that the W and b in the two attention part is different and independent.
where γ is a positive real number that ensure a margin between positive and negative examples. And [z]+ means max(0, z). The intuition of this training strategy is to guarantee the score of positive question-answer pairs be higher than negative ones with a margin. The objective function is as follows.
min ∑ q 1 |Pq| ∑ a∈Pq ∑ a′∈Nq Lq,a,a′ (10)
We adopt stochastic gradient descent (SGD) to implement the learning process, shuffled minibatches are utilized.
3.2.5 Inference
In testing stage, given the candidate answer setCq, we have to calculate S(q, a) for each a ∈ Cq, and find out the maximum value Smax.
Smax = argmax a∈Cq
{S (q, a)} (11)
It is worth noting that many questions have more than one answer, so it is improper to set the candidate answer which have the maximum value as the final answer. Instead, we take advantage of the margin γ. If the score of an candidate answer is within the margin compared with Smax, we put it in the final answer set.
A = {â|Smax − S (q, â) < γ} (12)
3.3 Combining Global Knowledge
In this section, we elaborate how the global information of a KB could be leveraged. As stated before, we try to take into account the complete knowledge information of the KB. To this end, we adopt TransE model (Bordes et al., 2013) and integrate its outcome into our training process. In TransE, relations are considered as translations in the embedding space. For consistency, we denote each fact as (s, p, o). TransE utilizes pairwise training strategy as well. Randomly sampled corrupted facts (s′, p, o′) are the negative examples. The distance measure d(s + p, o) is defined as ‖s+ p− o‖22. And the training loss is given as follows. Lk =
∑ (s,p,o)∈S ∑ (s′,p,o′)∈S′ [γk + d (s+ p, o)− d ( s′ + p, o′ ) ] +
(13) Where S is the set of KB facts and S′ is the corrupted facts. In our QA task, we filter out the completely unrelated facts to save time. Specifically, we first collect all the topic entities of all the questions as initial set. Then, we expand the set by adding directly connected and 2-hop entities. Finally, all the facts containing these entities form
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
the positive set, and the negative facts are randomly corrupted. This is a compromising solution due to the large scale of Freebase. To employ the global information in our training process, we adopt a multi-task training strategy. Specifically, we perform KB-QA training and TransE training in turn. After each epoch of KB-QA training, 100 epochs of TransE training are conducted, and the embeddings of the KB resources are shared and updated during both training processes. The proposed training process ensures that the global KB information act as additional supervision, and the interconnections among the resources are fully considered. In addition, as more KB resources are involved, the OOV problem is relieved. Since all the OOV resources have exactly the same attention towards a question, it will weaken the effectiveness of the attention model. So the alleviation of OOV is able to bring additional benefits to the attention model.
4 Experiments
To evaluate the proposed method, we conduct experiments on WebQuestions (Berant et al., 2013) dataset that includes 3,778 question-answer pairs for training and 2,032 for testing. The questions are collected from Google Suggest API, and the answers are labeled manually by Amazon MTurk. All the answers are from Freebase. We use threequarter of the training data as training set, and the left as validate set. We use F1 score as evaluation matric, and the average result is computed by the script provided by Berant et al. (2013).
Note that our proposed approach is an entire end-to-end method, which totally depends on training data. It is worth noting that Yih et al. (2015; 2016) achieves much higher F1 scores than other methods. Their staged system is able to address more questions with constraints and aggregations. However, their approach applies numbers of manually designed rules and features, which come from the observations on the training set questions. These particular manual efforts reduce the adaptability of their approach. Moreover, there are some integrated systems such as Xu et al. (2016a; 2016b) achieve higher F1 scores which leverage Wikipedia free text as external knowledge, so their systems are not directly comparable to ours.
4.1 Settings
For KB-QA training, we use mini-batch stochastic gradient descent to minimize the pairwise train-
ing loss. The minibatch size is set to 100. The learning rate is set to 0.01. Both the word embedding matrix Ew and KB embedding matrix Ev are normalized after each epoch. The embedding size d = 512, then the hidden unit size is 256. Margin γ is set to 0.6. Negative example number k = 2000. We set the embedding dimension to 512 in TransE training process, and the minibatch size is also 100. γk is set to 1. All these hyperparameters of the proposed network is determined according to the performance on the validate set.
4.2 Results The effectiveness of the proposed approach
To demonstrate the effectiveness of the proposed approach, we compare our method with state-of-the-art end-to-end NN-based methods.
Table 1 shows the results on WebQuestions dataset. Bordes et al. (2014b) apply BOW method to obtain a single vector for both questions and answers. Bordes et al. (2014a) further improve their work by proposing the concept of subgraph embeddings. Besides the answer path, the subgraph contains all the entities and relations connected to the answer entity. The final vector is also obtained by bag-of-words strategy. Yang et al. (2014) follow the SP-based manner, but uses embeddings to map entities and relations into KB resources, then the question can be converted into logical forms. They jointly consider the two mapping processes. Dong et al. (2015) use three columns of CNNs to represent questions corresponding to three aspects of the answers, namely the answer context, the answer path and the answer type. Bordes et al. (2015) put KB-QA into the memory networks framework (Sukhbaatar et al., 2015), and achieves the state-of-the-art performance of endto-end methods. Our approach employs bidirectional LSTM, cross-attention model and global KB information.
From the results, we observe that our approach achieves the best performance of all the end-to-end
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
methods on WebQuestions. Bordes et al. (2014b; 2014a; 2015) all utilize BOW model to represent the questions, while ours takes advantage of the attention of answer aspects to dynamically represent the questions. Also note that Bordes et al. (2015) uses additional training data such as Reverb (Fader et al., 2011) and their original dataset SimpleQuestions. Dong et al. (2015) employs three fixed CNNs to represent questions, while ours is able to express the focus of each unique answer aspect to the words in the question. Besides, our approach employs the global KB information. So, we believe that the results faithfully shows that the proposed approach is more effective than the other competitive methods. Model Analysis
In this part, we further discuss the impacts of the components of our model. Table 2 indicates the effectiveness of different parts in the model.
Methods F1
LSTM 38.2
Bi LSTM 39.1
Bi LSTM+A-Q-ATT 41.6
Bi LSTM+C-ATT 41.8
Bi LSTM+GKI 40.4
Bi LSTM+A-Q-ATT+GKI 42.6
Bi LSTM+C-ATT+GKI 42.9
Table 2: The ablation results of our models.
LSTM employs unidirectional LSTM, and uses the last hidden state as the question representation. Bi LSTM adopts a bidirectional LSTM. A-Q-ATT denotes the answer-towards-question attention part, and C-ATT stands for our crossattention. GKI means global knowledge information. Bi LSTMS+C-ATT+GKI is our full proposed approach.From the results, we could observe the following.
1) Bi LSTM+C-ATT dramatically improves the F1 score by 2.7 points compared with Bi LSTM, 0.2 points higher than Bi LSTM+A-Q-ATT. Similarly, Bi LSTM+C-ATT+GKI significantly outperforms Bi LSTM+GKI by 2.5 points, improving Bi LSTM+A-Q-ATT+GKI by 0.3 points. The results prove that the proposed cross-attention model is effective.
2) Bi LSTM+GKI performs better than Bi LSTM, and achieves an improvement of 1.3 points. Similarly, Bi LSTM+C-ATT+GKI improves Bi LSTM+C-ATT by 1.1 points, which indicates that the proposed training strategy
successfully leverages the global information of the underlying KB.
3) Bi LSTM+C-ATT+GKI achieves the best performance as we expected, and improves the original Bi LSTM dramatically by 3.8 points. This directly shows the power of the attention model and the global KB information.
To illustrate the effectiveness of the attention mechanism clearly, we present the attention weights of a question in the form of heat map as shown in Figure 3.
where is the carpathian mountain range located
answer entity
answer type answer relation answer context
Figure 3: The visualized attention heat map. Answer entity: /m/06npd(Slovakia), answer relation: partially containedby, answer type: /location/country, answer context: (/m/04dq9kf, /m/01mp, ...)
From this example we observe that our methods is able to capture the attention properly. It is instructive to figure out the attention part of the question when dealing with different answer aspects. The heat map will help us understand which parts are most useful for selecting correct answers. For instance, from Figure 3, we can see that location.country is paying great attention to “Where”, indicating that “Where” is much more important than the other parts in the question when dealing with this type. In other words, the other parts are not that crucial since “Where” is strongly implying that the question is asking about a location. As for Q-A attention part, we see that answer type and answer relation are more important than other answer aspects in this example.
4.3 Error Analysis
We randomly sample 100 imperfectly answered questions and categorize the errors into two main classes as follows. Wrong attention
In some occasions (17 in 100 questions, 17%), we find the generated attention weights unreasonable. For instance, for question “What are the songs that Justin Bieber wrote?”, answer type /music/composition pays the most attention on “What” rather than “songs”. We think this is due to the bias of the training data, and we believe these errors could be solved by introducing more instructive training data. Complex questions and label errors
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Another challenging problem is the complex questions (34%). For example, “When was the last time Knicks won the championship?” is actually to ask the last championship, but the predicted answers give all the championships. This is due to that the model cannot learn what “last” mean in the training process. In addition, the label mistakes also influence the evaluation (3%), such as, “What college did John Nash teach at?”, where the labeled answer is Princeton University, but Massachusetts Institute of Technology should also be an answer, and the proposed method is able to answer it correctly. Other errors include topic entity generation error and the multiple answers error (giving more answers than expected). We guess these errors are caused by the simple implementations of the related steps in our method, and we will not explain them in detail due to space limitation.
5 Related Work
5.1 Neural Network-based KB-QA
In recent years, deep neural networks have been applied to many NLP tasks, showing promising results. Bordes et al. (2014b) was the first to introduce NN-based method to solve KB-QA problem. The questions and KB triples were represented by vectors in a low dimensional space. Thus the cosine similarity could be used to find the most possible answer. BOW method was employed to obtain a single vector for both the questions and the answers. Pairwise training was utilized, and the negative examples were randomly selected from the KB facts. Bordes et al. (2014a) further improved their work by proposing the concept of subgraph embeddings. The key idea was to involve as much information as possible in the answer end. Besides the answer triple, the subgraph contained all the entities and relations connected to the answer entity. The final vector was also obtained by bag-of-words strategy.
Yih et al. (2014) focused on single-relation questions. The KB-QA task was divided into two steps. Firstly, they found the topic entity of the question. Then, the rest of the question was represented by CNNs and used to match relations. Yang et al. (2014) tackled entity and relation mapping as joint procedures. Actually, these two methods followed the SP-based manner, but they took advantage of neural networks to obtain intermediate mapping results.
The most similar work to ours is Dong et
al. (2015). They considered the different aspects of answers, using three columns of CNNs to represent questions respectively. The difference is that our approach uses cross-attention mechanism for each unique answer aspect, so the question representation is not fixed to only three types. Moreover, we utilize the global KB information.
Xu et al. (2016a; 2016b) proposed integrated systems to address KB-QA problems incorporating Wikipedia free text, in which they used multichannel CNNs to extract relations.
5.2 Attention-based Model
The attention mechanism has been widely used in different areas. Bahdanau et al. (2015) first applied attention model in NLP. They improved the encoder-decoder Neural Machine Translation (NMT) framework by jointly learning align and translation. They argued that representing source sentence by a fixed vector is unreasonable, and proposed a soft-align method, which could be understood as attention mechanism. Rush et al. (2015) implemented sentence-level summarization task. They utilized local attention-based model that generated each word of the summary conditioned on the input sentence.
Yin et al. (2016) tackled simple question answering by an attentive convolutional neural network. They stacked an attentive maxpooling above convolution layer to model the relationship between predicates and question patterns. Our approach differs from previous work in that we use attentions to help represent question dynamically, not generating current word from vocabulary as before.
6 Conclusion
In this paper, we focus on KB-QA task. Firstly, we consider the impacts of the different answer aspects when representing the question, and propose a novel cross-attention model for KB-QA. Specifically, we employ the focus of the answer aspects to each question word and the attention weights of the question towards the answer aspects. This kind of dynamic representation is more precise and flexible. Secondly, we leverage the global KB information, which could take full advantage of the complete KB, and also alleviate the OOV problem for the attention model. The extensive experiments demonstrate that the proposed approach could achieve better performance compared with state-of-the-art end-to-end methods.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

26: An End-to-End Model for Question Answering over Knowledge Base with Cross-Attention Combining Global Knowledge  This paper presents an approach for factoid question answering over a knowledge graph (Freebase), by using a neural model that attempts to learn a semantic correlation correspondence between various  aspects  of the candidate answer (e.g., answer type, relation to question entity, answer semantic, etc.) and a subset of words of the question. A separate correspondence component is learned for each  aspect  of the candidate answers. The two key contributions of this work are: (1) the creation of separate components to capture different aspects of the candidate answer, rather than relying on a single semantic representation, and (2) incorporating global context (from the KB) of the candidate answers.  The most interesting aspect of this work, in my opinion, is the separation of candidate answer representation into distinct aspects, which gives us (the neural model developer) a little more control over guiding the NN models towards information that would be more beneficial in its decision making. It sort of harkens to the more traditional algorithms that rely on feature engineering. But in this case the  feature engineering  (i.e., aspects) is more subtle, and less onerous. I encourage the authors to continue refining this system along these lines.  While the high-level idea is fairly clear to a reasonably informed reader, the devil in the details would make it hard for some audience to immediately grasp key insights from this work. Some parts of the paper could benefit from more explanation... Specifically:  (1) Context aspect of candidate answers (e_c) is not clearly explained in the paper. Therefore, the last two sentences of Section 3.2.2 seem unclear.  (2) Mention of OOV in the abstract and introduction need more explanation. As such, I think the current exposition in the paper assumes a deep understanding of prior work by the reader.  (3) The experiments conducted in this paper restrict comparison to IR-based system -- and the reasoning behind this decision is reasonable. But it is not clear then why the work of Yang et al. (2014) -- which is described to be SP-based -- is part of the comparison. While, I am all for including more systems in the comparison, there seem to be some inconsistencies in what should and should not be compared. Additionally, I see not harm in also mentioning the comparable performance numbers for the best SP-based systems.  I observe in the paper that the embeddings are learned entirely from the training data. I wonder how much impact the random initialization of these embeddings has on the end performance. It would be interesting to determine (and list) the variance if any. Additionally, if we were to start with pre-trained embeddings (e.g., from word2vec) instead of the randomly initialized ones, would that have any impact?  As I read the paper, one possible direction of future work that occurred to me was to possibly include structured queries (from SP-based methods) as part of the cross-attention mechanism. In other words, in addition to using the various aspects of the candidate answers as features, one could include structured queries that generate the produce that candidate answer as an additional aspect of the candidate answer. An attention mechanism could then also focus on various parts of the structured query, and its (semantic) matches to the input question as an additional signal for the NN model. Just a thought.  Some notes regarding the positioning of the paper:  I hesitate to call the model proposed here  attention  models, because (per my admittedly limited understanding) attention mechanisms apply to  encoder-decoder  situations, where semantics expressed in one structured form (e.g., image, sentence in one language, natural language question, etc.) are encoded into an abstract representation, and then generated into another structured form (e.g., caption, sentence in another language, structured query, etc.). The attention mechanism allows the  encoder  to jump around and attend to different parts of the input (instead of sequentially) as the output is being generated by the decoder. This paper does not appear to fit this notion, and may be confusing to a broader audience.  ------  Thank you for clarifications in the author response.