Generating and Exploiting Large-scale Pseudo Training Data for Zero Pronoun Resolution
Most existing approaches for zero pronoun resolution are heavily relying on annotated data, which is often released by shared task organizers. Therefore, the lack of annotated data becomes a major obstacle in the progress of zero pronoun resolution task. Also, it is expensive to spend manpower on labeling the data for better performance. To alleviate the problem above, in this paper, we propose a simple but novel approach to automatically generate large-scale pseudo training data for zero pronoun resolution. Furthermore, we successfully transfer the cloze-style reading comprehension neural network model into zero pronoun resolution task and propose a two-step training mechanism to overcome the gap between the pseudo training data and the real one. Experimental results show that the proposed approach significantly outperforms the state-of-the-art systems with an absolute improvements of 3.1% F-score on OntoNotes 5.0 data.
1 Introduction
Previous works on zero pronoun (ZP) resolution mainly focused on the supervised learning approaches (Han, 2006; Zhao and Ng, 2007; Iida et al., 2007; Kong and Zhou, 2010; Iida and Poesio, 2011; Chen and Ng, 2013). However, a major obstacle for training the supervised learning models for ZP resolution is the lack of annotated data. An important step is to organize the shared task on anaphora and coreference resolution, such as the ACE evaluations, SemEval-2010 shared task on Coreference Resolution in Multiple Languages (Marta Recasens, 2010) and CoNLL2012 shared task on Modeling Multilingual Unre-
stricted Coreference in OntoNotes (Sameer Pradhan, 2012). Following these shared tasks, the annotated evaluation data can be released for the following researches. Despite the success and contributions of these shared tasks, it still faces the challenge of spending manpower on labeling the extended data for better training performance and domain adaptation.
To address the problem above, in this paper, we propose a simple but novel approach to automatically generate large-scale pseudo training data for zero pronoun resolution. Inspired by data generation on cloze-style reading comprehension, we can treat the zero pronoun resolution task as a special case of reading comprehension problem. So we can adopt similar data generation methods of reading comprehension to the zero pronoun resolution task. For the noun or pronoun in the document, which has the frequency equal to or greater than 2, we randomly choose one position where the noun or pronoun is located on, and replace it with a specific symbol 〈blank〉. Let query Q and answer A denote the sentence that contains a 〈blank〉, and the noun or pronoun which is replaced by the 〈blank〉, respectively. Thus, a pseudo training sample can be represented as a triple:
〈D,Q,A〉 (1)
For the zero pronoun resolution task, a 〈blank〉 represents a zero pronoun (ZP) in query Q, and A indicates the corresponding antecedent of the ZP. In this way, tremendous pseudo training samples can be generated from the various documents, such as news corpus.
Towards the shortcomings of the previous approaches that are based on feature engineering, we propose a neural network architecture, which is an attention-based neural network model, for zero pronoun resolution. Also we propose a two-step
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
training method, which benefit from both largescale pseudo training data and task-specific data, showing promising performance.
To sum up, the contributions of this paper are listed as follows.
• To our knowledge, this is the first time that utilizing reading comprehension neural network model into zero pronoun resolution task.
• We propose a two-step training approach, namely pre-training-then-adaptation, which benefits from both the large-scale automatically generated pseudo training data and taskspecific data.
• Towards the shortcomings of the feature engineering approaches, we first propose an attention-based neural network model for zero pronoun resolution.
2 The Proposed Approach
In this section, we will describe our approach in detail. First, we will describe our method of generating large-scale pseudo training data for zero pronoun resolution. Then we will introduce twostep training approach to alleviate the gaps between pseudo and real training data. Finally, the attention-based neural network model as well as associated unknown words processing techniques will be described.
2.1 Generating Pseudo Training Data
In order to get large quantities of training data for neural network model, we propose an approach, which is inspired by (Hermann et al., 2015), to automatically generate large-scale pseudo training data for zero pronoun resolution. However, our approach is much more simple and general than that of (Hermann et al., 2015). We will introduce the details of generating the pseudo training data for zero pronoun resolution as follows.
First, we collect a large number of documents that are relevant (or homogenous in some sense) to the released OntoNote 5.0 data for zero pronoun resolution task in terms of its domain. In our experiments, we used large-scale news data for training.
Given a certain document D, which is composed by a set of sentences D = {s1, s2, ..., sn},
we randomly choose an answer wordA in the document. Note that, we restrictA to be either a noun or pronoun, where the part-of-speech is identified using LTP Toolkit (Che et al., 2010), as well as the answer word should appear at least twice in the document. Second, after the answer word A is chosen, the sentence that contains A is defined as a queryQ, in which the answer wordA is replaced by a specific symbol 〈blank〉. In this way, given the queryQ and documentD, the target of the prediction is to recover the answer A. That is quite similar to the zero pronoun resolution task. Therefore, the automatically generated training samples is called pseudo training data. Figure 1 shows an example of a pseudo training sample.
In this way, we can generate tremendous triples of 〈D,Q,A〉 for training neural network, without making any assumptions on the nature of the original corpus.
2.2 Two-step Training
It should be noted that, though we have generated large-scale pseudo training data for neural network training, there is still a gap between pseudo training data and the real zero pronoun resolution task in terms of the query style. So we should do some adaptations to our model to deal with the zero pronoun resolution problems ideally.
In this paper, we used an effective approach to deal with the mismatch between pseudo training data and zero pronoun resolution task-specific data. Generally speaking, in the first stage, we use a large amount of the pseudo training data to train a fundamental model, and choose the best model according to the validation accuracy. Then we continue to train from the previous best model using the zero pronoun resolution task-specific training data, which is exactly the same domain and query type as the standard zero pronoun resolution task data.
The using of the combination of proposed pseudo training data and task-specific data, i.e. zero pronoun resolution task data, is far more effective than using either of them alone. Though there is a gap between these two data, they share many similar characteristics to each other as illustrated in the previous part, so it is promising to utilize these two types of data together, which will compensate to each other.
The two-step training procedure can be concluded as,
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Document: 1 ||| welcome both of you to the studio to participate in our program , 欢迎两位呢来演播室参与我们的节目， 2 ||| it happened that i was going to have lunch with a friend at noon . 正好因为我也和朋友这个，这个中午一起吃饭。 3 ||| after that , i received an sms from 1860 . 然后我就收到 1860 的短信。 4 ||| uh-huh , it was by sms . 嗯，是通过短信 的方式 ， 5 ||| uh-huh , that means , er , you knew about the accident through the source of radio station . 嗯，就是说呃你是通过台里面的一 个信息的渠道知道这儿出了这样的事故。 6 ||| although we live in the west instead of the east part , and it did not affect us that much , 虽然我们生活在西部不是在东部，对我们影响不是 很大 ， 7 ||| but i think it is very useful to inform people using sms . 但是呢，我觉得 有这样 一个短信告诉大家呢 是非常有用的啊。 Query: 8 ||| some car owners said that <blank> was very good。 有车主表示，说这 <blank> 非常的好。 Answer: sms 短信
Figure 1: Example of pseudo training sample for zero pronoun resolution system. (The original data is in Chinese, we translate this sample into English for clarity)
• Pre-training stage: by using large-scale training data to train the neural network model, we can learn richer word embeddings, as well as relatively reasonable weights in neural networks than just training with a small amount of zero pronoun resolution task training data;
• Adaptation stage: after getting the best model that is produced in the previous step, we continue to train the model with task-specific data, which can force the previous model to adapt to the new data, without losing much knowledge that has learned in the previous stage (such as word embeddings).
As we will see in the experiment section that the proposed two-step training approach is effective and brings significant improvements.
2.3 Attention-based Neural Network Model
Our model is primarily an attention-based neural network model, which is similar to Attentive Reader proposed by (Hermann et al., 2015). Formally, when given a set of training triple 〈D,Q,A〉, we will construct our network in the following way.
Firstly, we project one-hot representation of document D and query Q into a continuous space with the shared embedding matrix We. Then we input these embeddings into different bidirectional RNN to get their contextual representations respectively. In our model, we used the bidirectional Gated Recurrent Unit (GRU) as RNN implementation (Cho et al., 2014).
e(x) =We · x, where x ∈ D,Q (2)
−→ hs = −−−→ GRU(e(x)); ←− hs = ←−−− GRU(e(x)) (3)
hs = [ −→ hs; ←− hs] (4)
For the query representation, instead of concatenating the final forward and backward states as its representations, we directly get an averaged representations on all bi-directional RNN slices, which can be illustrated as
hquery = 1
n n∑ t=1 hquery(t) (5)
For the document, we place a soft attention over all words in document (Bahdanau et al., 2014), which indicate the degree to which part of document is attended when filling the blank in the query sentence. Then we calculate a weighted sum of all document tokens to get the attended representation of document.
m(t) = tanh(W · hdoc(t) + U · hquery) (6)
α(t) = exp(Ws ·m(t)) n∑ j=1 exp(Ws ·m(j)) (7)
hdoc att = hdoc · α (8)
where variable α(t) is the normalized attention weight at tth word in document, hdoc is a matrix that concatenate all hdoc(t) in sequence.
hdoc = concat[hdoc(1), hdoc(2), ..., hdoc(t)] (9)
Then we use attended document representation and query representation to estimate the final answer, which can be illustrated as follows, where V
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Bi-GRU Encoder
Σ
d1 d2 d3 d4 q1 q2 q3
Query
Softmax Layer
Concat Layer
AttentionLayer
Answer
Document
Embedding Layer
Figure 2: Architecture of attention-based neural network model for zero pronoun resolution task.
is the vocabulary,
r = concat[hdoc att, hquery] (10)
P (A|D,Q) ∝ softmax(Wr · r) , s.t. A ∈ V (11)
Figure 2 shows the proposed neural network architecture.
Note that, for zero pronoun resolution task, antecedents of zero pronouns are always noun phrases (NPs), while our model generates only one word (a noun or a pronoun) as the result. To better adapt our model to zero pronoun resolution task, we further process the output result in the following procedure. First, for a given zero pronoun, we extract a set of NPs as its candidates utilizing the same strategy as (Chen and Ng, 2015). Then, we use our model to generate an answer (one word) for the zero pronoun. After that, we go through all the candidates from the nearest to the far-most. For an NP candidate, if the produced answer is its head word, we then regard this NP as the antecedent of the given zero pronoun. By doing so, for a given zero pronoun, we generate an NP as the prediction of its antecedent.
2.4 Unknown Words Processing
Because of the restriction on both memory occupation and training time, it is usually suggested to use a shortlist of vocabulary in neural network training. However, we often replace the out-ofvocabularies to a unique special token, such as 〈unk〉. But this may place an obstacle in real
world test. When the model predicts the answer as 〈unk〉, we do not know what is the exact word it represents in the document, as there may have many 〈unk〉s in the document.
In this paper, we propose to use a simple but effective way to handle unknown words issue. The idea is straightforward, which can be illustrated as follows.
• Identify all unknown words inside of each 〈D,Q,A〉;
• Instead of replacing all these unknown words into one unique token 〈unk〉, we make a hash table to project these unique unknown words to numbered tokens, such as 〈unk1〉, 〈unk2〉, ..., 〈unkN〉 in terms of its occurrence order in the document. Note that, the same words are projected to the same unknown word tokens, and all these projections are only valid inside of current sample. For example, 〈unk1〉 indicate the first unknown word, say “apple”, in the current sample, but in another sample the 〈unk1〉 may indicate the unknown word “orange”. That is, the unknown word labels are indicating position features rather than the exact word;
• Insert these unknown marks in the vocabulary. These marks may only take up dozens of slots, which is negligible to the size of shortlists (usually 30K ∼ 100K).
(a) The weather today is not as pleasant as the weather of yesterday.
(b) The <unk> today is not as <unk> as the <unk> of yesterday.
(c) The <unk1> today is not as <unk2> as the <unk1> of yesterday.
Figure 3: Example of unknown words processing. a) original sentence; b) original unknown words processing method; c) our method
We take one sentence “The weather of today is not as pleasant as the weather of yesterday.” as an example to show our unknown word processing method, which is shown in Figure 3.
If we do not discriminate the unknown words and assign different unknown words with the same token 〈unk〉, it would be impossible for us to know what is the exact word that 〈unk〉 represents for in the real test. However, when using our proposed unknown word processing method, if the model predicts a 〈unkX〉 as the answer,
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
we can simply scan through the original document and identify its position according to its unknown word number X and replace the 〈unkX〉 with the real word. For example, in Figure 3, if we adopt original unknown words processing method, we could not know whether the 〈unk〉 is the word “weather” or “pleasant”. However, when using our approach, if the model predicts an answer as 〈unk1〉, from the original text, we can know that 〈unk1〉 represents the word “weather”.
3 Experiments
3.1 Data
In our experiments, we choose a selection of public news data to generate large-scale pseudo training data for pre-training our neural network model (pre-training step)1. In the adaptation step, we used the official dataset OntoNotes Release 5.02 which is provided by CoNLL-2012 shared task, to carry out our experiments. The CoNLL2012 shared task dataset consists of three parts: a training set, a development set and a test set. The datasets are made up of 6 different domains, namely Broadcast News (BN), Newswires (NW), Broadcast Conversations (BC), Telephone Conversations (TC), Web Blogs (WB), and Magazines (MZ). We closely follow the experimental settings as (Kong and Zhou, 2010; Chen and Ng, 2014, 2015, 2016), where we treat the training set for training and the development set for testing, because only the training and development set are annotated with ZPs. The statistics of training and testing data is shown in Table 1 and 2 respectively.
Sentences # Query #
General Train 18.47M 1.81M Domain Train 122.8K 9.4K Validation 11,191 2,667
Table 1: Statistics of training data, including pseudo training data and OntoNotes 5.0 training data.
3.2 Neural Network Setups
Training details of our neural network models are listed as follows.
1The news data is available at http://www.sogou. com/labs/dl/cs.html
2http://catalog.ldc.upenn.edu/ LDC2013T19
• Embedding: We use randomly initialized embedding matrix with uniformed distribution in the interval [-0.1,0.1], and set units number as 256. No pre-trained word embeddings are used.
• Hidden Layer: We use GRU with 256 units, and initialize the internal matrix by random orthogonal matrices (Saxe et al., 2013). As GRU still suffers from the gradient exploding problem, we set gradient clipping threshold to 10.
• Vocabulary: As the whole vocabulary is very large (over 800K), we set a shortlist of 100K according to the word frequency and unknown words are mapped to 20 〈unkX〉 using the proposed method.
• Optimization: We used ADAM update rule (Kingma and Ba, 2014) with an initial learning rate of 0.001, and used negative loglikelihood as the training objective. The batch size is set to 32.
All models are trained on Tesla K40 GPU. Our model is implemented with Theano (Theano Development Team, 2016) and Keras (Chollet, 2015).
3.3 Experimental results
Same to the previous researches that are related to zero pronoun resolution, we evaluate our system performance in terms of F-score (F). We focus on AZP resolution process, where we assume that gold AZPs and gold parse trees are given3. The same experimental setting is utilized in (Chen and Ng, 2014, 2015, 2016). The overall results are shown in Table 3, where the performances of each domain are listed in detail and overall performance is also shown in the last column.
• Overall Performance We employ four Chinese ZP resolution baseline systems on OntoNotes 5.0 dataset. As we can
3All gold information are provided by the CoNLL-2012 shared task dataset
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
NW (84) MZ (162) WB (284) BN (390) BC (510) TC (283) Overall
Kong and Zhou (2010) 34.5 32.7 45.4 51.0 43.5 48.4 44.9 Chen and Ng (2014) 38.1 31.0 50.4 45.9 53.8 54.9 48.7 Chen and Ng (2015) 46.4 39.0 51.8 53.8 49.4 52.7 52.2 Chen and Ng (2016) 48.8 41.5 56.3 55.4 50.8 53.1 52.2
Our Approach† 59.2 51.3 60.5 53.9 55.5 52.9 55.3
Table 3: Experimental result (F-score) on the OntoNotes 5.0 test data. The best results are marked with bold face. † indicates that our approach is statistical significant over the baselines (using t-test, with p < 0.05). The number in the brackets indicate the number of AZPs.
see that our model significantly outperforms the previous state-of-the-art system (Chen and Ng, 2016) by 3.1% in overall F-score, and substantially outperform the other systems by a large margin. When observing the performances of different domains, our approach also gives relatively consistent improvements among various domains, except for BN and TC with a slight drop. All these results approve that our proposed approach is effective and achieves significant improvements in AZP resolution.
In our quantitative analysis, we investigated the reasons of the declines in the BN and TC domain. A primary observation is that the word distributions in these domains are fairly different from others. The average document length of BN and TC are quite longer than other domains, which suggest that there is a bigger chance to have unknown words than other domains, and add difficulties to the model training. Also, we have found that in the BN and TC domains, the texts are often in oral form, which means that there are many irregular expressions in the context. Such expressions add noise to the model, and it is difficult for the model to extract useful information in these contexts. These phenomena indicate that further improvements can be obtained by filtering stop words in contexts, or increasing the size of task-specific data, while we leave this in the future work.
• Effect of UNK processing As we have mentioned in the previous section, traditional unknown word replacing methods are vulnerable to the real word test. To alleviate this issue, we proposed the UNK processing mechanism to recover the UNK tokens to the real words. In Table 4, we compared the performance that with and without the proposed UNK processing,
to show whether the proposed UNK processing method is effective. As we can see that, by applying our UNK processing mechanism, the model do learned the positional features in these lowfrequency words, and brings over 3% improvements in F-score, which indicated the effectiveness of our UNK processing approach.
• Effect of Domain Adaptation We also tested out whether our domain adaptation method is effective. In this experiments, we used three different types of training data: only pseudo training data, only task-specific data, and our adaptation method, i.e. using pseudo training data in the pre-training step and task-specific data for domain adaptation step. The results are given in Table 5. As we can see that, using either pseudo training data or task-specific data alone can not bring inspiring result. By adopting our domain adaptation method, the model could give significant improvements over the other models, which demonstrate the effectiveness of our proposed two-step training approach. An intuition behind this phenomenon is that though pseudo training data is fairly big enough to train a reliable model parameters, there is still a gap to the real zero pronoun resolution tasks. On the contrary, though task-specific training data is exactly the same type as the real test, the quantity is not enough to train a reasonable model (such as word embedding). So it is better to make use of both to
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
take the full advantage. However, as the original task-specific data is fairly small compared to pseudo training data, we also wondered if the large-scale pseudo training data is only providing rich word embedding information. So we use the large-scale pseudo training data for embedding training using GloVe toolkit (Pennington et al., 2014), and initialize the word embeddings in the “only task-specific data” system. From the result we can see that the pseudo training data provide more information than word embeddings, because though we used GloVe embeddings in “only task-specific data”, it still can not outperform the system that uses domain adaptation which supports our claim.
F-score
Only Pseudo Training Data 41.1 Only Task-Specific Data 44.2 Only Task-Specific Data + GloVe 50.9 Domain Adaptation 55.3
Table 5: Performance comparison of using different training data.
4 Error Analysis
To better evaluate our proposed approach, we performed a qualitative analysis of errors, where two major errors are revealed by our analysis, as discussed below.
4.1 Effect of Unknown Words
Our approach does not do well when there are lots of 〈unk〉s in the context of ZPs, especially when the 〈unk〉s appears near the ZP. An example is given below, where words with # are regarded as 〈unk〉s in our model.
φ 登上# 太平山# 顶 , 将香港岛# 和维多 利亚港# 的美景尽收眼底。 φ Successfully climbed# the peak of [Taiping Mountain]#, to have a panoramic view of the beauty of [Hong Kong Island]# and [Victoria Harbour]#.
In this case, the words “登上/climbed” and “太 平山/Taiping Mountain” that appears immediately after the ZP “φ” are all regarded as 〈unk〉s in our model. As we model the sequence of words by RNN, the 〈unk〉s make the model more difficult to capture the semantic information of the sentence, which in turn influence the overall performance. Especially for the words that are near
the ZP, which play important roles when modeling context information for the ZP. By looking at the word “顶/peak”, it is hard to comprehend the context information, due to the several surrounding 〈unk〉s. Though our proposed unknown words processing method is effective in empirical evaluation, we think that more advanced method for unknown words processing would be of a great help in improving comprehension of the context.
4.2 Long Distance Antecedents
Also, our model makes incorrect decisions when the correct antecedents of ZPs are in long distance. As our model chooses answer from words in the context, if there are lots of words between the ZP and its antecedent, more noise information are introduced, and adds more difficulty in choosing the right answer. For example:
我帮不了那个人 ... ...那天结束后 φ 回到 家中。 I can’t help that guy ... ... After that day, φ return home.
In this case, the correct antecedent of ZP “φ” is the NP candidate “我/I”. By seeing the contexts, we observe that there are over 30 words between the ZP and its antecedent. Although our model does not intend to fill the ZP gap only with the words near the ZP, as most of the antecedents appear just a few words before the ZPs, our model prefers the nearer words as correct antecedents. Hence, once there are lots of words between ZP and its nearest antecedent, our model can sometimes make wrong decisions. To correctly handle such cases, our model should learn how to filter the useless words and enhance the learning of longterm dependency.
5 Related Work
5.1 Zero pronoun resolution
For Chinese zero pronoun (ZP) resolution, early studies employed heuristic rules to Chinese ZP resolution. Converse (2006) proposes a rule-based method to resolve the zero pronouns, by utilizing Hobbs algorithm (Hobbs, 1978) in the CTB documents. Then, supervised approaches to this task have been vastly explored. Zhao and Ng (2007) first present a supervised machine learning approach to the identification and resolution of Chinese ZPs. Kong and Zhou (2010) develop a tree-kernel based approach for Chinese ZP resolution. More recently, unsupervised approaches
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
have been proposed. Chen and Ng (2014) develop an unsupervised language-independent approach, utilizing the integer linear programming to using ten overt pronouns. Chen and Ng (2015) propose an end-to-end unsupervised probabilistic model for Chinese ZP resolution, using a salience model to capture discourse information. Also, there have been many works on ZP resolution for other languages. These studies can be divided into rule-based and supervised machine learning approaches. Ferrández and Peral (2000) proposed a set of hand-crafted rules for Spanish ZP resolution. Recently, supervised approaches have been exploited for ZP resolution in Korean (Han, 2006) and Japanese (Isozaki and Hirao, 2003; Iida et al., 2006, 2007; Sasano and Kurohashi, 2011). Iida and Poesio (2011) developed a cross-lingual approach for Japanese and Italian ZPs where an ILPbased model was employed to zero anaphora detection and resolution.
In sum, most recent researches on ZP resolution are supervised approaches, which means that their performance highly relies on large-scale annotated data. Even for the unsupervised approach (Chen and Ng, 2014), they also utilize a supervised pronoun resolver to resolve ZPs. Therefore, the advantage of our proposed approach is obvious. We are able to generate large-scale pseudo training data for ZP resolution, and also we can benefit from the task-specific data for fine-tuning via the proposed two-step training approach.
5.2 Cloze-style Reading Comprehension
Our neural network model is mainly motivated by the recent researches on cloze-style reading comprehension tasks, which aims to predict one-word answer given the document and query. These models can be seen as a general model of mining the relations between the document and query, so it is promising to combine these models to the specific domain.
A representative work of cloze-style reading comprehension is done by Hermann et al. (2015). They proposed a methodology for obtaining large quantities of 〈D,Q,A〉 triples. By using this method, a large number of training data can be obtained without much human intervention, and make it possible to train a reliable neural network. They used attention-based neural networks for this task. Evaluation on CNN/DailyMail datasets showed that their approach is much effective than
traditional baseline systems. While our work is similar to Hermann et al. (2015), there are several differences which can be illustrated as follows. Firstly, though we both utilize the large-scale corpus, they require that the document should accompany with a brief summary of it, while this is not always available in most of the document, and it may place an obstacle in generating limitless training data. In our work, we do not assume any prerequisite of the training data, and directly extract queries from the document, which makes it easy to generate large-scale training data. Secondly, their work mainly focuses on reading comprehension in the general domain. We are able to exploit large-scale training data for solving problems in the specific domain, and we proposed two-step training method which can be easily adapted to other domains as well.
6 Conclusion
In this study, we propose an effective way to generate and exploit large-scale pseudo training data for zero pronoun resolution task. The main idea behind our approach is to automatically generate large-scale pseudo training data and then utilize an attention-based neural network model to resolve zero pronouns. For training purpose, two-step training approach is employed, i.e. a pre-training and adaptation step, and this can be also easily applied to other tasks as well. The experimental results on OntoNotes 5.0 corpus are encouraging, showing that the proposed model and accompanying approaches significantly outperforms the stateof-the-art systems.
The future work will be carried out on two main aspects: First, as experimental results show that the unknown words processing is a critical part in comprehending context, we will explore more effective way to handle the UNK issue. Second, we will develop other neural network architecture to make it more appropriate for zero pronoun resolution task.

- Strengths:  The approach is novel and the results are very promising, beating state-of-the-art.  - Weaknesses:   The linguistic motivation behind the paper is troublesome (see below). I feel that the paper would benefit a lot from a more thoughtful interpretation of the results.  - General Discussion:  This paper presents an approach for Zero Pronoun Resolution in Chinese. The authors advocate a novel procedure for generating large amount of relevant data from unlabeled documents. These data are then integrated smartly in an NN-based architecture at a pre-training step. The results improve on state-of-the-art.  I have mixed feelings about this study. On the one hand, the approach seems sound and shows promising results, beating very recent systems (e.g., Chen&Ng 2016). On the other hand, the way the main contribution is framed is very disturbing from the linguistic point of view. In particular, (zero) pronoun resolution is, linguistically speaking, a context modeling task, requiring accurate interpretation of discourse salience, semantic and syntactic clues. It starts from the assumption that (zero) pronouns are used in specific contexts, where full NPs shouldn't normally be possible. From this perspective, generating ZP data via replacing nominal with zeroes ( blank ) doesn't sound very convincing. And indeed, as the authors themselves show, the pre-training module alone doesn't achieve a reasonable performance. To sum it up, i don't think that these generated pseudo-data can be called AZP data. It seems more likely that they encode some form of selectional preferences (?). It would be nice if the authors could invest some effort in better understanding what exactly the pre-training module learns -- and then reformulate the corresponding sections.   The paper can benefit from a proofreading by a native speaker of English -- for example, the sentence on lines 064-068 is not grammatical.  -- other points --  lines 78-79: are there any restrictions on the nouns and especially pronouns? for example, do you use this strategy for very common pronouns (as English  it )? if so, how do you guarantee that the two occurrences of the same token  are indeed coreferent?  line 91: the term antecedent is typically used to denote a preceding mention coreferent with the anaphor, which is not what you mean here  line 144: OntoNotes (typo)  lines 487-489: it has been shown that evaluation on gold-annotated data does not provide reliable estimation of performance. and, indeed, all the recent studies of coreference evaluate on system mentions. for example, the studies of Chen&Ng you are citing, provide different types of evaluation, including those on system mentions. please consider rerunning your experiments to get a more realistic evaluation setup  line 506: i don't understand what the dagger over the system's name means. is your improvement statistically significant on all the domains? including bn and tc??  line 565: learn (typo)  section 3.3: in this section you use the abbreviation AZP instead of ZP without introducing it, please unify the terminology  references -- please double-check for capitalization


Attention-over-Attention Neural Networks for Reading Comprehension
Cloze-style reading comprehension is a representative problem in mining relationship between document and query. In this paper, we present a simple but novel model called attention-over-attention reader for better solving cloze-style reading comprehension task. Our model aims to place another attention mechanism over the document-level attention and induces “attended attention” for final answer predictions. One advantage of our model is that it is simpler than related works while giving excellent performance. We also propose an N-best re-ranking strategy to double check the validity of the candidates and further improve the performance. Experimental results show that the proposed methods significantly outperform various state-of-the-art systems by a large margin in public datasets, such as CNN and Children’s Book Test.
1 Introduction
To read and comprehend the human languages are challenging tasks for the machines, which requires that the understanding of natural languages and the ability to do reasoning over various clues. Reading comprehension is a general problem in the real world, which aims to read and comprehend a given article or context, and answer the questions based on it. Recently, the cloze-style reading comprehension problem has become a popular task in the community. The cloze-style query (Taylor, 1953) is a problem that to fill in an appropriate word in the given sentences while taking the context information into account.
To teach the machine to do cloze-style reading comprehensions, large-scale training data is nec-
essary for learning relationships between the given document and query. To create large-scale training data for neural networks, Hermann et al. (2015) released the CNN/Daily Mail news dataset, where the document is formed by the news articles and the queries are extracted from the summary of the news. Hill et al. (2015) released the Children’s Book Test dataset afterwards, where the training samples are generated from consecutive 20 sentences from books, and the query is formed by 21st sentence. Following these datasets, a vast variety of neural network approaches have been proposed (Kadlec et al., 2016; Cui et al., 2016; Chen et al., 2016; Dhingra et al., 2016; Sordoni et al., 2016; Trischler et al., 2016), and most of them stem from the attention-based neural network (Bahdanau et al., 2014), which has become a stereotype in most of the NLP tasks and is wellknown by its capability of learning the “importance” distribution over the inputs.
In this paper, we present a novel neural network architecture, called attention-over-attention model. As we can understand the meaning literally, our model aims to place another attention mechanism over the existing document-level attention. Unlike the previous works, that are using heuristic merging functions (Cui et al., 2016), or setting various pre-defined non-trainable terms (Trischler et al., 2016), our model could automatically generate an “attended attention” over various document-level attentions, and make a mutual look not only from query-to-document but also document-to-query, which will benefit from the interactive information.
To sum up, the main contributions of our work are listed as follows.
• To our knowledge, this is the first time that the mechanism of nesting another attention over the existing attentions is proposed, i.e.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
attention-over-attention mechanism.
• Unlike the previous works on introducing complex architectures or many non-trainable hyper-parameters to the model, our model is much more simple but outperforms various state-of-the-art systems by a large margin.
• We also propose an N-best re-ranking strategy to re-score the candidates in various aspects and further improve the performance.
2 Cloze-style Reading Comprehension
In this section, we will give a brief introduction to the cloze-style reading comprehension task at the beginning. And then, several existing public datasets will be described in detail.
2.1 Task Description
Formally, a general Cloze-style reading comprehension problem can be illustrated as a triple:
〈D,Q,A〉
The triple consists of a documentD, a queryQ and the answer to the queryA. Note that the answer is usually a single word in the document, which requires the human to exploit context information in both document and query. The type of the answer word varies from predicting a preposition given a fixed collocation to identifying a named entity from a factual illustration.
2.2 Existing Public Datasets
Large-scale training data is essential for training neural networks. Several public datasets for the cloze-style reading comprehension has been released. Here, we introduce two representative and widely-used datasets. CNN/Daily Mail.1 Hermann et al. (2015) have firstly published two datasets: CNN and Daily Mail news data. They construct these datasets with web-crawled CNN and Daily Mail news data. One of the characteristics of these datasets is that the news article is often associated with a summary. So they first regard the main body of the news article as the Document, and the Query is formed by the summary of the article, where one entity word is replaced by a special placeholder to indicate the missing word. The replaced entity word
1The pre-processed CNN and Daily Mail datasets are available at http://cs.nyu.edu/˜kcho/DMQA/
will be the Answer of the Query. Apart from releasing the dataset, they also proposed a methodology that anonymizes the named entity tokens in the data, and these tokens are also re-shuffle in each sample. The motivation is that the news articles are containing limited named entities, which are usually celebrities, and the world knowledge can be learned from the dataset. So this methodology aims to exploit general relationships between anonymized named entities within a single document rather than the common knowledge. The following research on these datasets showed that the entity word anonymization is not that effective than expected (Chen et al., 2016). Children’s Book Test. 2 There was also a dataset called the Children’s Book Test (CBTest) released by Hill et al. (2015), which is built on the children’s book story through Project Gutenberg. Different from the CNN/Daily Mail datasets, there is no summary available in the children’s book. So they proposed another way to extract query from the original data. The document is composed of 20 consecutive sentences in the story, and the 21st sentence is regarded as the query, where one word is blanked with a special placeholder. In the CBTest datasets, there are four types of sub-datasets available which are classified by the part-of-speech tag of the answer word, containing Named Entities (NE), Common Nouns (CN), Verbs and Prepositions. In their studies, they have found that the answering of verbs and prepositions are relatively less dependent on the content of document, and the humans can even do preposition blank-filling without the presence of the document. As the aim of reading comprehension is to exploit relations between document and query, most of the following studies are only focusing on the NE and CN datasets.
3 Attention-over-Attention Reader
In this section, we will give a detailed introduction to the proposed Attention-over-Attention Reader (AoA Reader). Our model is primarily motivated by Kadlec et al., (2016), which aims to directly estimate the answer from the document-level attention instead of calculating blended representations of the document. As previous studies by Cui et al. (2016) showed that the further investigation
2The CBTest datasets are available at http: //www.thespermwhale.com/jaseweston/babi/ CBTest.tgz
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Document
Query
!(“$%&'”|*,,) = / 01 = 02 + 04
1∈7(“89:;”,<) Mary
sits
beside
him ...
he
loves
Mary
he
loves
X
dot product
Column-wise softmax
Row-wise softmax
Column-wise Average
dot product
Embedding Layer
bi-GRU Layer
Individual ATT Layer
ATT-over-ATT Layer
Sum ATT Layer
Figure 1: Neural network architecture of the proposed Attention-over-Attention Reader (AoA Reader).
of query representation is necessary, and it should be paid more attention to utilizing the information of query. In this paper, we propose a novel work that placing another attention over the primary attentions, to indicate the “importance” of each attentions.
Now, we will give a formal description of our proposed model. When a cloze-style training triple 〈D,Q,A〉 is given, the proposed model will be constructed in the following steps.
Contextual Embedding. We first transform every word in the documentD and queryQ into one-hot representations and then convert them into continuous representations with a shared embedding matrix We. The motivation of using shared embedding weights is that the length of the query is shorter than the document, and thus the embedding weights will not be fully learned by only using a small amount of training data. By embedding sharing, both the document and query can participate in the learning of embedding and both of them will benefit from this mechanism. After that, we use two bi-directional RNNs to get contextual representations of the document and query individually, where the representation of each word
is formed by concatenating the forward and backward hidden states. After making a trade-off between model performance and training complexity, we choose the Gated Recurrent Unit (GRU) (Cho et al., 2014) as recurrent unit implementation.
e(x) =We · x, where x ∈ D,Q (1) −−−→ hs(x) = −−−→ GRU(e(x)) (2)
←−−− hs(x) = ←−−− GRU(e(x)) (3)
hs(x) = [ −−−→ hs(x); ←−−− hs(x)] (4)
We take hdoc ∈ R|D|∗2d and hquery ∈ R|Q|∗2d to denote the contextual representations of document and query, where d is the dimension of GRU (oneway). Pair-wise Matching Score. After obtaining the contextual embeddings of the document hdoc and query hquery, we calculate a pair-wise matching matrix, which indicates the pair-wise matching degree of one document word and one query word. Formally, when given ith word of the document and jth word of query, we can compute a matching score by their dot product.
M(i, j) = hdoc(i) T · hquery(j) (5)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
In this way, we can calculate every pair-wise matching score between each document and query word, forming a matrix M ∈ R|D|∗|Q|, where the value of ith row and jth column is filled by M(i, j). Individual Attentions. After getting the pair-wise matching matrixM , we apply a column-wise softmax function to get probability distributions in each column, where each column is an individual document-level attention when considering a single query word. We denote α(t) ∈ R|D| as the document-level attention regarding query word at time t, which can be seen as a query-to-document attention.
α(t) = softmax(M(1, t), ...,M(|D|, t)) (6) α = [α(1), α(2), ..., α(|Q|)] (7)
Attention-over-Attention. Instead of using naive heuristics (such as summing or averaging) to combine these individual attentions into a final attention, we introduce another attention mechanism to automatically decide the importance of each individual attention.
First, we calculate a reversed attention, that is, for every document word at time t, we calculate the “importance” distribution on the query, to indicate which query words are more important given a single document word. We apply a row-wise softmax function to the pair-wise matching matrix M to get query-level attentions. We denote β(t) ∈ R|Q| as the query-level attention regarding document word at time t, which can be seen as a document-to-query attention.
β(t) = softmax(M(t, 1), ...,M(t, |Q|)) (8)
So far, we have obtained both query-todocument attention α and document-to-query attention β. Our motivation is to exploit mutual information between the document and query. However, most of the previous works are only relying on query-to-document attention, that is, only calculate one document-level attention when considering the whole query.
Then we average all the β(t) to get an averaged query-level attention β. Note that, we do not apply another softmax to the β, because averaging individual attentions do not break the normalizing condition.
β = 1
n |D|∑ t=1 β(t) (9)
Finally, we calculate dot product of α and β to get the “attended document-level attention” s ∈ R|D|, i.e. the attention-over-attention mechanism. Intuitively, this operation is calculating a weighted sum of each individual document-level attention α(t) when looking at query word at time t. In this way, the contributions by each query word can be learned explicitly, and the final decision (document-level attention) is made through the voted result by the importance of each query word.
s = αTβ (10)
Final Predictions. Following Kadlec et al. (2016), we use sum attention mechanism to get aggregated results. Note that the final output should be reflected in the vocabulary space V , rather than document-level attention |D|, which will make a significant difference in the performance, though Kadlec et al. (2016) did not illustrate this clearly.
P (w|D,Q) = ∑
i∈I(w,D)
si, w ∈ V (11)
where I(w,D) indicate the positions that word w appear in the document D. As the training objectives, we seek to maximize the log-likelihood of the correct answer.
L = ∑ i log(p(x)) , x ∈ A (12)
The proposed neural network architecture is depicted in Figure 1. Note that, as our model mainly adds limited steps of calculations to the AS Reader (Kadlec et al., 2016) and do not employ any additional weights, the computational complexity is similar to the AS Reader.
4 N-best Re-ranking Strategy
Intuitively, when we do cloze-style reading comprehensions, we often refill the candidate into the blank of the query to double-check its appropriateness, fluency and grammar to see if the candidate we choose is the most suitable one. If we do find some problems in the candidate we choose, we will choose the second possible candidate and do some checking again.
To mimic the process of double-checking, we propose to use N-best re-ranking strategy after generating answers from our neural networks. The procedure can be illustrated as follows. N-best decoding. Instead of only picking the candidate that has the highest possibility as answer,
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
461
462
463
464
465
466
467
468
469
470
498
499
CNN News CBT NE CBT CN Train Valid Test Train Valid Test Train Valid Test
# Query 380,298 3,924 3,198 108,719 2,000 2,500 120,769 2,000 2,500 Max # candidates 527 187 396 10 10 10 10 10 10 Avg # candidates 26 26 25 10 10 10 10 10 10 Avg # tokens 762 763 716 433 412 424 470 448 461 Vocabulary 118,497 53,063 53,185
we can also extract follow-up candidates in the decoding process, which forms an N-best list. Refill the candidate into query. As a characteristic of the cloze-style problem, each candidate can be refilled into the blank of the query to form a complete sentence. This allows us to check the candidate according to its context. Feature scoring. The candidate sentences can be scored in many aspects. In this paper, we exploit three features to score the N-best list.
• Global N-gram LM: This is a fundamental metric in scoring sentence, which aims to evaluate its fluency. This model is trained on the document part of training data.
• Local N-gram LM: Different from global LM, the local LM aims to explore the information with the given document, so the statistics are obtained from the test-time document. It should be noted that the local LM is trained sample-by-sample, it is not trained on the entire test set, which is not legal in the real test case. This model is useful when there are many unknown words in the test sample.
• Word-class LM: Similar to global LM, the word-class LM is also trained on the document part of training data, but the words are converted to its word class ID. The word class can be obtained by using clustering methods. In this paper, we simply utilized the mkcls tool for generating 1000 word classes (Josef Och, 1999).
Weight Tuning. To tune the weights among these features, we adopt the K-best MIRA algorithm (Cherry and Foster, 2012) to automatically optimize the weights on the validation set, which is widely used in statistical machine translation tuning procedure.
Re-scoring and Re-ranking. After getting the weights of each feature, we calculate the weighted sum of each feature in the N-best sentences and then choose the candidate that has the lowest cost as the final answer.
5 Experiments
5.1 Experimental Setups
The general settings of our neural network model are listed below in detail.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
CNN News CBTest NE CBTest CN Valid Test Valid Test Valid Test
Deep LSTM Reader (Hermann et al., 2015) 55.0 57.0 - - - - Attentive Reader (Hermann et al., 2015) 61.6 63.0 - - - - Human (context+query) (Hill et al., 2015) - - - 81.6 - 81.6 MemNN (window + self-sup.) (Hill et al., 2015) 63.4 66.8 70.4 66.6 64.2 63.0 AS Reader (Kadlec et al., 2016) 68.6 69.5 73.8 68.6 68.8 63.4 CAS Reader (Cui et al., 2016) 68.2 70.0 74.2 69.2 68.2 65.7 Stanford AR (Chen et al., 2016) 72.4 72.4 - - - - GA Reader (Dhingra et al., 2016) 73.0 73.8 74.9 69.0 69.0 63.9 Iterative Attention (Sordoni et al., 2016) 72.6 73.3 75.2 68.6 72.1 69.2 EpiReader (Trischler et al., 2016) 73.4 74.0 75.3 69.7 71.5 67.4 AoA Reader 73.1 74.4 77.8 72.0 72.2 69.4 AoA Reader + Reranking - - 79.6 74.0 75.7 73.1 MemNN (Ensemble) 66.2 69.4 - - - - AS Reader (Ensemble) 73.9 75.4 74.5 70.6 71.1 68.9 GA Reader (Ensemble) 76.4 77.4 75.5 71.9 72.1 69.4 EpiReader (Ensemble) - - 76.6 71.8 73.6 70.6 Iterative Attention (Ensemble) 74.5 75.7 76.9 72.0 74.1 71.0 AoA Reader (Ensemble) - - 78.9 74.5 74.7 70.8 AoA Reader (Ensemble + Reranking) - - 80.3 75.6 77.0 74.1
Table 3: Results on the CNN news, CBTest NE and CN datasets. The best baseline results are depicted in italics, and the overall best results are in bold face.
Dimensions of embedding and hidden layer for each task are listed in Table 2. In re-ranking step, all language models are 8-gram, and trained by SRILM toolkit (Stolcke, 2002). The results are reported with the best model, which is selected by the performance of validation set. The ensemble model is made up of four best models, which are trained using different random seed. Implementation is done with Theano (Theano Development Team, 2016) and Keras (Chollet, 2015), and all models are trained on Tesla K40 GPU.
5.2 Overall Results
Our experiments are carried out on public datasets: CNN news datasets (Hermann et al., 2015) and CBTest NE/CN datasets (Hill et al., 2015). The statistics of these datasets are listed in Table 1, and the experimental results are given in Table 3.
As we can see that, our AoA Reader outperforms state-of-the-art systems by a large margin, where 2.3% and 2.0% absolute improvements over EpiReader in CBTest NE and CN test sets, which demonstrate the effectiveness of our model. Also by adding additional features in the re-ranking step, there is another significant boost 2.0% to
3.7% over AoA Reader in CBTest NE/CN test sets. We have also found that our single model could stay on par with the previous best ensemble system, and even we have an absolute improvement of 0.9% beyond the best ensemble model (Iterative Attention) in the CBTest NE validation set. When it comes to ensemble model, our AoA Reader also shows significant improvements over previous best ensemble models by a large margin and set up a new state-of-the-art system.
To investigate the effectiveness of employing attention-over-attention mechanism, we also compared our model to CAS Reader, which used predefined merging heuristics, such as sum or avg etc. Instead of using pre-defined merging heuristics, and letting the model explicitly learn the weights between individual attentions results in a significant boost in the performance, where 4.1% and 3.7% improvements can be made in CNN validation and test set against CAS Reader.
5.3 Effectiveness of Re-ranking Strategy
As we have seen that the re-ranking approach is effective in cloze-style reading comprehension task. To have a thorough investigation in the re-ranking
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
CBTest NE CBTest CN Valid Test Valid Test
AoA Reader 77.8 72.0 72.2 69.4 +Global LM 78.3 72.6 73.9 71.2 +Local LM 79.4 73.8 74.7 71.7 +Word-class LM 79.6 74.0 75.7 73.1
Table 4: Detailed results of 5-best re-ranking on CBTest NE/CN datasets. Each row includes all of the features from previous rows. LMglobal denotes the global LM, LMlocal denotes the local LM, LMwc denotes the word-class LM.
step, we listed the detailed improvements while adding each feature mentioned in Section 4.
From the results in Table 4, we found that the NE and CN category both benefit a lot from the re-ranking features, but the proportions are quite different. Generally speaking, in NE category, the performance is mainly boosted by the LMlocal feature. However, on the contrary, the CN category benefit from LMglobal and LMwc rather than the LMlocal.
CBTest NE CBTest CN
NN 0.64 0.20 Global LM 0.16 0.10 Word-class LM 0.04 0.39 Local LM 0.16 0.31 RATIO η 1.25 1.58
Table 5: Weight of each feature in N-best reranking step. NN denotes the feature (probability) produced by baseline neural network model.
Also, we listed the weights of each feature in Table 5. The LMglobal and LMwc are all trained by training set, which can be seen as Global Feature. However, the LMlocal is only trained within the respective document part of test sample, which can be seen as Local Feature.
η = LMglobal + LMwc
LMlocal (13)
We calculated the ratio between the global and local features and found that the NE category is much more dependent on local features than CN category. Because it is much more likely to meet a new named entity than a common noun in the test phase, so adding the local LM provides much more information than that of common
noun. However, on the contrary, answering common noun requires less local information, which can be learned in the training data relatively.
6 Quantitative Analysis
In this section, we will give a quantitative analysis to our AoA Reader. The following analyses are carried out on CBTest NE dataset. First, we investigate the relations between the length of the document and corresponding accuracy. The result is depicted in Figure 2.
As we can see that the AoA Reader shows consistent improvements over AS Reader on the different length of the document. Especially, when the length of document exceeds 700, the improvements become larger, indicating that the AoA Reader is more capable of handling long documents.
18 486 758 525 370 262 61
AoA Reader AS Reader
Ac cu
ra cy
0.65
0.70
0.75
0.80
0.85
0.90
Length of Document 100 200 300 400 500 600 700 800
Figure 2: Test accuracy against the length of the document. The bar below the figure indicates the number of samples in each interval.
Furthermore, we also investigate if the model tends to choose a high-frequency candidate than a lower one, which is shown in Figure 3. Not surprisingly, we found that both models do a good job when the correct answer appear much frequent in the document than the other candidates. This is because that the correct answer that has the highest frequency among the candidates takes up over 40% of the test set (1071 out of 2500). But interestingly we have also found that, when the frequency rank of correct answer exceeds 7 (less frequent among candidates), these models also give a relatively high performance. Empirically, we think that these models tend to choose extreme cases in terms of candidate frequency (either too
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
1071 588 354 264 127 59 28 8 1 1
AoA Reader AS Reader
Ac cu
ra cy
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Rank of the answer 1 2 3 4 5 6 7 8 9 10
Figure 3: Test accuracy against the frequency rank of the answer. The bar below the figure indicates the number of samples in each rank.
high or too low). One possible reason is that the model is hard to choose a candidate that has a neutral frequency as the correct answer, because of its ambiguity (neutral choices are hard to made).
7 Related Work
Cloze-style reading comprehension tasks have been widely investigated in recent studies. We will take a brief revisit to the related works.
Hermann et al. (2015) have proposed a method for obtaining large quantities of 〈D,Q,A〉 triples through news articles and its summary. Along with the release of cloze-style reading comprehension dataset, they also proposed an attention-based neural network to handle this task. Experimental results showed that the proposed neural network is effective than traditional baselines.
Hill et al. (2015) released another dataset, which stems from the children’s books. Different from Hermann et al. (2015)’s work, the document and query are all generated from the raw story without any summary, which is much more general than previous work. To handle the reading comprehension task, they proposed a window-based memory network, and self-supervision heuristics is also applied to learn hard-attention.
Unlike previous works, that using blended representations of document and query to estimate the answer, Kadlec et al. (2016) proposed a simple model that directly pick the answer from the document, which is motivated by the Pointer Network (Vinyals et al., 2015). A restriction of this model is that the answer should be a single word
and appear in the document. Results on various public datasets showed that the proposed model is effective than previous works.
Liu et al. (2016) proposed to exploit reading comprehension models to other tasks. They first applied the reading comprehension model into Chinese zero pronoun resolution task with automatically generated large-scale pseudo training data. The experimental results on OntoNotes 5.0 data showed that their method significantly outperforms various state-of-the-art systems.
Our work is primarily inspired by Cui et al. (2016) and Kadlec et al. (2016) , where the latter model is widely applied to many follow-up works (Sordoni et al., 2016; Trischler et al., 2016; Cui et al., 2016). Unlike the CAS Reader (Cui et al., 2016), we do not assume any heuristics to our model, such as using merge functions: sum, avg etc. We used a mechanism called “attentionover-attention” to explicitly calculate the weights between different individual document-level attentions, and get the final attention by computing the weighted sum of them. Also, we find that our model is typically general and simple than the recently proposed model, and brings significant improvements over these cutting edge systems.
8 Conclusion
We present a novel neural architecture, called attention-over-attention reader, to tackle the clozestyle reading comprehension task. The proposed AoA Reader aims to compute the attentions not only for the document but also the query side, which will benefit from the mutual information. Then a weighted sum of attention is carried out to get an attended attention over the document for the final predictions. Among several public datasets, our model could give consistent and significant improvements over various state-of-theart systems by a large margin.
The future work will be carried out in the following aspects. We believe that our model is general and may apply to other tasks as well, so firstly we are going to fully investigate the usage of this architecture in other tasks. Also, we are interested to see that if the machine really “comprehend” our language by utilizing neural networks approaches, but not only serve as a “document-level” language model. In this context, we are planning to investigate the problems that need comprehensive reasoning over several sentences.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  -- A well-motivated approach, with a clear description and solid results.  - Weaknesses:  -- Nothing substantial other than the comments below.   - General Discussion:  The paper describes a new method called attention-over-attention for reading comprehension. First layers of the network compute a vector for each query word and document word, resulting in a |Q|xK matrix for the query and a |D|xK for the document. Since the answer is a document word, an attention mechanism is used for assigning weights to each word, depending on their interaction with query words. In this work, the authors deepen a traditional attention mechanism by computing a weight for each query word through a separate attention and then using that to weight the main attention over document words. Evaluation is properly conducted on benchmark datasets, and various insights are presented through an analysis of the results as well as a comparison to prior work. I think this is a solid piece of work on an important problem, and the method is well-motivated and clearly described, so that researchers can easily reproduce results and apply the same techniques to other similar tasks.  - Other remarks:  -- p4, Equation 12: I am assuming i is iterating over training set and p(w) is referring to P(w|D,Q) in the previous equation? Please clarify to avoid confusion.  -- I am wondering whether you explored discussed initializing word embeddings with existing vectors such as Google News or Glove? Is there a reason to believe the general-purpose word semantics would not be useful in this task?  -- p6 L589-592: It is not clear what the authors are referring to when they say 'letting the model explicitly learn weights between individual attentions'? Is this referring to their own architecture, more specifically the GRU output indirectly affecting how much attention will be applied to each query and document word? Clarifying that would be useful. Also, I think the improvement on validation is not 4.1, rather 4.0 (72.2-68.2).  -- p7 Table 5: why do you think the weight for local LM is relatively higher for the CN task while the benefit of adding it is less? Since you included the table, I think it'll be nice to provide some insights to the reader.  -- I would have liked to see the software released as part of this submission.  -- Typo p2 L162 right column:  is not that effective than expected  -to  is not as effective as expected ?  -- Typo p7 L689 right column:  appear much frequent  -to  appears more frequently ?  -- Typo p8 L719-721 left column:  the model is hard to  -to  it is hard for the model to ? &  hard to made  -to  hard to make ?