Selective Encoding for Abstractive Sentence Summarization
We propose a selective encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization. It consists of a sentence encoder, a selective gate network, and an attention equipped decoder. The sentence encoder and decoder are built with recurrent neural networks. The selective gate network constructs a second level sentence representation by controlling the information flow from encoder to decoder. The second level representation is tailored for sentence summarization task, which leads to better performance. We evaluate our model on the English Gigaword, DUC 2004 and MSR abstractive sentence summarization datasets. The experimental results show that the proposed selective encoding model outperforms the state-ofthe-art baseline models.
1 Introduction
Sentence summarization aims to shorten a given sentence and produce a brief summary of it. This is different from document level summarization task since it is hard to apply existing techniques in extractive methods, such as extracting sentence level features and ranking sentences. Early works propose using rule-based methods (Zajic et al., 2007), syntactic tree pruning methods (Knight and Marcu, 2002), statistical machine translation techniques (Banko et al., 2000) and so on for this task. We focus on abstractive sentence summarization task in this paper.
Recently, neural network models have been applied in this task. Rush et al. (2015) use autoconstructed sentence-headline pairs to train a neural network summarization model. They use a
Convolutional Neural Network (CNN) encoder and feed-forward neural network language model decoder for this task. Chopra et al. (2016) extend their work by replacing the decoder with Recurrent Neural Network (RNN). Nallapati et al. (2016) follow this line and change the encoder to RNN to make it a full RNN based sequence-tosequence model (Sutskever et al., 2014).
All the above works fall into the encodingdecoding paradigm, which first encodes the input sentence to an abstract representation and then decodes the intended output sentence based on the encoded information. As an extension of the encoding-decoding framework, attentionbased approach (Bahdanau et al., 2015) has been broadly used: the encoder produces a list of vectors for all tokens in the input, and the decoder uses an attention mechanism to dynamically extract encoded information and align with the output tokens. This approach achieves huge success in tasks like neural machine translation, where alignment between all parts of the input and output are required. However, in abstractive sentence summarization, there is no explicit alignment relationship between input sentence and the summary except for the extracted common words. The
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
challenge here is not to infer the alignment, but to select the highlights while filtering out secondary information in the input. A desired work-flow for abstractive sentence summarization is encoding, selection, and decoding. After selecting the important information from an encoded sentence, the decoder produces the output summary using the selected information. For example, in Figure 1, given the input sentence, the summarization system first selects the important information, and then rephrases or paraphrases to produce a well-organized summary. Although this is implicitly modeled in the encoding-decoding framework, we argue that abstractive sentence summarization shall benefit from explicitly modeling this selection process.
In this paper we propose Selective Encoding for Abstractive Sentence Summarization (SEASS). We treat the sentence summarization as a threephase task: encoding, selection, and decoding. It consists of a sentence encoder, a selective gate network, and a summary decoder. First, the sentence encoder reads the input words through an RNN unit to construct the first level sentence representation. Then the selective gate network selects the encoded information to construct the second level sentence representation. The selective mechanism controls the information flow from encoder to decoder by applying a gate network according to the sentence information, which helps improve encoding effectiveness and release the burden of the decoder. Finally, the attention-equipped decoder generates the summary using the second level sentence representation. We conduct experiments on English Gigaword, DUC 2004 and Microsoft Research Abstractive Text Compression test sets. Our SEASS model achieves 17.54 ROUGE-2 F1, 9.56 ROUGE-2 recall and 10.63 ROUGE-2 F1 on these test sets respectively, which improves performance compared to the state-of-the-art methods.
2 Problem Formulation
For sentence summarization, given an input sentence x = (x1, x2, . . . , xn), where n is the sentence length, xi ∈ Vs and Vs is the source vocabulary, the system summarizes x by producing y = (y1, y2, . . . , yl), where l ≤ n is the summary length , yi ∈ Vt and Vt is the target vocabulary.
If |y| ⊆ |x|, which means all words in summary y must appear in given input, we denote this as extractive sentence summarization. If |y| * |x|,
which means not all words in summary come from input sentence, we denote this as abstractive sentence summarization. Table 1 provides an example. We focus on abstracive sentence summarization task in this paper.
3 Model
As shown in Figure 2, our model consists of a sentence encoder using the Gated Recurrent Unit (GRU) (Cho et al., 2014), a selective gate network and an attention-equipped GRU decoder. First, the bidirectional GRU encoder reads the input words x = (x1, x2, . . . , xn) and builds its representation (h1, h2, . . . , hn). Then the selective gate selects and filters the word representations according to the sentence meaning representation to produce a tailored sentence word representation for abstractive sentence summarization task. Lastly, the GRU decoder produces the output summary with attention to the tailored representation. In the following sections, we introduce the sentence encoder, the selective mechanism, and the summary decoder respectively.
3.1 Sentence Encoder
The role of the sentence encoder is to read the input sentence and construct the basic sentence representation. Here we employ a bidirectional GRU (BiGRU) as the recurrent unit, where GRU is defined as:
zi = σ(Wz[xi, hi−1]) ri = σ(Wr[xi, hi−1])
h̃i = tanh(Wh[xi, ri hi−1])
hi = (1− zi) hi−1 + zi h̃i
(1)
(2)
(3)
(4)
The BiGRU consists of a forward GRU and a backward GRU. The forward GRU reads the input sentence word embeddings from left to right and gets a sequence of hidden states, (~h1,~h2, . . . ,~hn). The backward GRU reads the input sentence
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
𝑥1
ℎ1
𝑥2
ℎ2
𝑥3
ℎ3
𝑥4
ℎ4
𝑥5
ℎ5
𝑥6
ℎ6
ℎ1′ ℎ2′ ℎ3′ ℎ4′ ℎ5′ ℎ6′
ℎ𝑖 𝑠
MLP GRU
Attention
𝑐𝑡−1 𝑠𝑡−1 𝑦𝑡−1
𝑠𝑡
𝑐𝑡
maxout
𝑦𝑡
Selective Gate Network
Encoder
Decoder
softmax
embeddings reversely, from right to left, and results in another sequence of hidden states, ( ~h1, ~h2, . . . , ~hn):
~hi = GRU(xi,~hi−1) ~hi = GRU(xi, ~hi+1)
(5)
(6)
The initial states of the BiGRU are set to zero vectors, i.e., ~h1 = 0 and ~hn = 0. After reading the sentence, the forward and backward hidden states are concatenated, i.e., hi = [~hi; ~hi], to get the basic sentence representation.
3.2 Selective Mechanism
In the sequence-to-sequence machine translation (MT) model, the encoder and decoder are responsible for encoding input sentence information and decoding the sentence representation to generate an output sentence. Some previous works apply this framework to summarization generation tasks. However, abstractive sentence summarization is different from MT in two ways. First, there is no explicit alignment relationship between the input sentence and the output summary except for the common words. Second, summarization task needs to keep the highlights and remove the unnecessary information, while MT needs to keep all information literally.
Herein, we propose a selective mechanism to model the selection process for abstractive sentence summarization. The selective mechanism extends the sequence-to-sequence model by constructing a tailored representation for abstractive sentence summarization task. Concretely, the selective gate network in our model takes two vector
inputs, the sentence word vector hi and the sentence representation vector s. The sentence word vector hi is the output of the BiGRU encoder and represents the meaning and context information of word xi. The sentence vector s is used to represent the meaning of the sentence. For each word xi, the selective gate network generates a gate vector sGatei using hi and s, then the tailored representation is constructed, i.e., h′i.
In detail, we concatenate the last forward hidden state ~hn and backward hidden state ~h1 as the sentence representation s:
s =
[ ~h1
~hn
] (7)
For each time step i, the select gate takes the sentence representation s and BiGRU hidden hi as inputs to compute the select gate sGatei:
sGatei = σ(Wshi +Uss+ b) h′i = hi sGatei (8) (9)
where σ is sigmoid activation function and is element-wise multiplication. After the selective gate network, we obtain another sequence of vectors (h′1, h ′ 2, . . . , h ′ n). This new sequence is then used as the input sentence representation for the decoder to generate the summary.
3.3 Summary Decoder
On top of the sentence encoder and the selective gate network, we use GRU with attention as the decoder to produce the output summary.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
At each decoding time step t, the GRU reads the previous word embedding wt−1 and previous context vector ct−1 as inputs to compute the new hidden state st. To initialize the GRU hidden state, we use a linear layer with the last backward encoder hidden state ~h1 as input:
st = GRU(wt−1, ct−1, st−1)
s0 = tanh(Wd ~h1 + b)
(10)
(11)
The context vector ct for current time step t is computed through the concatenate attention mechanism (Luong et al., 2015), which matches the current decoder state st with each encoder hidden state h′i to get an importance score. The importance scores are then normalized to get the current context vector by weighted sum:
et,i = v > a tanh(Wast−1 +Uah ′ i) αt,i = exp(et,i)∑n i=1 exp(et,i)
ct = n∑
i=1
αt,ih ′ i
(12)
(13)
(14)
We then combine the previous word embedding wt−1, the current context vector ct, and the decoder state st to construct the readout state rt. The readout state is then passed through a maxout hidden layer (Goodfellow et al., 2013) to predict the next word with a softmax layer over the decoder vocabulary.
rt = Wrwt−1 +Urct +Vrst
mt = [max{rt,2j−1, rt,2j}]>j=1,...,d p(yt|y1, . . . , yt−1) = softmax(Womt)
(15)
(16)
(17)
where rt is a 2d-dimensional vector.
3.4 Objective Function
Our goal is to maximize the output summary probability given the input sentence. Therefore, we optimize the negative log-likelihood loss function:
J(θ) = − 1 |D| ∑ (x,y)∈D log p(y|x) (18)
where D denotes a set of parallel sentencesummary pairs and θ is the model parameter. We use Stochastic Gradient Descent (SGD) with minibatch to learn the model parameter θ.
4 Experiments
In this section we introduce the dataset we use, the evaluation metric, the implementation details, the baselines we compare to, and the performance of our system.
4.1 Dataset
Training Set For our training set, we use a parallel corpus which is constructed from the Annotated English Gigaword dataset (Napoles et al., 2012) as mentioned in Rush et al. (2015). The parallel corpus is produced by pairing the first sentence and the headline in the news article with some heuristic rules. We use the script1 released by Rush et al. (2015) to pre-process and extract the training and development datasets. The script performs various basic text normalization, including PTB tokenization, lower-casing, replacing all digit characters with #, and replacing word types seen less than 5 times with 〈unk〉. The extracted corpus contains about 3.8M sentence-summary pairs for the training set and 189K examples for the development set.
For our test set, we use the English Gigaword, DUC 2004, and Microsoft Research Abstractive Text Compression test sets.
English Gigaword Test Set We randomly sample 8000 pairs from the extracted development set as our development set since it is relatively large. For the test set, we use the same randomly heldout test set of 2000 sentence-summary pairs as Rush et al. (2015).2
We also find that except for the empty titles, this test set has some invalid lines like the input sentence containing only one word. Therefore, we further sample 2000 pairs as our internal test set and release it for future works.
DUC 2004 Test Set We employ DUC 2004 data for tasks 1 & 2 (Over et al., 2007) in our experiments as one of the test sets since it is too small to train a neural network model on. The dataset pairs each document with 4 different human-written reference summaries which are capped at 75 bytes. It has 500 input sentences with each sentence paired with 4 summaries.
1https://github.com/facebook/NAMAS 2Thanks to Rush et al. (2015), we acquired the test set they used. Following Chopra et al. (2016), we remove pairs with empty titles resulting in slightly different accuracy compared to Rush et al. (2015) for their systems. The cleaned test set contains 1951 sentence-summary pairs.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
MSR-ATC Test Set Toutanova et al. (2016) release a new dataset for sentence summarization task by crowdsourcing. This dataset contains approximately 6,000 source text sentences with multiple manually-created summaries (about 26,000 sentence-summary pairs in total). Toutanova et al. (2016) provide a standard split of the data into training, development, and test sets, with 4,936, 448 and 785 input sentences respectively. Since the training set is too small, we only use the test set as one of our test sets. We denote this dataset as MSR-ATC (Microsoft Research Abstractive Text Compression) test set in the following.
Table 2 summarizes the statistic information of the three datasets we used.
Data Set Giga DUC† MSR†
#(sent) 3.99M 500 785 #(sentWord) 125M 17.8K 29K #(summWord) 33M 20.9K 85.9K #(ref) 1 4 3-5 avgInputLen 31.35 35.56 36.97 avgSummLen 8.23 10.43 25.5
Table 2: Data statistics for the datasets. †DUC 2004 and MSR-ATC are for test purpose only.
4.2 Evaluation Metric
We employ ROUGE (Lin, 2004) as our evaluation metric. ROUGE measures the quality of summary by computing overlapping lexical units, such as unigram, bigram, trigram, and longest common subsequence (LCS). It becomes the standard evaluation metric for DUC shared tasks and popular for summarization evaluation. Following previous work, we use ROUGE-1 (unigram), ROUGE-2 (bigram) and ROUGE-L (LCS) as the evaluation metrics in the reported experimental results.
4.3 Implementation Details
Model Parameters The input and output vocabularies are collected from the training data, which have 119,504 and 68,883 word types respectively. We set the word embedding size to 300 and all GRU hidden state sizes to 512. We use dropout (Srivastava et al., 2014) with probability p = 0.5.
Model Training We initialize model parameters randomly using a Gaussian distribution with Xavier scheme (Glorot and Bengio, 2010). We use a combination of Adam (Kingma and Ba, 2015)
and simple SGD as our the optimizing algorithms. The training is separated into two phases, the first phase is optimizing the loss function with Adam and the second is with simple SGD. For the Adam optimizer, we set the learning rate α = 0.001, two momentum parameters β1 = 0.9 and β2 = 0.999 respectively, and = 10−8. We use Adam optimizer until the training perplexity decrease to threshold 50. Then we switch to a simple SGD optimizer with initial learning rate α = 0.5 and halve it for every 300K batches. We also apply gradient clipping (Pascanu et al., 2013) with range [−5, 5] for both Adam and SGD phases. To both speed up the training and converge quickly, we use mini-batch size 64 by grid search.
Beam Search We use beam search to generate multiple summary candidates to get better results. To avoid favoring shorter outputs, we average the ranking score along the beam path by dividing it by the number of generated words. To both decode fast and get better results, we set the beam size to 12 in our experiments.
4.4 Baseline
We compare SEASS model with the following state-of-the-art baselines:
ABS Rush et al. (2015) use a CNN encoder and NNLM decoder to do the sentence summarization task. ABS+ Based on ABS model, Rush et al. (2015) further tune their model using DUC 2003 dataset, which leads to improvements on DUC 2004 test set. CAs2s As an extension of the ABS model, Chopra et al. (2016) use a convolutional attention-based encoder and RNN decoder, which outperforms the ABS model. Feats2s Nallapati et al. (2016) use a full RNN sequence-to-sequence encoder-decoder model and add some features to enhance the encoder, such as POS tag, NER, and so on. Luong-NMT Neural machine translation model of Luong et al. (2015) with two-layer LSTMs for the encoder-decoder with 500 hidden units in each layer implemented in (Chopra et al., 2016). s2s+att We also implement a sequence-tosequence model with attention as our baseline and denote it as “s2s+att”.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
590
591
592
593
594
595
596
597
598
599
4.5 Results
We report ROUGE F1, ROUGE recall and ROUGE F1 for English Gigaword, DUC 2004 and MSRATC test sets respectively. We use the official ROUGE script (version 1.5.5) 3 to evaluate the summarization quality in our experiments. For English Gigaword4 and MSR-ATC5 test sets, the outputs have different lengths so we evaluate the system with F1 metric. As for the DUC 2004 test set6, the task requires the system to produce a fixed length summary (75 bytes), therefore we employ ROUGE recall as the evaluation metric. To satisfy the length requirement, we decode the output summary to a roughly expected length following Rush et al. (2015).
English Gigaword We acquire the test set from Rush et al. (2015) so we can make fair comparisons to the baselines.
Models RG-1 RG-2 RG-L
ABS (beam)‡ 29.55- 11.32- 26.42- ABS+ (beam)‡ 29.76- 11.88- 26.96- Feats2s (beam)‡ 32.67- 15.59- 30.64- CAs2s (greedy)‡ 33.10- 14.45- 30.25- CAs2s (beam)‡ 33.78- 15.97- 31.15- Luong-NMT (beam)‡ 33.10- 14.45- 30.71- s2s+att (greedy) 33.18- 14.79- 30.80- s2s+att (beam) 34.04- 15.95- 31.68- SEASS (greedy) 35.48 16.50 32.93 SEASS (beam) 36.15 17.54 33.63
Table 3: Full length ROUGE F1 evaluation results on the English Gigaword test set used by Rush et al. (2015). RG in the Table denotes ROUGE. Results with ‡ mark are taken from the corresponding papers. The supper script - indicates that our SEASS model with beam search performs significantly better than it as given by the 95% confidence interval in the official ROUGE script.
In Table 3, we report the ROUGE F1 score of our model and the baseline methods. Our SEASS model with beam search outperforms all baseline models by a large margin. Even for greedy search, our model still performs better than other methods
3http://www.berouge.com/ 4The ROUGE evaluation option is the same as Rush et al.
(2015), -m -n 2 -w 1.2 5The ROUGE evaluation option is, -m -n 2 -w 1.2 6The ROUGE evaluation option is, -m -b 75 -n 2 -w 1.2
which used beam search. For the popular ROUGE2 metric, our SEASS model achieves 17.54 F1 score and performs better than the previous works. Compared to the ABS model, our model has a 6.22 ROUGE-2 F1 relative gain. Compared to the highest CAs2s baseline, our model achieves 1.75 ROUGE-2 F1 improvement and passes the significant test according to the official ROUGE script.
Table 4 summarizes our results on our internal test set using ROUGE F1 evaluation metrics. The performance on our internal test set is comparable to our development set, which achieves 24.58 ROUGE-2 F1 and outperforms the baselines.
DUC 2004 We evaluate our model using the ROUGE recall score since the reference summaries
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
of the DUC 2004 test set are capped at 75 bytes. Therefore, we decode the summary to a fixed length 18 to ensure that the generated summary satisfies the minimum length requirement. As summarized in Table 5, our SEASS outperforms all the baseline methods and achieves 29.21, 9.56 and 25.51 for ROUGE 1, 2 and L recall. Compared to the ABS+ model which is tuned using DUC 2003 data, our model performs significantly better by 1.07 ROUGE-2 recall score and is trained only with English Gigaword sentence-summary data without being tuned using DUC data.
MSR-ATC We report the full length ROUGE F1 score on the MSR-ATC test set in Table 6. To the best of our knowledge, this is the first work that reports ROUGE metric scores on the MSR-ATC dataset. Note that we only compare our model with ABS since the others are not publicly available. Our SEASS achieves 10.63 ROUGE-2 F1 and outperforms the s2s+att baseline by 1.02 points.
Models RG-1 RG-2 RG-L
ABS (beam) 20.27- 5.26- 17.10- s2s+att (greedy) 15.15- 4.48- 13.62- s2s+att (beam) 22.65- 9.61- 21.39- SEASS (greedy) 19.77 6.44 17.36 SEASS (beam) 25.75 10.63 22.90
Table 6: Full length ROUGE F1 evaluation on MSR-ATC test set. Beam search are used in both the baselines and our method. The supper script - indicates that our SEASS model performs significantly better than it as given by the 95% confidence interval in the official ROUGE script.
5 Discussion
In this section, we first compare the performance of SEASS with the s2s+att baseline model to illustrate that the proposed method succeeds in selecting information and building tailored representation for abstractive sentence summarization. We then analyze selective encoding by visualizing the heat map.
Effectiveness of Selective Encoding We further test the SEASS model with different sentence lengths on English Gigaword test sets, which are merged from the Rush et al. (2015) test set and our internal test set. The length of sentences in the test sets ranges from 10 to 80. We group the sentences
with an interval of 4 and get 18 different groups and we draw the first 14 groups. We find that the performance curve of our SEASS model always appears to be on the top of that of s2s+att with a certain margin. For the groups of 16, 20, 24, 32, 56 and 60, the SEASS model obtains big improvements compared to the s2s+att model. Overall, these improvements on all groups indicate that the selective encoding method benefits the abstractive sentence summarization task.
Saliency Heat Map of Selective Gate Since the output of the selective gate network is a high dimension vector, it is hard to visualize all the gate values. We use the method in Li et al. (2016) to visualize the contribution of the selective gate to the final output, which can be approximated by the first derivative. Given sentence words x with associated output summary y, the trained model associates the pair (x, y) with a score Sy(x). The goal is to decide which gate g associated with x makes the most significant contribution to Sy(x). We approximate the Sy(g) by computing the first-order Taylor expansion
Sy(g) ≈ ∂(Sy)
∂g g + b (19)
We then draw the Euclidean norm of the first derivative of the output y with respect to the selective gate g associated with each input words.
Figure 4 shows an example of the first derivative heat map, in which most of the important words are selected by the selective gate such as “europe”, “slammed”, “unacceptable”, “conditions”,
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
th e co u n ci
l
o f e u ro
p e
's h u m
a n
ri g h ts co m
m is
si o n e r
sl a m
m e d
th u rs
d a y
a s `` u n a cc
e p ta
b le
'' co n d it
io n s
in fr a n ce 's o v e rc
ro w
d e d
a n d d ila
p id
a te
d
ja ils , w h e re so m e # # in m
a te
s
h a v e co m
m it
te d
su ic
id e
th is y e a r .
Figure 4: First derivative heat map of the output with respect to the selective gate. The important words are selected in the input sentence, such as “europe”, “slammed” and “unacceptable”. The output summary of our system is “council of europe slams french prison conditions” and the true summary is “council of europe again slams french prison conditions”.
and “france”. We can observe that the selective gate determines the importance of each word before decoder, which releases the burden of it by providing tailored sentence encoding.
6 Related Work
Abstractive sentence summarization, also known as sentence compression and similar to headline generation, is used to help compress or fuse the selected sentences in extractive document summarization systems since they may inadvertently include unnecessary information. There are some previous methods to solve this task, such as the linguistic rule-based method (Dorr et al., 2003). As for the statistical machine learning based methods, Banko et al. (2000) apply statistical machine translation techniques by modeling headline generation as a translation task and use 8000 article-headline pairs to train the system.
Rush et al. (2015) propose leveraging news data in Annotated English Gigaword (Napoles et al., 2012) corpus to construct large scale parallel data for sentence summarization task. They propose an ABS model, which consists of an attentive Convolutional Neural Network encoder and an neural network language model (Bengio et al., 2003) decoder. Chopra et al. (2016) extend this work, which keeps the CNN encoder but replaces the decoder with recurrent neural networks. Nallapati et al. (2016) further change the encoder to an RNN encoder, which leads to a full RNN sequence-tosequence model. Besides, they enrich the encoder with lexical and statistic features which play important roles in traditional feature based summarization systems, such as NER and POS tags, to improve performance. Experiments on the Gigaword and DUC 2004 test sets show that the above models achieve state-of-the-art results.
Gu et al. (2016) and Gulcehre et al. (2016) come up similar ideas that summarization task can
benefit from copying words from input sentences. Gu et al. (2016) propose CopyNet to model the copying action in response generation, which also applies for summarization task. Gulcehre et al. (2016) propose a switch gate to control whether to copy from source or generate from decoder vocabulary. Zeng et al. (2016) also propose using copy mechanism and add an extra gate on the GRU/LSTM gate for this task. Cheng and Lapata (2016) use an RNN based encoder-decoder for extractive summarization of documents
Yu et al. (2016) propose a segment to segment neural transduction model for sequence-tosequence framework. The model introduces a latent segmentation which determines correspondences between tokens of the input sequence and the output sequence. Shen et al. (2016) propose applying Minimum Risk Training (MRT) in NMT to directly optimize the evaluation metrics. Ayana et al. (2016) apply MRT on abstractive sentence summarization task and the results show that optimizing for ROUGE improves the test performance.
7 Conclusion
This paper proposes a selective encoding model which extends the sequence-to-sequence model for abstractive sentence summarization task. The selective mechanism mimics one of the human summarizers’ behaviors, selecting important information before writing down the summary. With the proposed selective mechanism, we build an end-to-end neural network summarization model which consists of three phases: encoding, selection, and decoding. Experimental results show that the selective encoding model greatly improves the performance with respect to the state-of-theart methods on English Gigaword, DUC 2004 and MSR-ATC test sets.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  The authors propose a selective encoding model as extension to the sequence-to-sequence framework for abstractive sentence summarization. The paper is very well written and the methods are clearly described. The proposed methods are evaluated on standard benchmarks and comparison to other state-of-the-art tools are presented, including significance scores.   - Weaknesses:  There are some few details on the implementation and on the systems to which the authors compared their work that need to be better explained.   - General Discussion:  * Major review:  - I wonder if the summaries obtained using the proposed methods are indeed abstractive. I understand that the target vocabulary is build out of the words which appear in the summaries in the training data. But given the example shown in Figure 4, I have the impression that the summaries are rather extractive. The authors should choose a better example for Figure 4 and give some statistics on the number of words in the output sentences which were not present in the input sentences for all test sets.  - page 2, lines 266-272: I understand the mathematical difference between the vector hi and s, but I still have the feeling that there is a great overlap between them. Both  represent the meaning . Are both indeed necessary? Did you trying using only one of them.  - Which neural network library did the authors use for implementing the system? There is no details on the implementation.  - page 5, section 44: Which training data was used for each of the systems that the authors compare to? Diy you train any of them yourselves?  * Minor review:  - page 1, line 44: Although the difference between abstractive and extractive summarization is described in section 2, this could be moved to the introduction section. At this point, some users might no be familiar with this concept.  - page 1, lines 93-96: please provide a reference for this passage:  This approach achieves huge success in tasks like neural machine translation, where alignment between all parts of the input and output are required.   - page 2, section 1, last paragraph: The contribution of the work is clear but I think the authors should emphasize that such a selective encoding model has never been proposed before (is this true?). Further, the related work section should be moved to before the methods section.  - Figure 1 vs. Table 1: the authors show two examples for abstractive summarization but I think that just one of them is enough. Further, one is called a figure while the other a table.  - Section 3.2, lines 230-234 and 234-235: please provide references for the following two passages:  In the sequence-to-sequence machine translation (MT) model, the encoder and decoder are responsible for encoding input sentence information and decoding the sentence representation to generate an output sentence ;  Some previous works apply this framework to summarization generation tasks.   - Figure 2: What is  MLP ? It seems not to be described in the paper.  - page 3, lines 289-290: the sigmoid function and the element-wise multiplication are not defined for the formulas in section 3.1.  - page 4, first column: many elements of the formulas are not defined: b (equation 11), W (equation 12, 15, 17) and U (equation 12, 15), V (equation 15).  - page 4, line 326: the readout state rt is not depicted in Figure 2 (workflow).  - Table 2: what does  #(ref)  mean?  - Section 4.3, model parameters and training. Explain how you achieved the values to the many parameters: word embedding size, GRU hidden states, alpha, beta 1 and 2, epsilon, beam size.  - Page 5, line 450: remove  the  word in this line?  SGD as our optimizing algorithms  instead of  SGD as our the optimizing algorithms.   - Page 5, beam search: please include a reference for beam search.  - Figure 4: Is there a typo in the true sentence?  council of europe again slams french prison conditions  (again or against?)  - typo  supper script  to  superscript  (4 times)


Singlish can be interesting to the ACL community both linguistically as a major creole based on English, and computationally for information extraction and sentiment analysis of regional social media. We investigate dependency parsing of Singlish by constructing a dependency treebank under the Universal Dependencies scheme, and then training a neural network model by integrating English syntactic knowledge into a state-ofthe-art parser trained on the Singlish treebank. Results show that English knowledge can lead to 36% relative error reduction, resulting in a parser of around 85% accuracies. We make both our annotation and parser available for further research.
1 Introduction
Languages evolve temporally and geographically, both in vocabulary as well as in syntactic structures. When major languages such as English or French are adopted in another culture as the primary language, they often mix with existing languages or dialects in that culture and evolve into a stable language called a creole. Examples of creoles include the French-based Haitian Creole, and Colloquial Singaporean English (Singlish) (MianLian and Platt, 1993), an English-based creole. While the majority of the natural language processing (NLP) research attention has been focused on the major languages, little work has been done on adapting the components to creoles. One notable body of work originated from the featured translation task of the EMNLP 2011 Workshop on Statistical Machine Translation (WMT11) to translate Haitian Creole SMS messages sent during the 2010 Haitian earthquake. This work high-
lights the importance of NLP tools on creoles in crisis situations for emergency relief (Hu et al., 2011; Hewavitharana et al., 2011).
Singlish is one of the major languages in Singapore, with borrowed vocabulary and grammars from a number of languages including Malay, Tamil, and Chinese dialects such as Hokkien, Cantonese and Teochew (Leimgruber, 2009, 2011), and it has been increasingly used in written forms on web media. Fluent English speakers unfamiliar with Singlish would find the creole hard to comprehend (Harada, 2009). Correspondingly, fundamental English NLP components such as POS taggers and dependency parsers perform poorly on such Singlish texts based on our observations. For example, Seah et al. (2015) adapted the Socher et al. (2013) sentiment analysis engine to the Singlish vocabulary, but failed to adapt the parser. Since dependency parsers are important for tasks such as information extraction (Miwa and Bansal, 2016) and discourse parsing (Li et al., 2015), this hinders the development of such downstream applications for Singlish in written forms and thus makes it crucial to build a dependency parser that can perform well natively on Singlish.
To address this issue, we start with investigating the linguistic characteristics of Singlish and specifically the causes of difficulties for understanding Singlish with English syntax. We found that, despite the obvious attribute of inheriting a large portion of basic vocabularies and grammars from English, Singlish not only imports terms from regional languages and dialects, its lexical semantics and syntax also deviate significantly from English (Leimgruber, 2009, 2011). We categorize the challenges and formalize their interpretation using Universal Dependencies (Nivre et al., 2016), which extends to the creation of a Singlish dependency treebank with 1,200 sentences.
Based on the intricate relationship between
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Singlish dependency
parser trained with small
Singlish treebank
English syntactic and
semantic knowledge learnt
from large treebank
Singlish sentences
Singlish dependency trees
Figure 1: Overall model diagram
Singlish and English, we build a Singlish parser by leveraging knowledge of English syntax as a basis. This overall approach is illustrated in Figure 1. In particular, we train a basic Singlish parser with the best off-the-shelf neural dependency parsing model using biaffine attention (Dozat and Manning, 2016), and improve it with knowledge transfer by adopting neural stacking (Chen et al., 2016; Zhang and Weiss, 2016) to integrate the English syntax. Since POS tags are important features for dependency parsing (Chen and Manning, 2014; Dyer et al., 2015), we train a POS tagger for Singlish following the same idea by integrating English POS knowledge using neural stacking (Chen et al., 2016; Zhang and Weiss, 2016).
Results show that English syntax knowledge brings 31.58% and 36.51% relative error reduction on POS tagging and dependency parsing respectively, resulting in a Singlish dependency parser with 84.21% unlabeled attachment score (UAS).
We make our Singlish dependency treebank, the source code for training a dependency parser and the trained model for the parser with the best performance freely available online1.
2 Related Work
Neural networks have led to significant advance in the performance for dependency parsing, including transition-based parsing (Chen and Manning, 2014; Zhou et al., 2015; Weiss et al., 2015; Dyer et al., 2015; Ballesteros et al., 2015; Andor et al., 2016), and graph-based parsing (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016). In particular, the biaffine attention method of Dozat and Manning (2016) uses deep bi-directional long
1https://github.com/ANONYMIZED/ ANONYMIZED
short-term memory (bi-LSTM) networks for highorder non-linear feature extraction, producing the state-of-the-art performance for English dependency parsing. We adopt this model as the basis for our Singlish parser.
Our work belongs to a line of work on transfer learning for parsing, which leverages English resources in Universal Dependencies to improve the parsing accuracies of low-resource languages (Hwa et al., 2005; Cohen and Smith, 2009; Ganchev et al., 2009). Seminal work employed statistical models. McDonald et al. (2011) investigated delexicalized transfer, where word-based features are removed from a statistical model for English, so that POS and dependency label knowledge can be utilized for training a model for lowresource language. Subsequent work considered syntactic similarities between languages for better feature transfer (Täckström et al., 2012; Naseem et al., 2012; Zhang and Barzilay, 2015).
Recently, a line of work leverages neural network models for multi-lingual parsing (Guo et al., 2015; Duong et al., 2015; Ammar et al., 2016). The basic idea is to map the word embedding spaces between different languages into the same vector space, by using sentence-aligned bilingual data. This gives consistency in tokens, POS and dependency labels thanks to the availability of Universal Dependencies (Nivre et al., 2016). Our work is similar to these methods in using a neural network model for knowledge sharing between different languages. However, ours is different in the use of a neural stacking model, which respects the distributional differences between Singlish and English words. This empirically gives higher accuracies for Singlish.
3 Singlish Dependency Treebank
3.1 Universal Dependencies for Singlish
Since English is the major genesis of Singlish, we choose English as the source of lexical feature transfer to assist Singlish dependency parsing. Universal Dependencies provides a set of multilingual treebanks with cross-lingually consistent dependency-based lexicalist annotations, designed to aid development and evaluation for cross-lingual systems, such as multilingual parsers (Nivre et al., 2016). The current version of Universal Dependencies comprises not only major treebanks for 47 languages but also their siblings for domain-specific corpora and di-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
alects. With the aligned initiatives for creating transfer-learning-friendly treebanks, we adopt the Universal Dependencies protocol for constructing the Singlish dependency treebank, both as a new resource for the low-resource languages and to facilitate knowledge transfer from English.
On top of the general Universal Dependencies guidelines, English-specific dependency relation definitions including additional subtypes are employed as the default standards for annotating the Singlish dependency treebank, unless augmented or redefined when necessary. The latest English corpus in Universal Dependencies v1.42 collection is constructed from the English Web Treebank (Bies et al., 2012), comprising of web media texts, which potentially smooths the knowledge transfer to our target Singlish texts in similar domains. The statistics of this dataset, from which we obtain English syntactic knowledge, is shown in Table 1 and we refer to this corpus as UD-Eng. This corpus uses 47 dependency relations and we show below how to conform to the same standard while adapting to unique Singlish grammars.
3.2 Challenges and Solutions for Annotating Singlish
The deviations of Singlish from English come from both the lexical and the grammatical levels (Leimgruber, 2009, 2011), which bring challenges for analysis on Singlish using English NLP tools. The former involves imported vocabularies from the first languages of the local people and the latter can be represented by a set of relatively localized features which collectively form 5 unique grammars of Singlish according to Leimgruber (2011). We find empirically that all these deviations can be accommodated by applying the existing English dependency relation definitions, which are explained with examples as follows.
Imported vocabulary: Singlish borrows a number of words and expressions from its nonEnglish origins (Leimgruber, 2009, 2011), such as “Kiasu”, which originates from Hokkien meaning “very anxious not to miss an opportunity”.3 These imported terms often constitute out-of-vocabulary (OOV) words with respect to a standard English treebank and result in difficulties for using English-trained tools on Singlish. All borrowed words are annotated according to their original
2Only guidelines for Universal Dependencies v2 but not the English corpus is available when this work is completed.
3Definition by the Oxford living Dictionaries for English.
(1) Drive this car sure draw looks .
root
det dobj
csubj
advmod dobj punct
(2) SG where got attap chu ?
root nsubj
advmod dobj compound
punct
(3) Inside tent can not see leh !
rootnmod aux
neg discourse
punct
case
(4) U betting more downside from here ?
root
nsubj dobj amod case
nmod punct
(5) Hope can close 22 today .
root ccomp
aux dobj nmod:tmod
punct
(6) Best to makan all , tio boh ?
root
mark xcomp
dobj
punct
neg
discourse
punct
(7) I never get it free one !
root
advmod nsubj dobj xcomp
discourse punct
Figure 2: Unique Singlish grammars.
meanings using existing UD-Eng POS tags and dependency relations. Table A3 in Appendix A summarizes all borrowed terms in our treebank.
Topic-prominence: This type of sentences start with establishing its topic, which often serves as the default one that the rest of the sentence refers to, and they typically employ an object-subjectverb sentence structure (Leimgruber, 2009, 2011). In particular, three subtypes of topic-prominence are observed in the Singlish dependency treebank and their annotations are addressed as follows:
First, topics framed as clausal arguments at the beginning of the sentence are labeled as “csubj” (clausal subject), as shown by “Drive this car” of (1) in Figure 2. This type can be regarded as a variant of the English it-extraposition sentence structure with the extraposition moved to the front and the clause marker removed.
Second, noun phrases used to modify the predicate with the presence of a preposition is regarded as a “nsubj” (nominal subject). This is a common order of words used in Chinese and one example is the “SG” of (2) in Figure 2.
Third, prepositional phrases moved in front are
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
still treated as “nmod” (nominal modifier) of their intended heads, following the exact definition but as a Singlish-specific form of exemplification, as shown by the “Inside tent” of (3) in Figure 2.
Copula deletion: Imported from the corresponding Chinese sentence structure, this copula verb is often optional and deleted in Singlish, which is one of its diagnostic characteristics (Leimgruber, 2009, 2011). In UD-Eng standards, predicative “be” is the only verb used as a copula, which often depends on its complement to avoid copular head. Thus the deleted copula and its “cop” (copula) arc are simply ignored to preserve the intactness of the dependency tree, as shown by (4) in Figure 2. The only possibility of a copular root is when the copula has a clausal argument or adjunct. However, such copula is typically not deleted in either Singlish or Chinese, and we have not encountered such cases in our treebank. If they appear by any chance, feasible solutions can be either recovering the copula with a special symbol or promoting the head of the clausal argument or adjunct as the sentence root.
NP deletion: Noun-phrase (NP) deletion often results in null subjects or objects. It may be regarded as a branch of “Topic-prominence” but is a distinctive feature of Singlish with relatively high frequency of usage (Leimgruber, 2011). Again, we do not recover such relations since the deleted NP imposes negligible alteration to the dependency tree, as exemplified by (5) in Figure 2.
Inversion: Inversion in Singlish involves either reversed order of subject and verb in interrogative sentences, or tag questions in polar interrogatives (Leimgruber, 2011). The former simply involves a change of word orders and thus requires no special treatments. On the other hand, tag questions should be carefully analyzed in two scenarios. One type is in the form of “isn’t it ?” or “haven’t you ?”, which are dependents of the sentence root with the “parataxis” relation.4 The other type is exemplified as “right ?”, and its Singlish equivalent “tio boh ?” (a transliteration from Hokkien) are labeled with the “discourse” (discourse element) relation with respect to the sentence root. See example (6) in Figure 2.
Discourse particles: Usage of clausal-final discourse particles, which originates from Hokkien
4Relation between the main verb of a clause and other sentential elements, such as sentential parenthetical clause, or adjacent sentences without any explicit coordination or subordination.
and Cantonese, is one of the most typical feature of Singlish (Leimgruber, 2009, 2011; Lim, 2007). All discourse particles that appear in our treebank are summarized in Table A3 in Appendix A with the imported vocabulary:. These words express the tone of the sentence and thus have the “INTJ” (interjection) POS tag and depend on the root of the sentence or clause labeled with “discourse”, as is shown by the “leh” of (3) in Figure 2. The word “one” is a special instance of this type with the sole purpose being a tone marker in Singlish but not English, as shown by (7) in Figure 2.
3.3 Data Selection and Annotation
Data Source: Singlish is used in written form mainly in social media and local Internet forums. After comparison, we chose the SG Talk Forum5 as our data source due to its relative abundance in Singlish contents. We crawled 84,459 posts using the Scrapy framework6 from pages dated up to 25th December 2016, retaining sentences of length between 5 and 50, which total 58,310. Sentences are reversely sorted according to the log likelihood of the sentence given by an English language model trained using the KenLM toolkit (Heafield et al., 2013)7 normalized by the sentence length, so that those most different from standard English can be chosen. Among the top 10,000 sentences, 1,977 sentences contain unique Singlish vocabularies defined by The Coxford Singlish Dictionary8, A Dictionary of Singlish and Singapore English9, and the Singlish Vocabulary Wikipedia page10. The average normalized log likelihood of these 10,000 sentences is -5.81, and the same measure for all sentences in UD-Eng is -4.81. This means these sentences with Singlish contents are 10 times less probable expressed as standard English than the UD-Eng contents in the web domain. This contrast indicates the degree of lexical deviation of Singlish from English. We chose 1,200 sentences from the first 10,000.
Annotation: The chosen texts are divided by random selection into training, development, and testing sets according to the proportion of sentences in the training, development, and test di-
5http://sgTalk.com 6https://scrapy.org/ 7Trained using the afp eng and xin eng sources of English
Gigaword Fifth Edition (Gigaword). 8http://72.5.72.93/html/lexec.php 9http://www.singlishdictionary.com
10https://en.wikipedia.org/wiki/ Singlish_vocabulary
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
UD English Singlish Sentences Words Sentences Words
Train 12,543 204,586 900 8,221 Dev 2,002 25,148 150 1,384 Test 2,077 25,096 150 1,381
Table 1: Division of training, development, and test sets for Singlish Treebank
vision for UD-Eng, as summarized in Table 1. The sentences are tokenized using the NLTK Tokenizer,11 and then annotated using the Dependency Viewer.12 In total, all 17 UD-Eng POS tags and 41 out of the 47 UD-Eng dependency labels are present in the Singlish dependency treebank. Besides, 100 sentences are randomly selected and double annotated by one of the coauthors, and the inter-annotator agreement has an unlabeled attachment score (UAS) of 85.30% and a labeled attachment score (LAS) of 75.72%. A full summary of the number of occurences of each POS tag and dependency labels is included in Appendix A.
4 Part-of-Speech Tagging
In order to obtain automatically predicted POS tags as features for a base English dependency parser, we train a POS tagger for UD-Eng using the baseline model of Chen et al. (2016), depicted in Figure 3. The bi-LSTM networks with a CRF layer (bi-LSTM-CRF) have shown state-of-the-art performance by globally optimizing the tag sequence (Huang et al., 2015; Chen et al., 2016). Based on this English POS tagging model, we train a POS tagger for Singlish using the featurelevel neural stacking model of Chen et al. (2016). Both the English and Singlish models consist of an input layer, a feature layer, and an output layer.
4.1 Base Bi-LSTM-CRF POS Tagger
Input Layer: Each token is represented as a vector by concatenating a word embedding from a lookup table with a weighted average of its character embeddings given by the attention model of Bahdanau et al. (2014). Following Chen et al. (2016), the input layer produces a dense representation for the current input token by concatenating its word vector and the ones for its surrounding context tokens in a window of finite size.
11http://www.nltk.org/api/nltk. tokenize.html
12http://nlp.nju.edu.cn/tanggc/tools/ DependencyViewer.exe
x2 x1
…
h1
h1
h2
h2
xn
hn
hn
Tanh Tanh Tanh
Linear Linear Linear
CRF
…
…
…
…
…
t1 t2 tn …
Output layer
Feature layer
Input layer
Figure 3: Base POS tagger
Feature Layer: This layer employs a bi-LSTM network to encode the input into a sequence of hidden vectors that embody global contextual information. Following Chen et al. (2016), we adopt bi-LSTM with peephole connections (Graves and Schmidhuber, 2005).
Output layer: This is a CRF layer to predict the POS tags for the input words by maximizing the conditional probability of the sequence of tags given input sentence.
4.2 POS Tagger with Neural Stacking
We adopt the deep integration neural stacking structure presented in Chen et al. (2016). As shown in Figure 4, the distributed vector representation for the target word at the input layer of the Singlish Tagger is augmented by concatenating the emission vector produced by the English Tagger with the original word and character-based embeddings, before applying the concatenation within a context window in section 4.1. During training, loss is back-propagated to all trainable parameters in both the Singlish Tagger and the pre-trained feature layer of the base English Tagger. At test time, the input sentence is fed to the integrated tagger model as a whole for inference.
4.3 Results
We use the publicly available source code13 by Chen et al. (2016) to train a 1-layer biLSTM-CRF based POS tagger on UD-Eng, using 50-dimension pre-trained SENNA word embeddings (Collobert et al., 2011). We set the hidden layer size to 300, the initial learning rate for Adagrad (Duchi et al., 2011) to 0.01, the regularization parameter λ to 10−6, and the dropout rate to 15%. The tagger gives 94.84% accuracy on the UD-Eng
13https://github.com/chenhongshen/ NNHetSeq
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
…
h1
h1
h2
h2
hn
hn
Tanh Tanh Tanh
Singlish Tagger output layer
…
…
…
English Tagger feature layer
…
x2 Linear x1 Linear
xn Linear
x1 x2 xn
Output
layer
Feature
layer
Input layer
Base English Tagger
Figure 4: POS tagger with neural stacking
System Accuracy ENG-on-SIN 80.67% Base-ICE-SIN 77.77% Stack-ICE-SIN 84.79%
Table 2: POS tagging accuracies
test set after 24 epochs, chosen according to development tests, which is comparable to the stateof-the-art accuracy of 95.17% reported by Plank et al. (2016). We use these settings to perform 10- fold jackknifing of POS tagging on the UD-Eng training set, with an average accuracy of 95.60%.
Similarly, we trained a POS tagger using the Singlish dependency treebank alone with pretrained word embeddings on The Singapore Component of the International Corpus of English (ICE-SIN) (Nihilani, 1992; Ooi, 1997), which consists of both spoken and written texts. However, due to limited amount of training data, the tagging accuracy is not satisfactory even with a large dropout rate to avoid overfitting. In contrast, the neural stacking structure on top of the English base model trained on UD-Eng achieves a POS tagging accuracy of 84.79%14, which corresponds to a 31.58% relative error reduction over the baseline Singlish model, as shown in Table 2. We use this for 10-fold jackknifing on Singlish parsing training data, and tagging the Singlish development and test data.
14We empirically find that using ICE-SIN embeddings in neural stacking model performs better than using English SENNA embeddings. Similar findings are found for the parser, of which more details are given in section 6.
Output
layer
Input layer
Feature
layer
x2x1
…
xn
…
…
…
…
…
……
…
ℎ1 1
ℎ1 1
ℎ𝑚 1
ℎ𝑚 1
ℎ𝑚 2
ℎ𝑚 2
ℎ1 2
ℎ1 2
ℎ𝑚 𝑛
ℎ𝑚 𝑛
ℎ1 𝑛
ℎ1 𝑛
MLPd MLPh MLPd MLPh MLPd MLPh
1
…
1
1
… … …
=
…
Hd + HhU +1 w S
Figure 5: Base parser
5 Dependency Parsing
We adopt the Dozat and Manning (2016) parser15 as our base model, as displayed in Figure 5, and apply neural stacking to achieve improvements over the baseline parser. Both the base and neural stacking models consist of an input layer, a feature layer, and an output layer.
5.1 Base Parser with Bi-affine Attentions
Input Layer: This layer encodes the current input word by concatenating a pre-trained word embedding with a trainable word embedding and POS tag embedding from the respective lookup tables.
Feature Layer: The two recurrent vectors produced by the multi-layer bi-LSTM network from each input vector are concatenated and mapped to multiple feature vectors in lower-dimension space by a set of parallel multilayer perceptron (MLP) layers. Following Dozat and Manning (2016), we adopt Cif-LSTM cells (Greff et al., 2016).
Output Layer: This layer applies biaffine transformation on the feature vectors to calculate the score of the directed arcs between every pair of words. The inferred trees for input sentence are formed by choosing the head with the highest score for each word and a cross-entropy loss is calculated to update the model parameters.
5.2 Parser with Neural Stacking
Inspired by the idea of feature-level neural stacking (Chen et al., 2016; Zhang and Weiss, 2016), we concatenate the pre-trained word embedding,
15https://github.com/tdozat/Parser
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
654
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
…
…
…
…
…
…
…
ℎ𝑚 𝑖
ℎ𝑚 𝑖
ℎ1 𝑖
ℎ1 𝑖
ℎ𝑚 𝑗
ℎ𝑚 𝑗
ℎ1 𝑗
ℎ1 𝑗
MLPd MLPh MLPd MLPh
English Parser Bi-LSTM
xi
ℎ𝑚 𝑖
ℎ𝑚 𝑖
xi
…
MLPd MLPh MLPd MLPh
…
…
…
…
…
…
…
…
Singlish Parser output layer
+ +… …
…… …
xj
ℎ𝑚 𝑗
ℎ𝑚 𝑗
+ +
……
xj
…… …
Output
layer
Input layer
Feature
layer
Base English Parser
Figure 6: Parser with neural stacking
trainable word and tag embeddings, with the two recurrent state vectors at the last bi-LSTM layer of the English Tagger as the input vector for each target word. In order to further preserve syntactic knowledge retained by the English Tagger, the feature vectors from its MLP layer is added to the ones produced by the Singlish Parser, as illustrated in Figure 6, and the scoring tensor of the Singlish Parser is initialized with the one from the trained English Tagger. Loss is back-propagated by reversely traversing all forward paths to all trainable parameter for training and the whole model is used collectively for inference.
6 Experiments and Discussion
6.1 Experimental Settings
We train an English parser on UD-Eng with the default model settings in Dozat and Manning (2016). It achieves a UAS of 88.88% and a LAS of 85.00%, which are close to the state-of-the-art 85.90% LAS on UD-Eng reported by Ammar et al. (2016), and the main difference is caused by us not using fine-grained POS tags. We apply the same settings for a baseline Singlish parser. We attempt to choose a better configuration of the number of bi-LSTM layers and the hidden dimension based on the development set performance, but the default settings turn out to perform the best. Thus we stick to all default hyperparameters in Dozat and Manning (2016) for training the Singlish parsers.
We experimented with different word embeddings, as with the raw text sources summarized in Table 3 and further described in 6.2. When using
the neural stacking model, we fix the model configuration for the base English parser model and choose the size of the hidden vector and the number of bi-LSTM layers stacked on top based on the development set. It turns out that a 1-layer biLSTM with 750 hidden dimension performs the best, where the bigger hidden layer accommodates the elongated input vector to the stacked bi-LSTM and the fewer number of recurrent layers avoids overfitting on the small Singlish dependency treebank, given the deep bi-LSTM English parser network at the bottom.
6.2 Investigating Distributed Lexical Characteristics
In order to learn characteristics of distributed lexical semantics for Singlish, we compare performances of the Singlish dependency parser using several sets of pre-rained word embeddings: GloVe6B, large-scale English word embeddings16; ICE-SIN, Singlish word embeddings trained using GloVe (Pennington et al., 2014) on the ICE-SIN (Nihilani, 1992; Ooi, 1997) corpus; Giga100M, a small-scale English word embeddings trained using GloVe (Pennington et al., 2014) with the same settings on a comparable size of English data randomly selected from the English Gigaword Fifth Edition for a fair comparison with ICE-SIN embeddings.
First, the English Giga100M embeddings marginally improve the Singlish parser from the baseline without pre-trained embeddings, giving a performance on par with using UD-Eng parser directly on Singlish, represented as “ENG-on-SIN”
16Trained with Wikipedia 2014 the Gigaword. Downloadable from http://nlp.stanford.edu/data/ glove.6B.zip
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Topic Prominence Copula Deletion NP Deletion Discourse Particles Others Sentences 15 19 21 51 67
UAS LAS UAS LAS UAS LAS UAS LAS UAS LAS ENG-on-SIN 73.15 56.48 61.87 51.08 70.29 60.57 70.75 60.00 81.90 71.83 Base-Giga100M 77.78 62.96 79.86 66.91 76.00 65.14 83.75 74.75 70.34 58.40 Base-ICE 74.07 64.81 71.94 61.87 79.43 70.86 87.25 77.50 77.61 63.99 Stack-ICE 79.63 69.44 79.86 71.22 78.86 71.43 88.00 82.75 85.07 76.87
Table 5: Error analysis with respect to grammar types
in Table 4. With much more English lexical semantics being fed to the Singlish parser using the English GloVe6B embeddings, the enhancement is still marginal. On the contrary, the Singlish ICESIN embeddings lead to more improvement, with 18.09% relative error reduction compared with 5.11% using the English Giga100M embeddings. It is even better than using the English GloVe6B embeddings despite the huge difference in sizes. This demonstrate the distributional differences between Singlish and English tokens, even though they share a large vocabulary. More detailed comparison is described in section 6.4.
6.3 Knowledge Transfer Using Neural Stacking
We train a parser with neural stacking and Singlish ICE-SIN embeddings, which achieves the best performance among all the models, with a UAS of 84.21% as shown in Table 4, which corresponds to 36.51% relative error reduction compared to the baseline. This demonstrates that knowledge from English can be successfully incorporated to boost the Singlish parser. This significant improvement is further explained below.
6.4 Improvements over Grammar Types
To analyze the sources of improvements for Singlish parsing using different model configurations, we conduct error analysis over 5 syntactic categories17, including 4 types of grammars mentioned in section 3.218, and 1 for all other cases, including sentences containing imported vocabularies but expressed in basic English syntax. The number of sentences and the results in each group of the test set are shown in Table 5.
The neural stacking model leads to the biggest improvement over nearly all categories except for a slightly lower yet competitive performance on “NP Deletion” cases, which explains the significant overall improvement.
17Multiple labels are allowed for one sentence. 18The “Inversion” type of grammar is not analyzed since
there is only 1 such sentence in the test set.
Comparing the base model with ICE-SIN embeddings with the base parser trained on UD-Eng, which contain syntactic and semantic knowledge in Singlish and English, respectively, the former outperforms the latter on all 4 types of Singlish grammars but not for the remaining samples. This suggests that the base English parser mainly contributes to analyzing basic English syntax, while the Singlish dependency treebank models unique Singlish grammars better.
Using the English Giga100M embeddings in the base model only helps to improve parsing for “Topic Prominence” and “Copula Deletion” compared to the Singlish ICE-SIN embeddings. The main reason can be that these two types of grammars preserve a high percentage of English syntax, with only alterations in word orders and deletions of one word. However, “NP Deletion” results in more radical sentence incompleteness and “Discourse Particles” are unique in Singlish. An interesting finding is that feeding English distributed lexical semantic information to the base Singlish parser undermines the performance even over basic English syntax, which again suggests the differences in distributed lexical semantics.
7 Conclusion
We have investigated dependency parsing for Singlish, an important English-based creole language, through annotations of a Singlish dependency treebank with 10,986 words, and building an enhanced parser by leveraging on knowledge transferred from a 20-times-bigger English treebank of Universal Dependencies. We demonstrate the effectiveness of using neural stacking for feature transfer by boosting the Singlish dependency parsing performance to 84.21% UAS, with a 36.51% error reduction. We release the annotated Singlish dependency treebank, the trained model and the source code for the parser with free public access. Possible future work include expanding the investigation to other regional languages such as Malay and Indonesian.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Statistics of Singlish Dependency Treebank
POS Tags ADJ 767 INTJ 510 PUNCT 623 ADP 487 NOUN 1717 SCONJ 126 ADV 912 NUM 147 SYM 11 AUX 429 PART 354 VERB 1678 CONJ 167 PRON 675 X 10 DET 386 PROPN 787
Table A1: Statistics of POS tags
Dependency labels acl 33 dobj 590 acl:relcl 29 expl 10 advcl 186 iobj 15 advmod 829 list 10 appos 18 mwe 100 amod 422 name 117 aux 377 neg 260 auxpass 47 nmod 364 case 463 nmod:npmod 26 cc 167 nmod:poss 153 ccomp 135 nmod:tmod 77 compound 420 nsubj 1005 compound:prt 27 nsubjpass 34 conj 229 nummod 94 cop 152 mark 275 csubj 30 parataxis 223 det 304 punct 626 det:predet 7 remnant 16 discourse 507 vocative 38 dislocated 2 xcomp 187
Table A2: Statistics of dependency labels
A-B act blur ah ah beng ah ne angpow arrowed aiyah ang ku kueh angmoh/ang moh ahpek / ah peks atas ba boh/bo boho jiak boh pian buay lin chu buen kuey C chai tow kway chao ah beng chap chye png char kway teow chee cheong fun / che cheong fen cheesepie cheong / chiong chiam / cham chiak liao bee / jiao liao bee chio ching chong chio bu / chiobu chui chop chop chow-angmoh chwee kueh D-F dey diam diam die kock standing die pain pain dun eat grass flip prata fried beehoon G gahmen / garment gam geylang gone case gong kia goreng pisang gui H-J hah / har / huh hai si lang heng hiak hiak hiak hiong hoot hor Hosay / ho say how lian huat jepun kia / jepun kias jialat / jia lak / jia lat K ka kaki kong kaki song kancheong kateks kautim kay kiang kayu kee chia kee siao kelong kena / kana kiam kiasu ki seow kkj kong si mi kopi kopi lui kopi-o kosong koyok ku ku bird L lagi lai liao laksa la / lah lao jio kong lao sai lau chwee nua lau leh liao / ler like dat / like that lim peh lobang loh / lor M mahjong kaki makan ma / mah masak masak mati mee mee pok mee rebus mee siam mee sua mei mei N-S nasi lemak pang sai piak sabo sai same same sia sianz / sian sia suay sibeh siew dai siew siew dai simi taisee soon kuey sotong suay / suey swee T tahan tak pakai te te kee talk cock / talk cock sing song tikopeh tio tio pian/dio pian tong tua U-Z umm zai up lorry / up one’s lorry wahlow / wah lau walaneh / wah lan eh wa / wah xiao ya ya zhun / buay zhun
Table A3: Imported vocabulary and discourse particles in the Singlish dependency treebank

The authors construct a new dataset of 1200 Singaporean English (Singlish) sentences annotated with Universal Dependencies. They show that they can improve the performance of a POS tagger and a dependency parser on the Singlish corpus by integrating English syntactic knowledge via a neural stacking model.  - Strengths: Singlish is a low-resource language. The NLP community needs more data for low resource languages, and the dataset accompanying this paper is a useful contribution. There is also relatively little NLP research on creoles, and the potential of using transfer-learning to analyze creoles, and this paper makes a nice contribution in that area.  The experimental setup used by the authors is clear. They provide convincing evidence that incorporating knowledge from an English-trained parser into a Singlish parser outperforms both an English-only parser and a Singlish-only parser on the Singlish data. They also provide a good overview of the relevant differences between English and Singlish for the purposes of syntactic parser and a useful analysis of how different parsing models handle these Singlish-specific constructions.  - Weaknesses:  There are three main issues I see with this paper: *  There is insufficient comparison to the UD annotation of non-English languages. Many of the constructions they bring up as specific to Singlish are also present in other UD languages, and the annotations should ideally be consistent between Singlish and these languages. *  I'd like to see an analysis on the impact of training data size. A central claim of this paper is that using English data can improve performance on a low-resource language like Singlish. How much more Singlish data would be needed before the English data became unnecessary? *  What happens if you train a single POS dep parsing model on the concatenated UD Web and Singlish datasets? This is much simpler than incorporating neural stacking. The case for neural stacking is stronger if it can outperform this baseline.  - General Discussion: Line 073:  POS taggers and dependency parsers perform poorly on such Singlish texts based on our observations d - be more clear that you will quantify this later. As such, it seems a bit hand-wavy.  Line 169: Comparison to neural network models for multi-lingual parsing. As far as I can tell, you don't directly try the approach of mapping Singlish and English word embeddings into the same embedding space.  Line 212: Introduction of UD Eng. At this point, it is appropriate to point out that the Singlish data is also web data, so the domain matches UD Eng.  Line 245:  All borrowed words are annotated according to their original meanings d. Does this mean they have the same POS as in  the language from which they were borrowed? Or the POS of their usage in Singlish?  Figure 2: Standard English glosses would be very useful in understanding the constructions and checking the correctness of the UD relations used.  Line 280: Topic prominence: You should compare with the  dislocated d label in UD. From the UD paper:  The dislocated relation captures preposed (topics) and postposed elements d. The syntax you are describing sounds similar to a topic-comment-style syntax; if it is different, then you should make it clear how.  Line 294:  Second, noun phrases used to modify the predicate with the presence of a preposition is regarded as a  nsubj d (nominal subject). d Here, I need a gloss to determine if this analysis makes sense. If the phrase is really being used to modify the predicate, then this should not be nsubj. UD makes a distinction between core arguments (nsubj, dobj, etc) and modifiers. If this is a case of modification, then you should use one of the modification relations, not a core argument relation. Should clarify the language here.  Line 308:  In UD-Eng standards, predicative  be d is the only verb used as a copula, which often depends on its complement to avoid copular head. d This is an explicit decision made in UD, to increase parallelism with non-copular languages (e.g., Singlish). You should call this out. I think the rest of the discussion of copula handling is not necessary.  Line 322:  NP deletion: Noun-phrase (NP) deletion often results in null subjects or objects. d This is common in other languages (zero-anaphora in e.g. Spanish, Italian, Russian, Japanese u2026 )Would be good to point this out, and also point to how this is dealt with in UD in those languages (I believe the same way you handle it).  Ling 330: Subj verb inversion is common in interrogatives in other languages ( Fue Marta al supermercado Did Marta go to the supermarket? d). Tag questions are present in English (though perhaps are not as frequent). You should make sure that your analysis is consistent with these languages.  Sec 3.3 Data Selection and Annotation: The way you chose the Singlish sentences, of course an English parser will do poorly (they are chosen to be disimilar to sentences an English parser has seen before). But do you have a sense of how a standard English parser does overall on Singlish, if it is not filtered this way? How common are sentences with out-of-vocabulary terms or the constructions you discussed in 3.2?  A language will not necessarily capture unusual sentence structure, particularly around long-distance dependencies. Did you investigate whether this method did a good job of capturing sentences with the grammatical differences to English you discussed in Section 3.2?  Line 415:  the inter-annotator agreement has an unlabeled attachment score (UAS) of 85.30% and a labeled attachment score (LAS) of 75.72%. d *  What  the agreement on POS tags? Is this integrated with LAS? *  Note that in Silveira et al 2014, which produced UD-Eng, they measured 94% inter-annotator agreement on a per-token basis. Why the discrepancy?  POS tagging and dep parsing sections: For both POS-tagging and dep parsing, I d like to see some analysis on the effect of training set size. E.g., how much more Singlish data would be needed to train a POS tagger dep parser entirely on Singlish and get the same accuracy as the stacked model?  What happens if you just concatenate the datasets? E.g., train a model on a hybrid dataset of EN and Singlish, and see what the result is?  Line 681: typo:  pre-rained d should be  pre-trained d  742  The neural stacking model leads to the biggest improvement over nearly all categories except for a slightly lower yet competitive performance on  NP Deletion d cases d --- seems that the English data strongly biases the parser to expect an explicit subj obj. you could try deleting subj obj from some English sentences to improve performance on this construction.