We show that discourse structure, as defined by Rhetorical Structure Theory and provided by an existing discourse parser, benefits text categorization. Our approach uses a recursive neural network and a newly proposed attention mechanism to compute a representation of the text that focuses on salient content, from the perspective of both RST and the task. Experiments consider variants of the approach and illustrate its strengths and weaknesses.
1 000
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Advances in text categorization have the potential to improve systems for analyzing sentiment, inferring authorship or author attributes, making predictions, and many more. Several past researchers have noticed that methods that reason about the relative saliency or importance of passages within a text can lead to improvements (Ko et al., 2004). Latent variables (Yessenalina et al., 2010), structured-sparse regularizers (Yogatama and Smith, 2014), and neural attention models (Yang et al., 2016) have all been explored.
Discourse structure, which represents the organization of a text as a tree (for an example, see Figure 1), might provide cues for the importance of different parts of a text. Some promising results on sentiment classification tasks support this idea: Bhatia et al. (2015) and Hogenboom et al. (2015) applied hand-crafted weighting schemes to the sentences in a document, based on its discourse structure, and showed benefit to sentiment polarity classification.
In this paper, we investigate the value of discourse structure for text categorization more broadly, considering five tasks, through the use of a recursive neural network built on an
R
Contrast
Elaboration
A B
Explanation
C Joint
D Constrast
E F
[Although the food was amazing]A [and I was in love with the spicy pork burrito,]B [the service was really awful.]C [We watched our waiter serve himself many drinks.]D [He kept running into the bathroom]E [instead of grabbing our bill.]F
automatically-derived document parse from a topperforming, open-source discourse parser, DPLP (Ji and Eisenstein, 2014). Our models learn to weight the importance of a document’s sentences, based on their positions and relations in the discourse tree. We introduce a new, unnormalized attention mechanism to this end.
Experimental results show that variants of our model outperform prior work on four out of five tasks considered. Our method unsurprisingly underperforms on the fifth task, making predictions about legislative bills—a rather different genre in which discourse conventions are quite different from those in the discourse parser’s training data. Further experiments show the effect of discourse parse quality on text categorization performance, suggesting that future improvements to discourse parsing will pay off for text categorization, and validate our new attention mechanism.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Our implementation is available at http:// anonymous.link.
2 Background: Rhetorical Structure
Theory
Rhetorical Structure Theory (RST; Mann and Thompson, 1988) is a theory of discourse that has enjoyed popularity in NLP. RST posits that a document can be represented by a tree whose leaves are elementary discourse units (EDUs, typically clauses or sentences). Internal nodes in the tree correspond to spans of sentences that are connected via discourse relations such as CONTRAST and ELABORATION. In most cases, a discourse relation links adjacent spans denoted “nucleus” and “satellite,” with the former more essential to the writer’s purpose than the latter.1
An example of a manually constructed RST parse for a restaurant review is shown in Figure 1. The six EDUs are indexed from A to F ; the discourse tree organizes them hierarchically into increasingly larger spans, with the last CONTRAST relation resulting in a span that covers the whole review. Within each relation, the RST tree indicates the nucleus pointed by an arrow from its satellite (e.g., in the ELABORATION relation, A is the nucleus and B is the satellite).
The information embedded in RST trees has motivated many applications in NLP research, including document summarization (Marcu, 1999), argumentation mining (Azar, 1999), and sentiment analysis (Bhatia et al., 2015). In most applications, RST trees are built by automatic discourse parsing, due to the expensive cost of manual annotation. In this work, we use a state-of-the-art open-source RST-style discourse parser, DPLP (Ji and Eisenstein, 2014).2
We follow recent work that suggests transforming the RST tree into a dependency structure (Yoshida et al., 2014).3 Figure 2(a) shows the corresponding dependency structure of the RST tree in Figure 1. It is clear that C is the root of the tree, and in fact this clause summarizes the review and suffices to categorize it as negative. This dependency representation of the RST tree offers a
1There are also a few exceptions in which a relation can be realized with multiple nuclei.
2 https://github.com/jiyfeng/DPLP
3The transformation is trivial and deterministic given the nucleus-satellite mapping for each relation. The procedure is analogous to the transformation of a headed phrase-structure parse in syntax into a dependency tree (e.g., Yamada and Matsumoto, 2003).
form of inductive bias for our neural model, helping it to discern the most salient parts of a text in order to assign it a label.
3 Model
Our model is a recursive neural network built on a discourse dependency tree. It includes a distributed representation computed for each EDU, and a composition function that combines EDUs and partial trees into larger trees. At the top of the tree, the representation of the complete document is used to make a categorization decision. Our approach is analogous to (and inspired by) the use of recursive neural networks on syntactic dependency trees, with word embeddings at the leaves (Socher et al., 2014).
3.1 Representation of Sentences
Let e be the distributed representation of an EDU. We use a bidirectional LSTM on the words’ embeddings within each EDU (details of word embeddings are given in section 4), concatenating the last hidden state vector from the forward LSTM ( e ) with that of the backward LSTM ( !e ) to get e.
There is extensive recent work on architectures for embedding representations of sentences and other short pieces of text, including, for example, (bi)recursive neural networks (Paulus et al., 2014) and convolutional neural networks (Kalchbrenner et al., 2014). Future work might consider alternatives; we chose the bidirectional LSTM due to its effectiveness in many settings.
3.2 Full Recursive Model
Given the discourse dependency tree for an input text, our recursive model builds a vector representation through composition at each arc in the tree. Let v
i denote the vector representation of EDU i and its descendants. For the base case where EDU i is a leaf in the tree, we let v
i = tanh(e i ), which is the elementwise hyperbolic tangent function.
For an internal node i, the composition function considers a parent and all of its children, whose indices are denoted by children(i). In defining this composition function, we seek for (i.) the contribution of the parent node e
i to be central; and (ii.) the contribution of each child node e
j be determined by its content as well as the discourse relation it holds with the parent. We therefore define
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
C
DA E
B F
Elab.
Cont. Exp. Exp.
Cont.
(a) dependency structure
tanh(e C
+ P
j2{A,D,E} ↵C,jWC,jvj)
tanh(e D )tanh(e A + ↵ A,B W A,B v B ) tanh(e E + ↵ F,E W F,E v F )
tanh(e B ) tanh(e F )
W A,B
W C,A
W C,D
W C,E
W F,E
(b) recursive neural network structure
Figure 2: (a.) The discourse dependency tree derived from the example RST tree in Figure 1. (b.) The corresponding recursive neural network model built on the tree.
v i = tanh
0
@e i +
X
j2children(i)
↵
i,j W ri,jvj
1
A ,
(1) where W
ri,j is a relation-specific composition matrix indexed by the relation between i and j, r
i,j . ↵
i,j
is an “attention” weight, defined as
↵
i,j
= ⇣ e> i W ↵ v j ⌘ , (2)
where is the elementwise sigmoid and W ↵ contains attention parameters (these are relationindependent). Our attention mechanism differs from prior work (Bahdanau et al., 2015), in which attention weights are normalized to sum to one across competing candidates for attention. Here, ↵
i,j does not depend on node i’s other children. This is motivated by RST, in which the presence of a node does not signify lesser importance to its siblings. Consider, for example, EDU D and text span E-F in Figure 1, which in parallel provide EXPLANATION for EDU C. This scenario differs from machine translation, where attention isused to implicitly and softly align output-language words to relatively few input-language words. It also differs from attention in composition functions used in syntactic parsing (Kuncoro et al., 2017), where attention can mimic head rules that follow from an endocentricity hypothesis of syntactic phrase representation.
Our recursive composition function, through the attention mechanism and the relation-specific weight matrices, is designed to learn how to differently weight EDUs for the categorization task. This idea of using a weighting scheme along with discourse structure is explored in prior works (Bhatia et al., 2015; Hogenboom et al., 2015), although they are manually designed, rather than learned from training data.
Once we have vroot of a text, the prediction of its category is given by softmax (W
o vroot + b). We refer to this model as the FULL model, since it makes use of the entire discourse dependency tree.
3.3 Unlabeled Model
The FULL model based on Equation 1 uses a dependency discourse tree with relations. Because alternate discourse relation labels have been proposed (e.g., Prasad et al., 2008), we seek to measure the effect of these labels. We therefore consider an UNLABELED model based only on the tree structure, without the relations:
v i = tanh
0
@e i +
X
j2children(i)
↵
i,j v j
1
A . (3)
Here, only attention weights are used to compose the children nodes’ representations, significantly reducing the number of model parameters.
This UNLABELED model is similar to the depth weighting scheme introduced by Bhatia et al. (2015), which also uses an unlabeled discourse dependency tree, but our attention weights are computed by a function whose parameters are learned. This approach sits squarely between Bhatia et al. (2015) and the flat document structure used by Yang et al. (2016); the UNLABELED model still uses discourse to bias the model toward some content (that which is closer to the tree’s root).
3.4 Simpler Variants
We consider two additional baselines that are even simpler. The first, ROOT, uses the discourse dependency structure only to select the root EDU, which is used to represent the entire text: vroot = eroot. No composition function is needed. This model variant is motivated by work on document summarization (Yoshida et al., 2014), where the
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
most central EDU is used to represent the whole text.
The second variant, ADDITIVE, uses all the EDUs with a simple composition function, and does not depend on discourse structure at all: vroot = 1 N P N
i=1 ei, where N is the total number of EDUs. This serves as a baseline to test the benefits of discourse, controlling for other design decisions and implementation choices. Although sentence representations {e
i } are built in a different way from the work of Yang et al. (2016), this model is quite similar to their HN-AVE model on building document representations.
4 Implementation Details
The parameters of all components of our model (top-level classification, composition, and EDU representation) are learned end-to-end using standard methods. We implement our learning procedure with the DyNet package (Neubig et al., 2017).
Preprocessing. For all datasets, we use the same preprocessing steps, mostly following recent work from language modeling (e.g., Mikolov et al., 2010). We lowercased all the tokens and removed tokens that contain only punctuation symbols. We replaced numbers in the documents with a special number token. Low-frequency word types were replaced by UNK; we reduce the vocabulary for each dataset until approximately 5% of tokens are mapped to UNK. The vocabulary sizes after preprocessing are also shown in Table 1.
Discourse parsing. Our model requires the discourse structure for each document. We used DPLP, the RST parser from Ji and Eisenstein (2014), which is one of the best discourse parsers on the RST discourse treebank benchmark (Carlson et al., 2001). It employs a greedy decoding algorithm for parsing, producing 2,000 parses per minute on average on a single CPU. DPLP provides discourse segmentation, breaking a text into EDUs, typically clauses or sentences, based on syntactic parses provided by Stanford CoreNLP. RST trees are converted to dependencies following the method of Yoshida et al. (2014). DPLP as distributed is trained on 347 Wall Street Journal articles from the Penn Treebank (Marcus et al., 1993).
Word embeddings. In cases where there are 10,000 or fewer training examples, we used
pretrained GloVe word embeddings (Pennington et al., 2014), following previous work on neural discourse processing (Ji and Eisenstein, 2015). For larger datasets, we randomly initialize word embeddings and train them alongside other model parameters.
Learning and hyperparameters. Online learning was performed with the optimization method and initial learning rate as hyperparameters. To avoid the exploding gradient problem, we used the norm clipping trick with a threshold of ⌧ = 5.0. In addition, dropout rate 0.3 was used on both input and hidden layers to avoid overfitting. We performed grid search over the word vector representation dimensionality, the LSTM hidden state dimensionality (both {32, 48, 64, 128, 256}), the initial learning rate ({0.1, 0.01, 0.001}), and the update method (SGD and Adam, Kingma and Ba, 2015). For each corpus, the highest-accuracy combination of these hyperparameters is selected using development data or ten-fold cross validation, which will be specified in section 5.
5 Datasets
We selected five datasets of different sizes and corresponding to varying categorization tasks. Some information about these datasets are summarized in Table 1.
Sentiment analysis on Yelp reviews. Originally from the Yelp Dataset Challenge in 2015, this dataset contains 1.5 million examples. We used the preprocessed dataset from Zhang et al. (2015), which has 650,000 training and 50,000 test examples. The task is to predict an ordinal rating (1–5) from the text of the review. To select the best combination of hyperparameters, we randomly sampled 10% training examples as the development data. We compared with hierarchical attention networks (Yang et al., 2016), which use the normalized attention mechanism on both word and sentence layers with a flat document structure, and provide the state-of-the-art result on this corpus.
Framing dimensions in news articles. The Media Frames Corpus (MFC; Card et al., 2015) includes around 4,200 news articles about immigration from 13 U.S. newspapers over the years 1980–2012. The annotations of these articles are in terms of a set of 15 general-purpose labels, such as ECONOMICS and MORALITY, designed to categorize the emphasis framing applied to the
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Number of docs.
Dataset Task Classes Total Training Development Test Vocab. size
Yelp Sentiment 5 700K 650K – 50K 10K MFC Frames 15 4.2K – – – 7.5K Debates Vote 2 1.6K 1,135 105 403 5K Movies Sentiment 2 2.0K – – – 5K Bills Survival 2 52K 46K – 6K 10K
Table 1: Information about the five datasets used in our experiments. To compare with prior work, we use different experimental settings. For Yelp and Bill corpora, we use 10% of the training examples as development data. For MFC and Movies corpora, we use 10-fold cross validation and report averages across all folds.
immigration issue within the articles. We focused on predicting the single primary frame of each article. The state-of-the-art result on this corpus is from Card et al. (2016), where they used logistic regression together with unigrams, bigrams and Bamman-style personas (Bamman et al., 2014) as features. The best feature combination in their model alongside other hyperparameters was identified by a Bayesian optimization method (Bergstra et al., 2015). To select hyperparameters, we used a small set of examples from the corpus as a development set. Then, we report average accuracy across 10-fold cross validation as in (Card et al., 2016).
Congressional floor debates. The corpus was originally collected by Thomas et al. (2006), and the data split we used was constructed by Yessenalina et al. (2010). The goal is to predict the vote (“yea” or “nay”) for the speaker of each speech segment. The most recent work on this corpus is from Yogatama and Smith (2014), which proposed structured regularization methods based on linguistic components, e.g., sentences, topics, and syntactic parses. Each regularization method induces a linguistic bias to improve text classification accuracy, where the best result we repeated here is from the model with sentence regularizers.
Movie reviews. This classic movie review corpus was constructed by Pang and Lee (2004) and includes 1,000 positive and 1,000 negative reviews. On this corpus, we used the standard tenfold data split for cross validation and reported the average accuracy across folds. We compared with the work from both Bhatia et al. (2015) and Hogenboom et al. (2015), which are two recent works on discourse for sentiment analysis. Bha-
tia et al. (2015) used a hand-crafted weighting scheme to bias the bag-of-word representations on sentences. Hogenboom et al. (2015) also considered manually-designed weighting schemes and a lexicon-based model as classifier, achieving performance inferior to fully-supervised methods like Bhatia et al. (2015) and ours.
Congressional bill corpus. This corpus, collected by Yano et al. (2012), includes 51,762 legislative bills from the 103rd to 111th U.S. Congresses. The task is to predict whether a bill will survive based on its content. We randomly sampled 10% training examples as development data to search for the best hyperparameters. To our knowledge, the best published results are due to Yogatama and Smith (2014), which is the same baseline as for the congressional floor debates corpus.
6 Experiments
We evaluated all variants of our model on the five datasets presented in section 5, comparing in each case to the published state of the art as well as the most relevant works.
Results. See Table 2. On four out of five datasets, our UNLABELED model (line 8) outperforms past methods. In the case of the very large Yelp dataset, our FULL model (line 9) gives even stronger performance, but not elsewhere, suggesting that it is overparameterized for the smaller datasets. Indeed, on the MFC and Movies tasks, the discourse-ignorant ADDITIVE outperforms the FULL model. On these datasets, the selected FULL model had nearly 20 times as many parameters as the UNLABELED model, which in turn had twice as many parameters as the ADDITIVE.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Method Yelp MFC Debates Movies Bills
Prior work 1. Yang et al. (2016) 71.0 — — — — 2. Card et al. (2016) — 56.8 — — — 3. Yogatama and Smith (2014) — — 74.0 — 88.5 4. Bhatia et al. (2015) — — — 82.9 — 5. Hogenboom et al. (2015) — — — 71.9 — Variants of our model 6. ADDITIVE 68.5 57.6 69.0 82.7 80.1 7. ROOT 54.3 51.2 60.3 68.7 70.5 8. UNLABELED 71.3 58.4 75.7 83.1 78.4 9. FULL 71.8 56.3 74.2 79.5 77.0
Table 2: Test-set accuracy across five datasets. Results from prior work are reprinted from the corresponding publications. Boldface marks performance stronger than the previous state of the art.
This finding demonstrates the benefit of explicit discourse structure—even the output from an imperfect parser—for text categorization in some genres. Even though the discourse parser is trained on news text, it still offers benefit to restaurant and movie reviews and to the genre of congressional debates. Even for news text, if the training dataset is small (e.g., MFC), a lighter-weight variant of discourse (UNLABELED) is preferred.
Legislative bills, which have technical legal content and highly specialized conventions (see Appendix A in the supplementary material for an example), are arguably the most distant genre from news among those we considered. On that task, we see discourse working against accuracy. Note that the corpus of bills is more than ten times larger than three cases where our UNLABELED model outperformed past methods, suggesting that the drop in performance is not due to lack of data.
It is also important to notice that the ROOT model performs quite poorly in all cases. This implies that discourse structure is not simply helping by finding a single EDU upon which to make the categorization decision.
Qualitative analysis. Figure 3 shows some example texts from the Yelp Review corpus with their discourse structures produced by DPLP, where the weights were generated with the FULL model. Figure 3(a) and 3(b) are two successful examples of the FULL model. Figure 3(a) shows a simple case with respect to the discourse structure. Figure 3(b) is slightly different—the text in this example may have more than one reasonable discourse structure, e.g., 2D could be a child of
2C instead of 2A. In both cases, discourse structures help the FULL model bias to the important sentences.
Figure 3(c), on the other hand, presents a negative example, where DPLP failed to identify the most salient sentence 3F . In addition, the weights produced by the FULL model do not make much sense, which we suspect the model was confused by the structure. Figure 3(c) also presents a manually-constructed discourse structure on the same text for reference.
Effect of parsing performance. A natural question is whether further improvements to RST discourse parsing would lead to even greater gains in text categorization. While advances in discourse parsing are beyond the scope of this paper, we can gain some insight by exploring degradation to the DPLP parser. An easy way to do this is to train it on subsets of the RST discourse treebank. We repeated the conditions described above for our FULL model, training DPLP on 25%, 50%, and 75% of the training set (randomly selected in each case) before re-parsing the data for the sentiment analysis task. We did not repeat the hyperparameter search. In Figure 4, we plot accuracy of the classifier (y-axis) against the F1 performance of the discourse parser (x-axis). Unsurprisingly, lower parsing performance implies lower classification accuracy. Notably, if the RST discourse treebank were reduced to 25% of its size, our method would underperform the discourseignorant model of Yang et al. (2016). While we cannot extrapolate with certainty, these findings suggest that further improvements to discourse
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
From DPLP:
1A 1B 1C
0.66
Elaboration
0.67
Cause
[This store is somewhat convenient but I can never find
any workers,]1A [it drives me crazy.]1B [I never come
here on the weekends or around holidays anymore.]1C
(a) true label: 2, predicted label: 2
From DPLP:
2A2B2C 2D 0.87
Evaluation
0.61
Elaboration
0.70
Evaluation
[I love these people.] 2A [They are very friendly and always ask about my
life.] 2B [They remember things I tell them then ask about it the next time
I’m in.] 2C [Patrick and Lily are the best but everyone there is wonderful in
their own ways.]
2D
(b) true label: 5, predicted label: 5
From DPLP:
3B
3A 3C 3D 3E 3F
0.47
Elaboration
0.32
Elaboration
0.62
Elaboration
0.16
Elaboration
0.32
Attribution
Manually constructed:
3F
3B3A
3C 3D
3E
Cause
Background
Explanation Explanation
Explanation
[We use to visit this pub 10 years ago because they had a nice english waitress and excellent fish and chips for the price.]3A [However we went back a few weeks ago and were disappointed.]3B [The price of the fish and chip dinner went up and they cut the portion in half.]3C [No one assisted us in putting two tables together we had to do it ourselves.]3D [Two guests wanted a good English hot tea and they didn’t brew it in advance.]3E [So we’ve decided there are newer and better places to eat fish and chips especially up in north phoenix.]3F
(c) true label: 1, predicted label: 3
Figure 3: Some example texts (with light revision for readability) from the Yelp Review corpus and their corresponding dependency discourse parses. The numbers on dependency edges are attention weights produced by the FULL model.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Figure 4: Varying the amount of training data for the discourse parser, we can see how parsing F1 performance affects accuracy on the Yelp review task.
parsing, through larger annotated datasets or improved models, could lead to greater gains.
Attention mechanism. In section 3, we contrasted our new attention mechanism (Equation 2), which is inspired by RST’s lack of “competition” for salience among satellites, with the attention mechanism used in machine translation (Bahdanau et al., 2015). We consider here a variant of our model with normalized attention:
↵0 i = softmax
0
BB@
2
664 ... v> j
...
3
775
j2children(i)
W ↵ · e i
1
CCA .
(4) The result here is a vector↵0
i , with one element for each child node j 2 children(i), and which sums to one.
This variant of the FULL model achieves 70.3% accuracy, giving empirical support to our theoretically-motivated design decision not to normalize attention. Of course, further architecture improvements may yet be possible.
7 Related Work
Early work on text categorization often treated text as a bag of words (e.g., Joachims, 1998; Yang and Pedersen, 1997). Representation learning, for example through matrix decomposition (Deerwester et al., 1990) or latent topic variables (Ramage et al., 2009), has been considered to avoid overfitting in the face of sparse data.
The assumption that all parts of a text should influence categorization equally persists even as
more powerful representation learners are considered. Zhang et al. (2015) treat a text as a sequence of characters, proposing to a deep convolutional neural network to build text representation. Xiao and Cho (2016) extended that architecture by inserting a recurrent neural network layer between the convolutional layer and the classification layer.
In contrast, our contributions follow Ko et al. (2004), who sought to weight the influence of different parts of an input text on the task. Two works that sought to learn the importance of sentences in a document are Yessenalina et al. (2010) and Yang et al. (2016). The former used a latent variable for the informativeness of each sentence, and the latter used a neural network to learn an attention function. Neither used any linguistic bias, relying only on task supervision to discover the latent variable distribution or attention function. Our work builds the neural network directly on a discourse dependency tree, favoring the most central EDUs over the others but giving the model the ability to overcome this bias.
Another way to use linguistic information was presented by Yogatama and Smith (2014), who used a bag-of-words model. The novelty in their approach was a data-driven regularization method that encouraged the model to collectively ignore groups of features found to coocur. Most related to our work is their “sentence regularizer,” which encouraged the model to try to ignore training-set sentences that were not informative for the task. Discourse structure was not considered.
Discourse for sentiment analysis. Recently, discourse structure has been considered for sentiment analysis, which can be cast as a text categorization problem. Bhatia et al. (2015) proposed two discourse-motivated models for sentiment polarity prediction. One of the models is also based on discourse dependency trees, but using a handcrafted weighting scheme. Our method’s attention mechanism automates the weighting.
8 Conclusion
We conclude that automatically-derived discourse structure is often helpful to categorization of many kinds of text, and the benefit increases with the accuracy of discourse parsing. This effect reverses for legislative bills, a text genre whose discourse structure diverges from that of news. These findings motivate further improvements to discourse parsing, especially for new genres.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
- Strengths:  The main strength of this paper is the incorporation of discourse structure in the DNN's attention model, which allows the model to learn the weights given to different EDUs.  Also the paper is very clear, and provides a good explanation of both RST and how it is used in the model. Finally, the evaluation experiments are conducted thoroughly with strong, state-of-the-art baselines.  - Weaknesses:  The main weakness of the paper is that the results do not strongly support the main claim that discourse structure can help text classification. Even the UNLABELED variant, which performs best and does outperform the state of the art, only provides minimal gains (and hurts in the legal bills domain). The approach (particularly the FULL variant) seems to be too data greedy but no real solution is provided to address this beyond the simpler UNLABELED and ROOT variants.  - General Discussion:  In general, this paper feels like a good first shot at incorporating discourse structure into DNN-based classification, but does not fully convince that RST-style structure will significantly boost performance on most tasks (given that it is also very costly to build a RST parser for a new domain, as would be needed in the legal bill domains described in this paper). I wish the authors had explored or at least mentioned next steps in making this approach work, in particular in the face of data sparsity. For example, how about defining (task-independent) discourse embeddings? Would it be possible to use a DNN for discourse parsing that could be incorporated in the main task DNN and optimized jointly  end-to-end? Again, this is good work, I just wish the authors had pushed it a little further given the mixed results.