Implicit relation classification on Penn Discourse TreeBank (PDTB) 2.0 is a common benchmark task for evaluating the understanding of discourse relations. However, the lack of consistency in preprocessing and evaluation poses challenges to fair comparison of results in the literature. In this work, we highlight these inconsistencies and propose an improved evaluation protocol. Paired with this protocol, we report strong baseline results from pretrained sentence encoders, which set the new state-of-the-art for PDTB 2.0. Furthermore, this work is the first to explore fine-grained relation classification on PDTB 3.0. We expect our work to serve as a point of comparison for future work, and also as an initiative to discuss models of larger context and possible data augmentations for downstream transferability. Understanding discourse relations in natural language text is crucial to end tasks involving larger context, such as question-answering (Jansen et al., 2014) and conversational systems grounded on documents (Saeidi et al., 2018; Feng et al., 2020). One way to characterize discourse is through relations between two spans or arguments (ARG 1/A RG 2) as in the Penn Discourse TreeBank (PDTB) (Prasad et al., 2008, 2019). For instance: [ Arg 1 I live in this world ,] [ Arg 2 assuming that there is no morality, God or police. ] (wsj_0790) Label: EXPANSION .M ANNER .A RG 2AS-MANNER The literature has focused on implicit discourse relations from PDTB 2.0 (Pitler et al., 2009; Lin et al., 2009), on which deep learning has yielded substantial performance gains (Chen et al., 2016; Liu and Li, 2016; Lan et al., 2017; Qin et al., 2017; Bai and  Work done while at IBM Research. Zhao, 2018; Nguyen et al., 2019, i.a. ). However, inconsistencies in preprocessing and evaluation such as different label sets (Rutherford et al., 2017) pose challenges to fair comparison of results and to analyzing the impact of new models. In this paper, we revisit prior work to explicate the inconsistencies and propose an improved evaluation protocol to promote experimental rigor in future work. Paired with this guideline, we present a set of strong baselines from pretrained sentence encoders on both PDTB 2.0 and 3.0 that set the state-of-the-art. We furthermore reflect on the results and discuss future directions. We summarize our contributions as follows:  We highlight preprocessing and evaluation inconsistencies in works using PDTB 2.0 for implicit discourse relation classification. We expect our work to serve as a comprehensive guide to common practices in the literature. We lay out an improved evaluation protocol using section-based cross-validation that preserves document-level structure. We report state-of-the-art results on both top-level and second-level implicit discourse relation classification on PDTB 2.0, and the first set of results on PDTB 3.0. We expect these results to serve as simple but strong baselines that motivate future work. We discuss promising next steps in light of the strength of pretrained encoders, the shift to PDTB 3.0, and better context modeling. In PDTB, two text spans in a discourse relation are labeled with either one or two senses from a three-level sense hierarchy. PDTB 2.0 contains around 43K annotations with 18.4K explicit and 16K implicit relations in over 2K Wall Street Journal (WSJ) articles. Identifying implicit relations (i.e., without explicit discourse markers such as Model Ji Lin P&K X-Accuracy Majority class 26.18 26.11 28.54 26.42 Adversarial Net (Qin et al., 2017) 46.23 44.65 -Seq2Seq+MemNet (Shi and Demberg, 2019) 47.83 45.82 -41.29  ELMo (Bai and Zhao, 2018) 48.22 45.73 -ELMo, Memory augmented (Bai et al., 2019) 49.15 46.08 -Multitask learning (Nguyen et al., 2019) 49.95 46.48 -BERT+MNLI (Nie et al., 2019) -53.7 -BERT+DisSent Books 5 (Nie et al., 2019) -54.7 BERT (base, uncased) 52.13 (  0 . 50 ) 51.41 (  1 . 02 ) 52.00 (  1 . 02 ) 49.68 (  0 . 35 ) BERT (large, uncased) 57.34  (  0 . 79 ) 55.07  (  1 . 01 ) 55.61 (  1 . 32 ) 53.37 (  0 . 22 ) XLNet (base, cased) 54.73 (  1 . 26 ) 55.82  (  0 . 79 ) 54.71 (  0 . 45 ) 52.98 (  0 . 29 ) XLNet (large, cased) 61.29  (  1 . 49 ) 58.77  (  0 . 99 ) 59.90  (  0 . 96 ) 57.74 (  0 . 90 ) Table 1: Accuracy on PDTB 2.0 L2 classification. but ) is more challenging than explicitly signaled relations (Pitler et al., 2008). The new version of the dataset, PDTB 3.0 (Prasad et al., 2019), introduces a new annotation scheme with a revised sense hierarchy as well as 13K additional datapoints. 2 The third-level in the sense hierarchy is modified to only contain asymmetric (or directional) senses. We survey the literature to identify several sources of variation in preprocessing and evaluation that could lead to inconsistencies in the results reported. Choice of label sets. Due to the hierarchical annotation scheme and skewed label distribution, a range of different label sets has been employed for formulating classification tasks (Rutherford et al., 2017). The most popular choices for PDTB 2.0 are: (1) top-level senses (L1) comprised of four labels, and (2) finer-grained Level-2 senses (L2). For L2, the standard protocol is to use 11 labels after eliminating five infrequent labels as proposed in Lin et al. (2009). Sometimes ENTREL is also included in the L2 label set (Xue et al., 2015). Level-3 senses (L3) are not often used due to label sparsity. Data partitioning. The variability of data splits used in the literature is substantial. This is problematic considering the small number of examples in a typical setup with 1-2 WSJ sections as test sets. For instance, choosing sections 23-24 rather than 21-22 results in an offset of 149, and a label offset as large as 71 (COMPARISON .C ONTRAST ). This is a large enough difference to cast doubt on claims for state-of-the-art, considering the small size of the test sets (  1000 ). We illustrate the variability of split choices in published work in Appendix B. Recently, splits recommended by Prasad et al. (2008) and Ji and Eisenstein (2015) ( Ji ) are the most common, but splits from Patterson and Kehler (2013) ( P&K ), Li and Nenkova (2014), i.a., have also been used. The Prasad et al. split is frequently attributed to Lin et al. (2009) ( Lin ), and thus we adopt this naming convention. Multiply-annotated labels. Span pairs in PDTB are optionally annotated with multiple sense labels. The common practice is either taking only the first label or the approach in Qin et al. (2017), i.a., where instances with multiple annotations are treated as separate examples during training. A prediction is considered correct if it matches any of the labels during testing. However, a subtle inconsistency exists even across works that follow the latter approach. In PDTB, two connectives (or inferred connectives for implicit relations) are possible for a span pair, where the second connective is optional. A connective can each have two semantic classes (i.e., the labels), where the second class is optional. Thus, a maximum of four distinct labels are possible for each span pair. However, in the actual dataset, the maximum number of distinct labels turns out to be two. An inconsistency arises depending on which of the four possible label fields are counted. For instance, Qin et al. (2017) treat all four fields (SCLASS 1A, SCLASS 1B, SCLASS 2A, SCLASS 2B; see link) as possible labels, whereas Bai and Zhao (2018); Bai et al. (2019) use only SCLASS 1A,SC LASS 2A. Often, this choice is implicit and can only be deduced from the codebase. Random initialization. Different random initializations of a network often lead to substantial variability (Dai and Huang, 2018). It is important to consider this variability especially when the reported margin of improvement can be as small as half a percentage point (see cited papers in Table 1). We report the mean over 5 random restarts for existing splits, and the mean of mean cross-validation accuracy over 5 random restarts. 3 3 Proposed Evaluation Protocol While Xue et al. (2015) lay out one possible protocol, it does not fully address the issues we have raised in Section 2. Another limitation is the unavailability of the preprocessing code as of the date of this submission. We describe our proposal below, which will be accompanied by a publicly available preprocessing code. 4 In addition to accounting for the variation previously discussed, we take Shi and Demberg (2017)'s concerns into consideration. Cross-validation. We advocate using cross-validation for L2 classification, sharing the concerns of Shi and Demberg (2017) on label sparsity. However, we propose using cross-validation at section -level rather than individual example -level as suggested by Shi and Demberg (2017). This is to preserve paragraph and document structures, which are essential for investigating the effect of modeling larger context (e.g., Dai and Huang 2018). We further illustrate the potential utility of document structure in Section 4. We suggest dividing the 25 sections of PDTB into 12 folds with 2 development, 2 test and 21 training sections in each fold. We used a sliding window of two sections starting from P&K (dev: 0-1, test: 23-24, train: 2-22). All but one section (22) is used exactly once for testing. Whether future works should evaluate on these particular cross-validation splits or on randomized splits (Gorman and Bedrick, 2019) is an open issue; we provide an additional discussion in Appendix F. Label sets. We recommend reporting results on both L1 and L2, using the standard 11-way classification for L2 in PDTB 2.0. A standardized label set 3 Due to limitations of compute, we only report random restarts of cross-validation (5 seeds x 12 folds) for our main results. For additional experiments in Section 4, we report the average over folds only. Generally, variance over seeds were smaller than over folds for our models. 4 https://github.com/najoungkim/pdtb3 does not exist yet for L2 in PDTB 3.0 (L1 remains unchanged). We propose using only the labels with > 100 instances, which leaves us with 14 senses from L2 (see Appendix A for counts). We suggest using all four possible label fields if the senses are multiply-annotated, as discussed in Section 2.1. Following our proposed protocol, we report baseline results from two strong sentence encoder models: BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019), using a publicly available codebase. 5 See Appendix C for training details. We present L2 results on PDTB 2.0 in Table 1 and results on PDTB 3.0 in Table 2 (see Appendix D for L1 re-sults). To maintain backwards compatibility to the literature, we also report PDTB 2.0 results on Ji, Lin and P&K splits (see Section 2.1). Ji & Lin are the most common splits, and P&K is the split used by Nie et al. (2019) who claim the current state-of-the-art for L2. For PDTB 2.0 (Table 1), our baselines showed strong performance on all splits. XLNet-large was the single best model, significantly outperforming every best reported result. 6 3.2 Single-span baselines Table 4 lists the performance of single-span (ei-ther ARG 1 or ARG 2) baseline models for both PDTB 2.0 and 3.0. This baseline adapts the idea of hypothesis-only baselines in Natural Language Inference (Poliak et al., 2018), where we limit the training data by only showing the models one of the two spans that are in a discourse relation. We discuss these baselines further in Section 4. 6 We used the N  1  2 test to compare proportions instead of a matched test like McNemar's, because we only had access to reported accuracies (rather than raw predictions) of the best models in the literature. over PDTB 2.0. For instance, the annotation manual (Prasad et al., 2019) remarks that LIST was removed since it was not in practice distinguishable from CONJUNCTION . Indeed, models trained on PDTB 2.0 behaved exactly so, classifying most of LIST as CONJUNCTION (but not vice versa, likely due to frequency effect; see Appendix G). We conducted an additional experiment testing the impact of the new annotation scheme, in an attempt to address the question If we want to detect relation X in a downstream task, which PDTB should we use to train our models?. We trained the same model (BERT-large) twice on the same set of datapoints, only varying the annotation scheme. Since PDTB 3.0 has both added and removed examples, we filtered the datasets so that the two PDTBs contained exactly the same span pairs. With the model and inputs fixed, the labeling scheme should be the only effective factor. After filtering, the majority-class baseline for both were less than 30%. Table 5 suggests that PDTB 3.0's annotation scheme does lead to improved distinguishability of CONJUNCTION . 7 PDTB 3.0 overall yielded better 7 We used pooled cross-validation accuracy (compared us-(or unchanged) distinguishability of shared labels except for CONTRAST . This trend was especially salient for CONCESSION that was practically unlearnable from PDTB 2.0. This supports the utility of PDTB 3.0 over 2.0 if downstream transfer is considered, motivating a transition to 3.0. Unsurprisingly, the change in distinguishability was highly dependent on the change in label counts in the training data (Table 5,  ). But change in frequency alone does not give us the full picture. For instance, SYNCHRONOUS remained difficult to learn even with a substantial increase in labeled examples. The absolute size of the class was also not deterministic of performance. There were 192 training instances of SYNCHRONOUS in the filtered PDTB 2.0 and 261 for PDTB 3.0. Similar/smaller classes such as | ALTERNATIVE | = 118 in PDTB 2.0 and | SUBSTITUTION | = 191 in PDTB 3.0 were still learnable with 26% and 48% accuracy, respectively. This was mostly due to SYNCHRONOUS being mislabeled as CONJUNCTION , which was also the case in the unfiltered dataset (see Appendix G). anno-ing Fisher's exact test and Bonferroni correction) because label sparsity made fold-wise comparisons underpowered for small classes like ASYNCHRONOUS . tation scheme for PDTB 3.0 marks the directionality of relations (e.g., ARG 1vs ARG 2ASMANNER ). These relations are important for naturally-occurring discourse, where order-variable asymmetrical relations are common. For example, in Figure 1, span [2] is conditionally dependent on [3], and [5] has a dependency on [4]; such ordered dependencies must be correctly tracked across discourse contexts. We investigated whether directional labels are sufficiently identifiable with our models. We replaced L2 classes with L3 subclasses (L2+L3), if both subclasses had > 100 examples. Except for REASON and RESULT , the distribution of L3 classes under the same L2 is heavily skewed, which led to low performance (Table 3). This calls for a data augmentation that would balance subclass ratios and alleviate label sparsity at L3. informative, even for shallow discourse parsing. We have advocated for an evaluation scheme that preserves larger contexts. This is motivated by the fact that discourse relations are not independently distributed from one another (even when they are annotated in isolation, as in PDTB). For instance, implicit CONJUNCTION ( IC ) relations are likely to be adjacent; in PDTB 3.0, the probability of one IC following another is P ( IC 2 | IC 1 ) = 0 . 14 , when P ( IC ) = 0 . 08 . Implicit REASON is likely to be adjacent to RESULT ; P ( IReason | IResult ) = 0 . 12 , P ( IReason ) = 0 . 05 . Vanilla pretrained encoders are strong, but are overreliant on lexical cues. A simple fine-tuning of pretrained encoders yielded impressive gains. At the same time, they overrelied on lexical cues. For instance, ARG 2-initial to often signals PURPOSE ; 79.9% of such cases are true PURPOSE relations. It is reasonable for our models to utilize this strong signal, but the association was much amplified in their prediction. For example, XLNet-base predicted PURPOSE for 95.8% of the examples with ARG 2-initial to . We also found that model predictions were in general brittle; a simplistic lexical perturbation with no semantic effect, such as appending  ' to the beginning of spans, resulted in a 9% p drop in performance for BERT-large models. Overall, there still remains much overhead for improvement, with our best model at 66% accuracy on PDTB 3.0 L2 classification. Combining pretrained encoders and expanded context modeling to better capture document-level distributional 1 Why can't I receive recovery email? Aggregation of single-span baselines as decontextualized upper-bounds. Lexical cues continue to be informative even for implicit relations, as with the case of ARG 2-initial to . Although these signals could be genuine rather than artifactual, they require comparatively less multi-span reasoning. Then, how much of our dataset only requires shallower reasoning as such? To address this question, we constructed a de contextualized baseline by aggregating predictions of single-span models, and assuming that an oracle always chooses the right answer if it is in the prediction set. This provides an upper-bound estimate of the performance of a model that only disjointly considers the two input spans, but still has full lexical access. Comparing the final rows of Table 4 and Table 2, we see that no model reliably outperforms its decontextualized upper-bound counterpart. We have surveyed the literature to highlight experimental inconsistencies in implicit discourse relation classification, and suggested an improved protocol using section-level cross-validation. We provided a set of strong baselines for PDTB 2.0 and 3.0 following this protocol, as well as results on a range of existing setups to maintain comparability. We discussed several future directions, including data augmentation for downstream transferability, applicability of pretrained encoders to discourse, and utilizing larger discourse contexts. This work was supported by IBM Research. We thank the three anonymous reviewers for their insightful comments. We also thank Sadhwi Srinivas, Grusha Prasad and Tal Linzen for their advice on statistical analysis.
abstain abstain objective objective objective abstain abstain abstain abstain abstain abstain other abstain objective method result objective abstain method objective objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain other abstain method abstain abstain result result abstain result abstain abstain abstain result abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain result result method other other other