Pre-trained language models (PLMs) have shown great potentials in natural language processing (NLP) including rhetorical structure theory (RST) discourse parsing. Current PLMs are obtained by sentence-level pre-training, which is different from the basic processing unit, i.e. element discourse unit (EDU). To this end, we propose a second-stage EDU-level pretraining approach in this work, which presents two novel tasks to learn effective EDU representations continually based on well pre-trained language models. Concretely, the two tasks are (1) next EDU prediction (NEP) and (2) discourse marker prediction (DMP). We take a state-of-the-art transition-based neural parser as baseline, and adopt it with a light bi-gram EDU modification to effectively explore the EDU-level pre-trained EDU representation. Experimental results on a benckmark dataset show that our method is highly effective, leading a 2.1-point improvement in F1-score. All codes and pre-trained models will be released publicly to facilitate future studies. 1 1 Introduction Discourse analysis based on rhetorical structure theory (RST) has received increasing interest in the natural language processing (NLP) community (Yu et al., 2018; Liu et al., 2019a; Kobayashi et al., 2020; Zhang et al., 2020; Guz and Carenini, 2020; Koto et al., 2021; Zhang et al., 2021), which organizes discourse output through a well-defined tree structure. Figure 1 shows an example of an RST constituent tree, where the leaf nodes are element discourse units (EDUs). Given an EDU sequence, RST discourse parsing aims to automatically construct a hierarchical constituent tree 2 . Corresponding author. 1 http://github.com/yunan4nlp/ E-NNRSTParser 2 In this study, we focus on the tree construction task, assuming the gold standard EDU as inputs. e 1 [CNW Corp. said] e 2 [the final step in the acquisition of the company hasbeencompletedwiththemergerofCNWwithasubsidiaryofChicago&NorthWesternHoldingsCorp.] e 3 [As reported,] e 4 [CNW agreed to be acquired by a group of investors] e 5 [led by Blackstone Capital Partners Limited Partnership] e 6 [for $50 a share, or about $950 million.] e 1 e 2 e 3 e 4 e 5 e 6 elab -NS same -NN circ -SN attr -SN elab -SN Figure 1: An example of RST discourse tree. The shift-reduce transition-based model has been widely adopted in RST discourse parsing (Yu et al., 2018; Mabona et al., 2019), building the constituent tree incrementally with multiple steps by a sequence of actions. These models take EDU-level features as inputs to score transition actions at each step. Recently, neural network models have achieved state-of-the-art performance for this task by using sophisticated-designed neural modules (Yu et al., 2018; Liu et al., 2019a; Mabona et al., 2019; Zhang et al., 2020; Kobayashi et al., 2020). In particular, the contextualized pre-trained language models (PLMs) such as XLNet (Yang et al., 2020) is able to achieve an impressive performance, resulting in F1-score gains of more than 3 points according to previous studies (Koto et al., 2021; Zhang et al., 2021; Nguyen et al., 2021) and our preliminary findings. Although great successes have been observed by the contextualized PLMs (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b), an apparent mismatch in the basic processing units exists between the EDU-level RST parsing and the sentence-level contextualized language modeling, which might be unable to fully explore the pre-training paradigm. Several previous studies have been investigated to address the mismatch between the target tasks 4269 and the standard language model pre-training, e.g., SpanBERT (Joshi et al., 2020) for extractive question answering, BART (Lewis et al., 2020), and T5 (Raffel et al., 2020) for sequence-to-sequence (seq2seq) generation, and all these studies achieve improved performances for their target tasks. In this study, we investigate a second-stage EDU-level pre-training based on the above observation. Concretely, we conduct pre-training from a PLM with two EDU-level tasks in the second stage. The first task is next EDU prediction (NEP), which is inspired by next sentence prediction (NSP) in BERT (Devlin et al., 2019) learning, substituting the sentences with EDUs. The second task is discourse marker prediction (DMP), which is also inspired by the masked language modeling (MLM) in BERT (Devlin et al., 2019) learning, substituting the masked words with the masked discourse markers. To fully utilize contextualized pre-trained representations, we adapt a transition-based neural RST parser that exploits BiEDU representations with regard to the basic encoding unit instead of the standard single-EDU manner. We conduct experiments on RST discourse tree-bank (RST-DT) (Carlson et al., 2001) to evaluate the proposed model. First, we derive BiEDU representations directly from PLMs, and thus build a very strong transition-based neural RST parser. Then, we examine the proposed second-stage EDU-level pre-training approach. Experimental results show that the two second-stage pre-training tasks improve RST parsing greatly, and their combination leads to further increases. Our final model achieves the top performance among all the models reported in the literature. In summary, our contributions are as follows:  We present a second-stage EDU-level pretraining approach to address the inconsistency between the EDU-level RST parsing and the sentence-level contextualized language modeling, aiming for a better pre-training paradigm for RST parsing. We suggest BiEDU-based representations for neural RST parsing to exploit well pre-trained language models more effectively. We advance the state-of-the-art RST parsing performance. In this section, we introduce the proposed second-stage EDU-level pre-training approach. It has two CLS t j 1 t jk SEP t i 1 t in ... ... EDU-level pre-training tasks, termed by NEP and DMP, respectively. NEP requiries EDU pairs as inputs, and predicts whether each EDU pair is adjacent. DMP requiries EDU sequences as inputs, and predicts the masked discourse marker between two adjacent EDUs. NEP is inspired by NSP in BERT (Devlin et al., 2019) learning. NSP is a binary sentence-level classification task, which determines whether two sentences are continous. It integrates rich inter-sentence context features into BERT and thus has a positive effect on several downstream classification tasks, such as PDTB-style discourse relation classification (Shi and Demberg, 2019) and Stanford Natural Language Inference (SNLI) (Bowman et al., 2015). RST parsing involves the classification between two subtrees (a single EDU can also become a subtree), which is highly similar to above downstream tasks. Therefore, we believe that a similar second-stage pre-training task is effective for RST parsing. Considering that the basic inputs of RST parsing are EDUs, we substituting the sentences with EDUs. We reimplement a SOTA EDU segmenter 3 (Muller et al., 2019) and use it to segment large-scale unlabeled texts. Based on EDU segmentation data, we apply NEP to PLM. Figure 2 shows an overview of NEP. We sample the continuous EDU pairs as positive instances, and the non-continuous EDU pairs as negative instances. It should be noted that these positive and negative instances are sampled on the same scale. When the these instances are ready, we use Equation 4 to pack each EDU pair and calculate its corresponding EDU representation. Then we use a 3 We also use the RST-DT corpus to train an EDU segmenter, which achieves 96.0% F1-score. where W e and b e are the model parameters of the linear layer, and y e indicates whether the two EDUs are continous. We adopt a cross entropy function as the training objective of NEP. We further adopt DMP to pre-train PLMs in the second stage based on the following consideration. Pitler et al. (2009) point out that if discourse markers (Schiffrin, 1987) exist in PDTB-style discourse parsing, the classification of discourse relation types become easier. RST parsing aims to classify the relationship between two discourse fragments. By analogy, discourse markers can also make RST parsing easier. The framework of DMP is shown in Figure 3. The input of DMP is an EDU sequence. We only mask the first word in each EDU that starts with a discourse marker. 4 Then we use Equations 4 and 5 to obtain EDU representations of the masked EDU sequence. Finally, we feed them into a linear layer to calculate the discourse marker score: y m = W m h ei + b m (2) where W m and b m are the model parameters of the linear layer, and y m is the score distribution of the discourse markers. We also use a cross entropy function as the training objective of DMP. We adopt a transition-based neural RST parser to evaluate the second-stage EDU-level pre-training approach. The model has two key components, termed by a transition system and a neural network model, respectively. The transtion system, mainly borrowed from Yu et al. (2018), formalizes RST parsing into action sequence predictions, and the neural model yields EDU representations and outputs action sequences. As shown in Figure 4, our transition system consists of states and actions. A state has two parts, namely a stack stores partially parsed subtrees and a queue stores un-parsed EDUs. The initial state is an empty state, and the final state represents a full RST discourse tree. A action controls the transition of states. There are three kinds of actions:  A shift action pops the first EDU of the queue and pushes it into the stack. It can only be executed when the queue is not empty. A reduce action combines the top two subtrees of the stack into a new subtree with a unclearity label and a relation label. It can only be executed if there are more than two subtrees are in the stack. A pop root action pops a full discourse tree from the stack, and the parsing process is completed. It can only be executed when the queue is empty, and only one element is in the stack. In summary, the transition system converts a tree construction into a sequence of action predictions. By performing the actions, a RST discourse tree is constructed incrementally. Concretely, given the example in Figure 1, we perform actions shift, 4271 CLS t i 1 t i  1 m SEP t i 1 t in ... ... PLM x i  1 CLS x i  1 1 x i  1 m x i SEP x i 1 x in ... ... BiLSTM ... ... ... ... h ei ... ... Decoder Action i Action j ... ... ... EDU e i  1 EDU e i Figure 5: Framework of our neural network model. The input is an EDU sequence. For convenience, here we draw two adjacent EDUs e i  1 and e i . shift, reduceattr SN , shift, shift, shift, reduce-elab NS , shift, reducesame NN , reducecirc SN , reduceelab SN , pop root to construct a full RST discourse tree step by step. The Vanilla Representation We use PLM to encode each text, obtaining single-EDU representations. Concretely, given a text that has been segmented into EDUs e 1    e n , a special symbol [CLS] is placed at the beginning of each EDU. Then each EDU is tokenized by byte pair encoding (BPE) (Sennrich et al., 2016), and encoded by PLM to obtain contextualized word piece embeddings. Finally, for each EDU, we choose the following representation of [CLS] to represent it: e i = [CLS] , t i 1    t in x i CLS , x i 1    x in = PLM ( e i ) x ei = x i CLS (3) where [CLS] , t 1    t n are word pieces, x t CLS , x t 1    x tn are word piece embeddings, and x ei is the single-EDU representation. Extension with BiEDU The vanilla EDU-based representation exploits the information by treated an EDU as the first segmentation type, leaving its segmentation type unused. Here, we make an extension by using BiEDU representations. Each input unit is packaged by the current EDU as well as the previous EDU jointly, forming as BiEDU. Then [CLS] is placed before the first EDU and [SEP] before the second EDU. We also use BPE to tokenize it and use a PLM for encoding. We still choose the representation of [CLS] to represent each EDU as follow: ( e i  1 , e i ) = [CLS]    t i  1 m , [SEP]    t in x i  1 CLS    x i  1 m , x i SEP    x in = PLM ( e i  1 , e i ) x ei = x i  1 CLS (4) where [CLS]    t i  1 m , [SEP]    t in are tokens, x i  1 CLS    x i  1 m , x i SEP    x in are word piece embeddings, and x ei is the BiEDU representation. BiLSTM Encoding Furthermore, we follow Koto et al. (2021), using BiLSTM to obtain high-level EDU representations: h e 1    h eu = BiLSTM ( x e 1    x eu ) (5) where h e 1    h eu are final EDU representations. In addition, we follow Zhang et al. (2021) and Koto et al. (2021), using paragraph features to further enhance the high-level representations. Decoder The decoder part predicts the next-step action based on a given state. We follow Yu et al. (2018), selecting the three subtrees ( s 1 , s 2 , s 3 ) at the top of the stack and the first EDU ( q 1 ) in the queue to represent the current state. We calculate the subtree representation by the average of its EDU representations. We concatenate three subtree representations ( h s 1 , h s 2 , h s 3 ) and an EDU representation ( h eq 1 ), and input them into a linear layer to calculate the score distribution of the action: y i = W i ( h s 1  h s 2  h s 3  h eq 1 ) + b (6) where W i , b are model parameters and  is a concatenation operation. During the inference, at each step, we exploit the highest-scored action as the output. When actions are ready, we perform them to construct the coresponding RST discourse tree step by step according to the transition system introduced in Section 3.1. Training We adopt a cross-entropy loss plus with l 2 regularization term as an objective function to train our RST parser. Given a state, we obtain action scores according to the neural network model and compute the probability of the gold action by softmax. Finally, we feed it into the objective function for loss calculation as follows: p i = softmax( y i ) L (  ) =  log( p i [ a gi ]) +  ||  || 2 2 (7) 4272 where a gi is the gold-standard action of the i -th step,  is a set of model parameters of our RST parser, and  is the l 2 regularization factor. We use Adam algorithm (Kingma and Ba, 2015) to optimize the model parameters of our neural network model. Datasets To show the proposed model is comparable with previous state-of-the-art systems for RST parsing, we conduct experiments on RST-DT 5 (Carlson et al., 2001). It is a standard benchmark dataset for this task, which is collected from the Wall Street Journal news. It has been divided into training and test sets, which have 347 discourses and 38 discourses, respectively. We randomly select 35 discourses from the training set to develop our model. The original RST-DT contains 78 fine-grained discourse relations. Most of previous studies simplify these fine-grained discourse relations to 18 coarse-grained relations. To facilitate comparison with previous studies, we also use 18 simplified coarse-grained relations. To show the domain generalization capability of our proposed RST parser to unseen domain articles, we test it on the georgetown university multilayer (GUM) corpus 6 . It contains small-scale articles annotated based on RST in several domains, such as news, fiction, conversations, and etc. For more details, one can refer to their paper (Zeldes, 2017). The training corpus for second-stage EDU-level pre-training contains unlabeled large-scale collected from a English Wikipedia corpus 7 . Although using a unlabeled news corpus may lead to greater improvements, we find that using a Wikipedia corpus is sufficient to provide new SOTA results. Evaluation We use the evaluation recommended by Morey et al. (2017), which attaches nuclearity and relation labels to non-leaf trees to eliminate redundant evaluations. The evaluation includes four metrics, termd by Span, Nuclearity, Relation, and Full, respectively. Span evaluates the skeleton of the discourse tree. Nuclearity evaluates the discourse tree with nuclearity labels. Relation evaluates the discourse tree with relation labels. Full evaluates the complete discourse tree with nuclearity and relation labels. Hyper-parameters There are several hyper-parameters in our proposed second-stage EDU-level pre-training approach and RST parser. In NEP, the learning rate of PLM is set to 5e-6, and the learning rate of the other model parameters is set to 1e-3. The batch size is set to 50. The maximum norm of gradient clipping is set to 1. The maximum tranining epoch number is set to 10. In DMP, the learning rate of PLM is set to 1e-6, and the learning rate of the other model parameters is set to 1e-4. The batch size is set to 1. The output hidden size of LSTM is set to 200. The settings of maximum training iteration number and the norm of gradient cliping are the same as NEP. The hyper-parameters of our RST parser are tuned based on the preliminary results on the development set. The hidden size of all neural layers is set to 200. The dropout is set to 0.25. The learning rate of PLM is set to 2e-5, and the learning rate of other model parameters is set to 1e-3. The maximum norm of gradient clipping is set to 1, and the maximum training iteration number is set to 20. We use transformers library (Wolf et al., 2020) to implement PLM and use PyTorch (Paszke et al., 2019) to implement other neural network modules. We conduct several development experiments to show the important factors that influence the performance of our RST parser. Different Pre-trained Language Models First, we test our proposed RST parser based on several publicly available PLMs such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), XLNet (Yang et al., 2020), SpanBERT (Joshi et al., 2020), and DeBERTa (He et al., 2020). The max input length of BERT, RoBERTa, SpanBERT, and DeBERTa is 512 tokens. Therefore, we extend them with BiEDU to better exploit these PLMs. Since XLNet has no input length limit, we do not need to apply BiEDU extension to our XLNet RST parser. Table 1 shows the performances of with different PLMs. We find that our BiEDU extension is able to further improve the performances of these PLM-based RST parsers. The SpanBERT RST parser achieves worst performance among these RST parsers. It is probably because that the basic processing units of SpanBERT learning are not matched with RST parsing. The XLNet RST parser achieves the best performance among these RST parser. Therefore, following experiments are 4273 Models Full (dev) Full (test) BERT 49.0 45.2 BERT + BiEDU 51.4 48.9 RoBERTa 50.8 48.0 RoBERTa + BiEDU 51.7 49.5 SpanBERT 41.0 38.8 SpanBERT + BiEDU 42.5 39.2 DeBERTa 48.6 47.0 DeBERTa + BiEDU 49.8 48.1 XLNet 52.2 51.4 Table 1: Performances of our RST parser with different PLMs. Unlabeled Article Size We study how the unlabeled articles size in second-stage EDU-level pretraining influences the performance of our RST parser. First, we apply NEP to PLMs. As shown in Figure 6, the performance of our RST parser shows a similar trend when increasing the size of unlabeled articles to perform DMP based pre-training. When the size of the unlabeled articles reaches 30k, the Full metric reaches its peak. Therefore, we use 30k unlabeled articles in NEP. Then, we adopt DMP to further pre-train the PLM part of our RST parser in the second stage. As can be seen from Figure 6, the performance of our RST parser first increases and then decreases as the size of the unlabeled articles as the size the unlabeled articles gradually increases from 0 to 240k. When the size of the unlabeled articles reaches 120k, the Full metric reaches its peak. Therefore, we use 120k unlabeled articles in DMP. Above experimental results show that we do not need an ultra large-scale unlabeled corpus for our proposed second-stage EDU-level pre-training approach. As shown in Table 2, we report main results on the RST-DT test set. Our proposed RST parser achieves 73.4 on the Span metric, 63.3 on the Nuclearity metric, 52.4 on the Relation metric, and Models S N R F XLNet (transition-based) 73.4 63.3 52.4 51.4 + NEP + DMP 76.4 66.1 54.5 53.5 XLNet (top-down) 73.3 62.7 51.9 49.7 + NEP + DMP 72.9 62.7 52.5 50.5 Feng and Hirst (2014) 68.6 55.9 45.8 44.6 Ji and Eisenstein (2014) 64.1 54.2 46.8 46.3 Joty et al. (2015) 65.1 55.5 45.1 44.3 Surdeanu et al. (2015) 65.3 54.2 45.1 44.2 Li et al. (2016) 64.5 54.0 38.1 36.6 Hayashi et al. (2016) 65.1 54.6 44.7 44.1 Braud et al. (2016) 59.5 47.2 34.7 34.3 Braud et al. (2017) 62.7 54.5 45.5 45.1 Yu et al. (2018) 71.4 60.3 49.2 48.1 Mabona et al. (2019) 67.1 57.4 45.5 45.0 Zhang et al. (2020) 67.2 55.5 45.3 44.3 Nguyen et al. (2021) 74.3 64.3 51.6 50.2 Koto et al. (2021) 73.1 62.3 51.5 50.3 Zhang et al. (2021) 76.3 65.5 55.6 53.8 Human 78.7 66.8 57.1 55.0 Table 2: Final results of RST parsing on the test set. 51.4 on the Full metric, exceeding most of the previous state-of-the-art systems. When we apply second-stage EDU-level pre-training to XLNet, it achieves 76.4 on the Span metric and 66.1 on the Nucleairty metric, resulting a Full metric improvement 53.5 51.4 = 2.1. The Span, nuclearity, and relation metrics have similar tendencies as well. In addition, we implement a top-down RST parser, and also enhance it with using our proposed second-stage EDU-level pre-training approach. We find that the proposed approach is able to improve the performance of top-down RST parser as well. We compare our proposed RST parser with previous state-of-the-art systems. Feng and Hirst (2014) propose a linear-chain conditional random field (CRF) parser. Ji and Eisenstein (2014) adopt a statistical transition-based parser with a representation learning. Surdeanu et al. (2015) employ a perceptron and a logistic regression to parse a text. Li et al. (2016) propose a hierarchical neural parser with attention. Joty et al. (2015) propose an intra-sentential and multi-sentential parser. Hayashi et al. (2016) reimplement the HILDA parser (Heilman and Sagae, 2015), using a linear SVM classification to parse a text from the bottom up. Braud et al. (2016) present a BiLSTM RST parser with multi-task learning. Braud et al. (2017) propose a neural greedy parser with cross-lingual recourse. Yu et al. (2018) propose a transition-based neural parser, and further enhance it with hidden-layer vectors extracted from a neural syntax parser. Mabona et al. (2019) propose a generative RST parser with beam search. Zhang et al. (2020) propose a top-4274 Models S N R F Our proposed model 76.4 66.1 54.4 53.5 DMP 74.8 64.3 53.1 52.1 NEP 75.3 65.0 53.8 52.6 NEP DMP 73.4 63.3 52.4 51.4 NEP DMP Para 73.1 62.6 51.9 50.5 Table 3: Ablation study on the test set. down neural parser. Koto et al. (2021) propose a transformer top-down parser with dynamic or-cale. Nguyen et al. (2021) propose a seq2seq neural parser based on point network. Koto et al. (2021) propose a sequence labelling parser with dynamic oracle. Zhang et al. (2021) propose a neural top-down parser with adversarial learning. As shown in Table 2, our transition-based XLNet RST parser achieves the best performance among the systems studied on the Span and the Nuclearity metrics. We find that the Relation and the Full metrics of our RST parser are lower than that of Zhang et al. (2021). It is probably because that our proposed second-stage EDU-level pre-training approach only requires predicted EDU segmentation, lacking the information of predicted RST discourse trees. Ablation Studies Here we conduct several ablation experiments to examine the effectiveness of our proposed second-stage EDU-level pre-training approach and paragraph features. As shown in Table 3, we find that NEP and DMP are effective for RST discourse parsing. NEP improves our XLNet RST parser by an increase of 52.1 51.4 = 0.7 on the Full metric. The tendency of DMP is similar to NEP, obtaining an increase of 52.6 51.4 = 0.8 on the Full metric. Our proposed model can be further improved when two EDU-level tasks are applied to XLNet, resulting the Full metric improvement 53.5 51.4 = 2.1. In addition, the paragraph features is also effective for RST discourse parsing, which results the overall improvements. Effect of EDU Segmentation Performance As mentioned earlier, the second-stage EDU-level pretraining approach requires EDU segmentation produced by a supervised EDU segmenter. Predicted EDU segmentation could have errors, which may propagate into RST parsing. Here we examine how +NEP+DMP 51 52 53 54 F u ll ( % ) 0 (0.0%) 10 (57.0%) 100 (80.5%) 300 (96.0%) Figure 7: Effect of EDU segmenter performances on our proposed RST parser. the performance of the supervised EDU segmenter influence the performance of RST parsing. The full EDU segmenter is trained on 300 discourses. We retrain two weaker EDU segmenters on 10 and 100 discourses. Figure 7 shows the RST parsing performances with different EDU segmenters. We find that the EDU segmentation performance influences the RST parsing quality, indicating the importance of correct EDU segmentation. Analysis by Number of EDUs in Subtrees As mentioned earlier, NEP predicts whether each EDU pair is continous, and it is able to integrate rich inter-EDU context features into PLMs. Therefore, it is expected that the introduce of NEP may bring better improvements for the spans containing more EDUs. As such, here we investigate the benefit by using NEP. Table 4 shows the comparison results. We find that performances are improved significantly when spans contains more EDUs. Effect of Different Sampling Strategies Forther-more, we examine how different EDU pair sampling strategies influence RST discourse parsing. The training set of NEP is sampled from a large-scale unlabeled corpus. We sample the continuous EDU pairs as the positive instances and the noncontinuous EDU pairs as the negative instances. The difficulty of NEP changes depending on how the non-continuous EDU pairs are sampled. Here we compare four strategies of sampling the noncontinuous EDU pairs: from a sentence, two adjacent sentences, two sentences in an article, and two 4275 Sampling Strategies S N R F From a sentence 74.4 63.9 53.4 52.3 From adjacent sentences 74.9 64.6 53.4 52.2 From a article 74.8 64.3 53.1 52.1 From two articles 73.9 64.2 52.6 51.9 XLNet 73.4 63.3 52.4 51.4 Table 5: Influence of different sampling strategies on our XLNet RST parser. different articles, respectively. Table 5 shows the comparsion results. We find that these sampling strategies do not make difference to RST parsing. Analysis by Number of Discourse Markers As mentioned earlier, DMP predicts the masked discourse markers of an EDU sequence and discourse markers are essential cues for RST parsing. Therefore, it is expected that the introduce of DMP may bring better performance for the spans containing discourse markers. As such, here we investigate the benefit by adopting DMP. Table 6 shows the performance of our XLNet RST parser with DMP and without DMP. The performances are improved significantly when spans contain discourse markers, which is consistent with our intuitions. Effect of Different Masking Strategies Then we change the masking strategy in DMP to show how different masking strategies influences RST parsing. We use a random word set to replace the discourse marker set in DMP. The number of random words is the same as the number of discourse markers. Compared with discourse markers, these random words may be unable to offer key cues for discourse parsing. As shown in Table 7, the masking discourse markers strategy leads to performance improvement, and the masking random words strategy leads to slight performance degradation. Therefore, it is thus clear that discourse markers are useful for RST parsing. Result on GUM Corpus Finally, we test our proposed RST parser on the GUM corpus (Zeldes, 2017) to show the domain generalization capability of our proposed RST parser. As shown in Table 8, the performance of our XLNet RST parser declines significantly for these out-of-domain articles, especially in conversation and vlog domains. By using our proposed second-stage EDU-level pretraining approach, the performance of the XLNet RST parser can be improved in academic, conversation, textbook, and whow domains significantly, and the performances declines slightly in bio, speech, and vlog domains. Therefore, there is still a lot of room for improvement in the generalization ability of our proposed RST parser. RST discourse parsing is an important task in the NLP community, which has been studied since early (Soricut and Marcu, 2003). Early studies adopt statistical models for this task, using human-designed discrete features (Hernault et al., 2010; Feng and Hirst, 2012; Joty et al., 2013; Feng and Hirst, 2014; Heilman and Sagae, 2015; Wang et al., 2017). Recently, several neural network models show great promising for this task (Braud et al., 2016, 2017; Liu and Lapata, 2017; Yu et al., 2018; Mabona et al., 2019; Zhang et al., 2020; Guz and Carenini, 2020). With PLMs such EMLo (Pe-ters et al., 2018), BERT (Devlin et al., 2019), XLM-RoBERTa (CONNEAU and Lample, 2019), and XLNet (Yang et al., 2020), these neural RST parsers report high competitive performances (Liu et al., 2019a; Lin et al., 2019; Liu et al., 2020; Kobayashi et al., 2020; Zhang et al., 2021; Nguyen et al., 2021). We follow the line of these studies, using neural networks to perform RST parsing. Recently, several studies aim to alleviate the mismatch between pre-trained language models and target tasks. Joshi et al. (2020) use a span masked language modeling to pre-train a language model for extraction question answering. Lewis et al. (2020) propose a pre-training approach for text generation tasks, which maps corrupt documents 4276 Domains XLNet XLNet + NEP + DMP S N R F S N R F academic 65.8 50.7 35.2 34.6 66.6 (+0.8) 52.0 (+0.3) 35.6 (+0.4) 35.0 (+0.4) bio 57.3 41.3 32.0 31.6 57.2 (-0.1) 41.0 (-0.3) 31.0 (-1.0) 30.6 (-1.0) conversation 36.8 23.7 12.6 12.2 32.7 (+0.9) 22.0 (-1.7) 12.6 (+0.0) 12.4 (+0.2) fiction 57.3 41.0 28.1 27.4 57.1 (-0.2) 40.8 (-0.2) 28.1 (+0.0) 27.5 (+0.1) interview 61.5 43.8 31.5 30.8 61.8 (+0.3) 42.8 (-1.0) 31.5 (+0.0) 30.9 (+0.1) news 67.4 51.7 40.5 39.5 68.6 (+1.2) 52.5 (+0.8) 40.5 (+0.0) 39.6 (+0.1) speech 71.5 58.2 45.6 45.6 70.7 (-0.8) 56.9 (-1.3) 44.2 (-1.4) 44.0 (-1.6) textbook 64.7 51.3 39.8 39.2 65.9 (+1.2) 53.2 (+0.9) 41.8 (+1.0) 41.7 (+1.5) vlog 47.0 33.2 21.1 20.2 44.7 (-2.3) 31.7 (-1.5) 19.2 (-1.9) 18.5 (-1.7) voyage 65.0 45.3 31.6 30.6 64.9 (-0.1) 45.5 (+0.2) 31.8 (+0.2) 30.6 (+0.0) whow 60.1 41.8 27.7 26.8 62.7 (+1.6) 44.5 (+2.7) 28.2 (+0.5) 27.4 (+0.6) Table 8: Performances on GUM corpus. to the original. Raffel et al. (2020) propose an unified text-to-text pre-training framework for several NLP tasks. Our work mainly inspired by above studies. In this paper, we propose a second-stage EDU-level pre-training approach to alleviate the mismatching between EDU-level RST parsing and sentence-level language modeling. There are several studies have shown that pesudo data is useful for RST parsing. Huber and Carenini (2019) use pesudo RST discourse trees to train a RST parser, which generated by distant supervision on a sentiment classification. Kobayashi et al. (2021) improve RST parsing with large-scale sliver agreement subtrees, which is produced by a well trained RST parser. Zhang et al. (2021) train a top-down RST paser with predicted RST discourse trees. Above approaches requires a well trained RST parser to generate pesudo RST discourse trees. In this work, the generation of our pesudo data merely requires an EDU segmenter and discourse markers, without using a well trained RST parser to further generate pesudo RST discourse trees. We proposed a second-stage EDU-level pretraining approach for PLM-based RST discourse parser, reducing the mismatch between the EDU-level RST discourse parsing and the pre-training of sentence-level contextualized language modeling. In addition, we extended our RST discourse parser with a light bi-gram EDU modification, finding that it is able to exploit PLMs more effectively. Experiments on RST-DT (Carlson et al., 2001) showed that the proposed approach can bring significantly better performance for RST discourse parsing. We further conducted several experimental analysis to better understand the proposed approach. and the GUM (Zeldes, 2017) corpora suggest two possibilities for future research. First, although the XLNet RST parser obtains significantly improvements when the second-stage EDU-level pretraining approach is adopted, the Relation and the Full metrics of our RST parser are still lower than the best system. Future research might extend the second-stage EDU-level pre-training task, using pesudo RST discourse trees. Second, the generalization ability of our proposed RST parser needs to be improved in multi-domain scenarios. So in future we may continue to explore the issue of domain adapation in RST parsing on the basis of the second-stage EDU-level pre-training framework. The authors would like to thank the anonymous reviewers for their constructive comments, which help to improve the paper. This work was supported by National Natural Science Foundation of China under grants 62076173, U1836222, and 62176180.
abstain abstain objective abstain objective result abstain other abstain abstain abstain other other abstain abstain abstain result abstain other objective method abstain abstain method abstain objective objective abstain result objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain method method abstain method abstain abstain abstain other abstain method abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain method abstain result other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain result abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain result abstain abstain other other other other method other other other other other abstain objective other other other other other method objective objective abstain objective abstain objective abstain objective objective other other