Implicit discourse relation recognition (IDRR) aims to identify logical relations between two adjacent sentences in the discourse. Existing models fail to fully utilize the contextual information which plays an important role in interpreting each local sentence. In this paper, we thus propose a novel graph-based Context Tracking Network (CT-Net) to model the discourse context for IDRR. The CT-Net firstly converts the discourse into the paragraph association graph (PAG), where each sentence tracks their closely related context from the intricate discourse through different types of edges. Then, the CT-Net extracts contextual representation from the PAG through a specially designed cross-grained updating mechanism, which can effectively integrate both sentence-level and token-level contextual semantics. Experiments on PDTB 2.0 show that the CT-Net gains better performance than models that roughly model the context. Implicit discourse relation recognition (IDRR) aims to identify logical relations between two adjacent sentences in discourse without the guidance of connectives (e.g., because, but), which is one of the major challenges in discourse parsing. With the rise of deep learning, lots of sentence-modeling based methods (Liu and Li, 2016; Rnnqvist et al., 2017; Bai and Zhao, 2018; Xu et al., 2019; Shi and Demberg, 2019) have emerged in the field of IDRR. These methods typically focus on modeling the local semantics of these two sentences, without considering wider discourse context. Contextual information plays an important role in understanding sentences. Take the paragraph P = { S 1 , S 2 , S 3 , S 4 } in Figure 1 as an example, the ground-truth relation between S 3 and S 4 is Comparison. Combining the contextual information carried by S 1 and S 2 , we can more easily identify the Comparison relation reflected by The manufacturer went public at $15.75 a share in August 1987. achieve that price (rising: $15.75 a share to $29 per-share) and softened (falling: $29 per-share to $25 a share). Dai and Huang (2018) move one step on utilizing wider discourse context, where they use a hierarchical BiLSTM (H-LSTM) to model the whole paragraph rather than only the two sentences, to obtain context-aware sentence representation. However, there are still two limitations in their model. First, they roughly merge all the information in the paragraph, which dilutes the role of key context that closely related to the current sentence. Second, the H-LSTM suffers from the long-distance forgetting problem, which may fail to model the long-distance and non-continuous dependency across multiple sentences (like green lines in Figure 1). To overcome these limitations, we propose a novel Context Tracking Network (CT-Net), which can track essential context for each sentence from the intricate discourse, without being affected by the spatial distance. The CT-Net computes contextual representation through two main steps. Firstly, it converts the paragraph into the paragraph association graph (PAG) (Figure 1), which contains three types of edges between sentences, namely (1) adjacency edge (black lines): connecting adjacent sentences, (2) co-reference edge (purple lines): connecting sentences with co-reference associations, and (3) lexical chain edge (green lines): connecting sentences containing related words. Each sentence can track closely related context along these S " S " S " S " C l a ss i fi e r L a y e r Input edges, including long-distance sentences involving the same object or topic. Secondly, the CT-Net extracts contextual representation over the PAG. To effectively incorporate fine-grained information carried by tokens, we propose the cross-grained updating mechanism, which will be executed multiple recurrent rounds. At each round, it performs semantic exchange via three processes:  Token-to-Sentence Updating : updating the sentence representation with its tokens to grasp fine-grained semantics. Sentence-to-Sentence Updating : performing interaction between sentences on the PAG to get context-aware sentence representation. Sentence-to-Token Updating : using the context-aware sentence representation to update tokens, so that each token can also incorporate contextual information. The obtained context-aware token representation will be used for the computation of the next round. After multiple rounds, the CT-Net obtains the contextual representation that fully combines sentence-level and token-level contextual semantics. Our main contributions are two folds. 1 First, we propose a novel CT-Net for IDRR, which builds the PAG to track closely related context for each sentence in the intricate discourse, and incorporates multi-grained contextual semantics via the cross-grained updating mechanism. Second, experiments on PDTB 2.0 demonstrate that the CT-Net gains better performance than a variety of approaches that roughly model the discourse context. The input of the CT-Net is a paragraph P = ( S 1 , S 2 , ..., S n  1 , S n ) . Here, S n  1 and S n are the adjacent sentences to be classified, while S 1 , ..., S n  2 are context with background information. Our goal is to identify the relation between S n  1 and S n . We firstly build a paragraph association graph (PAG) for P (Section 2.1), then employ the cross-grained updating mechanism on the PAG to extract the contextual representation of S n  1 and S n (Section 2.2). The contextual representation is then used for the final classification (Section 2.3). The CT-Net firstly converts the P into a PAG G = ( V , E ) , where V and E are the sets of nodes and edges respectively. As shown in Figure 2, the PAG contains sentence nodes (blue) and token nodes (orange). Each token node is connected with its corresponding sentence node. We carefully design the edges between sentence nodes so that each sentence only connects the ones that are closely related to it. Specifically, there are three types of edges between sentence nodes in the PAG:  Adjacency Edge (black edges). Adjacent sentences tend to carry important contextual information. Therefore, we add adjacency edges between the neighbors in the discourse. Co-reference Edge (purple edges). Sentences with co-reference associations tend to involve the same object and be highly related, so we add a co-reference edge between them. Lexical Chain Edge (green edges). Lexical chain tracks related words that run through the whole paragraph. Sentences containing the same words or synonyms (except stop words) tend to involve the same topic, therefore, we add a lexical chain edge between them. We give more details of the PAG in Section 3.2. The CT-Net then extracts contextual representation of S n  1 and S n from the PAG G through cross-grained updating mechanism, which is executed T rounds. At the t -th round, we denote the state of the i -th sentence node as g ti , and the state of the j -th token node of the i -th sentence as h ti,j . The states transition from the ( t 1 )-th to the t -th round consists of three computation processes: token-to-sentence updating, sentence-to-sentence updating and sentence-to-token updating. The first two processes are responsible for updating sentence nodes, while the last one is for updating token nodes. Node Initialization. When t = 0 , we initialize token nodes with the concatenation of char, GloVe (Pennington et al., 2014) and ELMo (Peters et al., 2018) embeddings. And the dimension is reduced: h 0 i,j = x i,j = W [ x chari,j ; x glovei,j ; x elmoi,j ] + b where W , b are parameters. The sentence node g 0 i is initialized as the average of its token nodes. Token-to-Sentence Updating. This process updates the sentence state g ti with the token states of last round h t  1 i,j . We employ Sentence-state LSTM (SLSTM) (Zhang et al., 2018) to achieve this. SLSTM is a novel graph RNN that converts a sentence into a graph with one global sentence node and several local word nodes, just like the sub-graph in the PAG (inside the dotted ellipse in Figure 2). At the t -th round, the hidden state of i -th sentence g ti is computed as follows: g ti =SLSTM h  g ( h t  1 i, 0 , h t  1 i, 1 ..., h t  1 i, | S i | , g t  1 i ) (2) where SLSTM h  g represents the process of updating the sentence state with token states by SLSTM, and its detailed equations are shown in Appendix A. | S i | is the number of tokens in S i . Sentence-to-Sentence Updating. After merging token semantics, sentences further grasp sentence-level contextual semantics through the interaction between sentence nodes on the PAG. Since there are three types of edges, we employ Multi-Relational GCN (Schlichtkrull et al., 2018) to get contextual sentence representation c ti of S i : c ti = where W g , W r are model parameters. R is the set of edge types between sentence nodes. N ri denotes neighbours of the i -th sentence node of relation r , where r  R . is the ReLU function. Sentence-to-Token Updating. This process is for updating token states. It conveys the sentence-level contextual information c t  1 i to the token, which is also achieved by the SLSTM. At the t th round, the hidden state of each token h ti,j is computed as follows: h ti,j =SLSTM g  h where x i,j is the initial token embedding. We show the detailed equations of SLSTM g  h in Appendix A. Then, the obtained h ti,j is used for the token-to-sentence updating of the next round. After T rounds, we get c Tn  1 and c Tn as the final contextual representations of S n  1 and S n , respectively, which fully combine token-level and sentence-level contextual semantics. After obtaining global contextual representations c Tn  1 and c Tn , we use a one-layer BiLSTM (Hochre-iter and Schmidhuber, 1997) to encode S n  1 into l n  1 by concatenating the last hidden states in two directions, and encode S n into l n in the same way. l n  1 and l n are local representations without considering wider context. We then concatenate global and local features as follows: X cls = concat( l n  1 , l n , c Tn  1 , c Tn ) X cls is then fed into a two-layer MLP (a fully-connected layer with ReLU activation followed by a softmax output layer) for classification. Multi-Task Training. Following previous works (Dai and Huang, 2018; Nguyen et al., 2019), we apply multi-task learning to improve the performance. The main task is implicit discourse relation recognition (IDRR), while the auxiliary tasks are explicit discourse relation recognition (EDRR) and connective prediction (CP). These three tasks share the same encoder but use three different MLPs. The objective function is as follows: L =   C idrr (cid:88) j =1 y jidrr log (cid:98) y jidrr   C edrr (cid:88) j =1 y jedrr log (cid:98) y jedrr   C cp (cid:88) j =1 y jcp log (cid:98) y jcp (6) where , ,  are adjustable hyper-parameters. y idrr , y edrr and y cp are ground-truth labels of IDRR, EDRR and CP respectively, while (cid:98) y idrr , (cid:98) y edrr and (cid:98) y cp are corresponding predictions. C idrr , C edrr and C cp represent the number of classes of IDRR, EDRR, and CP respectively. We conduct experiments on PDTB 2.0 (Prasad et al., 2008), which contains 16 , 224 implicit instances and 18 , 459 explicit instances. We perform one-vs-others binary classification and 4-way classification on 4 top-level discourse relations: comparison (Comp.), contingency (Cont.), expansion (Exp.), and temporal (Temp.). Following Pitler et al. (2009), we use sections 2 20 for training, sections 21 22 for test and sections 0 1 for validation. The metric is F1 score, and for 4-way classification, we calculate the macro-average F1 score. Details of the PAG. We set the number of sentences to build PAGs as 6 , and use zero padding when the text is less than 6 sentences. When building the PAG, we employ spaCy ( https: //spacy.io/ ) to identify co-reference chains, use simple matching to recognize the same words and use WordNet (Miller, 1995) to recognize synonyms. The WordNet covers 59 . 38% ( 7558 / 12632 ) training samples, 59 . 05 % ( 699 / 1183 ) developing samples, and 56 . 98% ( 596 / 1046 ) testing samples. The average number of edges in the PAG is 11 . Details of Parameters and Training. For the node embedding initialization, we use 150 dimensional char embedding obtained by a CNN (Kim, 2014) with kernel window size of [ 1 , 2 , 3 ], 300 -dimensional-GloVe embedding, and ELMo with 1024 dimension (the output of the second layer of BiLSTM). We reduce the dimension of node states as 512 , so that the dimensions of SLSTM and MR-GCN are also 512 . The iteration rounds of the cross-grained updating mechanism is set as 6 . The size of the BiLSTM which is used to compute local features (Section 2.3) is 128 . For multi-task learning, we set the  ,  ,  as 1 . 0 , 0 . 5 , 0 . 5 . The learning rate is 0 . 001 with batch size of 64 . The number of parameters of the CT-Net is about 16 M. We use the F1 score as the criterion when manually tuning the hyper-parameter values. The Model Comp. whole model is trained end to end with the ADAM optimizer (Kingma and Ba, 2014) on two Tesla P40s with 24 GB GPU memory, and the average runtime is about 6 hours. Main Results (Table 1). We carefully design four baselines with different paragraph encoders for a full comparison: (1) NoContext, the model only using BiLSTM to get local features without considering wider context. (2) BiLSTM, the model using BiLSTM to encode the paragraph. (3) H-LSTM, the model using hierarchical BiLSTM as paragraph encoder. (4) FCG-Net, the model replacing the PAG in the CT-Net with a fully-connected graph (FCG). Except for the way of encoding paragraph, the other settings of these models are the same as the CT-Net. We can draw the following three conclusions. First, NoCon-text obtains the worst performance in most cases, demonstrating the necessity of using contextual representations. Second, the CT-Net gains better performance than models with sequential paragraph encoders BiLSTM and H-LSTM, which proves the superiority of our graph-based CT-Net. The reason is that the CT-Net can track and model closely related context for sentences including long-distance ones. Third, replacing the PAG in the CT-Net with the FCG (FCG-Net) brings a quality drop, which proves the PAG effectively pick out appropriate context that benefits on sentence understanding. We also performed paired t-test between CT-Net and these 4 baselines. The CT-Net is significantly Model Comp. Analysis of the PAG (Table 2). The PAG contains three types of edges: adjacency edge (Adj.), co-reference edge (Coref.) and lexical chain edge (Lex.). To understand the impact of these edges, we conduct ablation experiments on 4-way classification. Rows 1 3 report the results of removing Adj., Coref., and Lex. respectively. Removing Adj." brings the biggest drop ( 0 . 97% ), which reflects that the adjacency edge plays the most important role in the PAG. We also explore the impact of the number of sentences in the PAG. Rows 4 6 report the results. The CT-Net gains the best performance when the PAG contains 6 sentences, and modeling a longer paragraph of 8 sentences causes a decline. We hypothesize that modeling a paragraph this is too long may introduce some irrelevant context, resulting in a reduction in performance. Comparison with Existing Systems (Table 3). Table 3 shows the comparison with existing systems. Our method outperforms other models on 4-way classification, and also gains the best performance on the binary classifications of temporal (Temp.) and expansion (Exp.). Ablation Study of Multi-task Learning (Ta-ble 4). Following Dai and Huang (2018) and Nguyen et al. (2019), we utilize the explicit discourse relation recognition (EDRR) and connective prediction (CP) as auxiliary tasks to help implicit discourse relation recognition (IDRR). We conduct ablation experiments of the two auxiliary tasks on 4-way classification (Table 4) to show their impact. Row 1 is the performance of the CT-Net. Rows 2 3 report the performance of removing the auxiliary task. As expected, the EDRR contributes more to the IDRR than the CP does, which is because that the EDRR is a more similar task with the IDRR. We propose a novel graph-based Context Tracking Network (CT-Net) to model the context for implicit discourse relation classification. The CT-Net first converts the paragraph into the paragraph association graph (PAG), where each sentence tracks their appropriate context through different edges, then employs the cross-grained updating mechanism to combine sentence-level and token-level contextual information. Experiments on PDTB 2.0 demonstrate that the CT-Net captures more effective contextual information than carefully designed baselines with different context encoders.
abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective objective objective abstain objective objective abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain objective abstain abstain