This paper presents the first study on using large-scale pre-trained language models for automated generation of an event-level temporal graph for a document. Despite the huge success of neural pre-training methods in NLP tasks, its potential for temporal reasoning over event graphs has not been sufficiently explored. Part of the reason is the difficulty in obtaining large training corpora with human-annotated events and temporal links. We address this challenge by using existing IE/NLP tools to automatically generate a large quantity (89,000) of system-produced document-graph pairs, and propose a novel formulation of the contextualized graph generation problem as a sequence-to-sequence mapping task. These strategies enable us to leverage and fine-tune pre-trained language models on the system-induced training data for the graph generation task. Our experiments show that our approach is highly effective in generating structurally and semantically valid graphs. Further, evaluation on a challenging hand-labeled, out-of-domain corpus shows that our method outperforms the closest existing method by a large margin on several metrics. We also show a downstream application of our approach by adapting it to answer open-ended temporal questions in a reading comprehension setting. 1 1 Introduction Temporal reasoning is crucial for analyzing the interactions among complex events and producing coherent interpretations of text data (Duran et al., 2007). There is a rich body of research on the use of temporal information in a variety of important application domains, including topic detection and tracking (Makkonen et al., 2003), information extraction (Ling and Weld, 2010), parsing of clinical records (Lin et al., 2016), discourse analy-1 Code and pre-trained models available at https:// github.com/madaan/temporal-graph-gen sis (Evers-Vermeul et al., 2017), and question answering (Ning et al., 2020). Graphs are a natural choice for representing the temporal ordering among events, where the nodes are the individual events, and the edges capture temporal relationships such as before, after or simultaneous. Representative work on automated extraction of such graphs from textual documents includes the early work by Chambers and Jurafsky (2009), where the focus is on the construction of event chains from a collection of documents, and the more recent CAEVO (Chambers et al., 2014) and Cogcomptime (Ning et al., 2018), which extract a graph for each input document instead. These methods focus on rule-based and statistical submodules to extract verb-centered events and the temporal relations among them. As an emerging area of NLP , large scale pre-trained language models have made strides in addressing challenging tasks like commonsense knowledge graph completion (Bosselut et al., 2019) and task-oriented dialog generation (Budzianowski and Vulic, 2019). These systems typically fine-tune large language models on a corpus of a task-specific dataset. However, these techniques have not been investigated for temporal graph extraction. This paper focuses on the problem of generation of an event-level temporal graph for each document, and we refer to this task as contextualized graph generation. We address this open challenge by proposing a novel reformulation of the task as a sequence-to-sequence mapping problem (Sutskever et al., 2014), which enables us to leverage large pre-trained models for our task. Further, different from existing methods, our proposed approach is completely end-to-end and eliminates the need for a pipeline of sub-systems commonly used by traditional methods. We also address a related open challenge, which is a prerequisite to our main goal: the difficulty of obtaining a large quantity of training graphs with Figure 1: Task overview: given a document (left), automatically extract a temporal graph (right). human-annotated events and temporal relations. To this end, we automatically produce a large collection of document-graph pairs by using CAEVO , followed by a few rule-based post-processing steps for pruning and noise reduction. We then encode the graph in each training pair as a string in the graph representation format DOT , transforming the text-to-graph mapping into sequence-to-sequence mapping. We fine-tune GPT -2 on this dataset of document-graph pairs, which yields large performance gains over strong baselines on system generated test set and outperforms CAEVO on TimeBank-Dense (Cassidy et al., 2014) on multiple metrics. Figure 1 shows an example of the input document and the generated graph by our system. In summary, our main contributions are: 1. We present the first investigation on using large pre-trained language models for contextualized temporal event graph generation by proposing a new formulation of the problem as a sequence-to-sequence mapping task. 2. We address the difficulty of obtaining a large collection of human-annotated graphs, which is crucial for effective fine-tuning of pre-trained models, by automatically producing a collection of 89,000 document-graph pairs. 3. Our experimental results on both the system-generated test set (which allows us to compare the relative performance of different models) and a hand-labeled, out-of-domain dataset (TimeBank-Dense), show the advantage of our proposed approach over strong baselines. Further, we show that our approach can help in generating plausible answers for open ended-temporal questions in a reading comprehension dataset, Torque (Ning et al., 2020). Temporal Graph Extraction Tempeval-3 (Uz-Zaman et al., 2013) introduced the task of temporal graph extraction as the ultimate task for evaluating an end-to-end system that goes from raw text to TimeML annotation. Notable systems developed in response include CAEVO (Chambers et al., 2014), followed by the more recent Cogcomptime (Ning et al., 2018). Both CAEVO and Cogcomptime use several statistical and rule-based methods like event extractors, dependency parsers, semantic role labelers, and time expression identifiers for the task. Our work differs from these systems in both the methodology and desired result in the following ways: i) Instead of using specialized sub-systems, we transform the task into a sequence-to-sequence mapping problem and use a single language model to generate such temporal graphs in an end-to-end fashion from text, subsuming all the intermediate-steps. ii) We develop our system using a corpus of 89,000 documents, which is  300x larger compared to datasets used by CAEVO (36 documents) and Cogcomptime on (276 documents); iii) We remove the noisy events included by CAEVO , but do not limit the extracted events to any specific semantic axis as done by Cogcomptime; and finally, iv) Our method generates graphs where the nodes are not simple verbs but augmented event phrases, containing the subject and the object of each verb. We use CAEVO over Cogcomptime to generate a large-scale corpus for our task and to evaluate our system for the following reasons: i) We found CAEVO to be much more scaleable, a critical feature for our task of annotating close to 100k documents, ii) CAEVO over-generates (and not excludes) verbs from its output, giving us the flexibility to filter out noisy events without inadvertently missing out on any critical events. However, our method makes no as-sumption specific to CAEVO and is adaptable to any other similar system (including Cogcomptime). Temporal relation extraction We note that the problem of temporal graph extraction is different from the more popular task of Temporal relation extraction (Temprel), which deals with classifying the temporal link between two already extracted events. State of the art Temprel systems use neural methods (Ballesteros et al., 2020; Ning et al., 2019b; Goyal and Durrett, 2019; Han et al., 2019; Cheng and Miyao, 2017), but typically use a handful of documents for their development and evaluation. Vashishtha et al. (2019) are a notable exception by using Amazon Mechanical Turks to obtain manual annotations over a larger dataset of 16,000 sentences. We believe that the techniques presented in our work can be applied to scale the corpus used for training Temprel systems. Language Models for Graph Generation Recently, Bosselut et al. (2019) proposed COMET , a system that fine-tunes GPT (Radford et al., 2018) on commonsense knowledge graphs like ATOMIC (Sap et al., 2019) and conceptnet (Speer et al., 2017) for commonsense kb completion. Similar to COMET , we adopt large-scale language models for such a conditional generation of text. However, our task differs from COMET in the complexity of both the conditioning text and generated text: we seek to generate temporal graphs grounded in a document, whereas COMET generates a short event/concept phrase conditioned on a relation and an input event/concept phrase. Madaan et al. (2020) and Rajagopal et al. (2021) aim to generate event influence graphs grounded in a situation. Similar to this work, these methods rely on pre-trained language models to generate informative structures grounded in text. Different from us, these methods break the generation process into a sequence of natural language queries. Each query results in an event node, which are finally assembled into a tree. In contrast, we propose a method to directly generate graphs with arbitrary topology from text. Additionally, the events generated by these methods are not present in text making event event prediction, rather than event extraction as their primary focus. You et al. (2018) formulate graphs as a sequence for learning generative models of synthetic and real-world graphs. Similar to their work, we formulate graph generation as an auto-regressive task. However, our goal is the conditional generation of temporal graphs, and not learning unconditional generative distributions. Finally, inspired by recent trends (Raffel et al., 2019), we do not make any graph specific modifications to the model or the decoding process and formulate the problem as a straightforward sequence-to-sequence mapping task. While our approach does not rely on any particular language model, it would be interesting to see the gains achieved by the much larger GPT 3 (Brown et al., 2020) on the dataset produced by our method. 2 3 Deriving Large-scale Dataset for the Temporal Graph Generation Definitions and Notations: Let G ( V , E ) be a temporal graph associated with a document D , such that vertices V are the events in document D , and the edges E are temporal relations (links) between the events. Every temporal link in E takes the form r ( e q , e t ) where the query event e q and the target event e t are in V , and r is a temporal relation (e.g., before or after). In this work, we undertake two related tasks of increasing complexity: i) Node generation, and ii) Temporal graph generation: Task 1: Node Generation: Let r ( e q , e t ) be an edge in E . Let C r be the set of sentences in the document D that contains the events e q or e t or are adjacent to them. Given a query consisting of C r , r , and e q , generate e t . Task 2: Temporal Graph Generation: Given a document D , generate the corresponding temporal graph G ( E , V ) . 2 Not available for research as of April 2021. Figure 1 illustrates the two tasks. Task 1 is similar to knowledge base completion, except that the output events e q are generated, and not drawn from a fixed set of events. Task 2 is significantly more challenging, requiring the generation of both the structure and semantics of G . The training data for both the tasks consists of tuples { ( x i , y i ) } Ni =1 . For Task 1, x i is the concatenation of the query tokens ( C r , e q , r ) , and y i consists of tokens of event e t . For Task 2, x i is the i th document D i , and y i is the corresponding temporal graph G i . We use the New York Times ( NYT ) Annotated Corpus 3 to derive our dataset of document-graph pairs. The corpus has 1.8 million articles written and published by NYT between 1987 and 2007. Each article is annotated with a hand-assigned list of descriptive terms capturing its subject(s). We filter articles with one of the following descriptors: { bomb, terrorism, murder, ri-ots, hijacking, assassination, kidnapping, arson, vandalism, hate crime, serial murder, manslaughter, extortion } , yielding 89,597 articles, with a total of 2.6 million sentences and 66 million tokens. For each document D , we use CAEVO (Chambers et al., 2014) to extract the dense temporal graph consisting of i) the set of verbs, and ii) the set of temporal relations between the extracted verbs. CAEVO extracts six temporal relations: before, after, includes, is included, simultaneous, and vague. We process each dense graph extracted by CAEVO with a series of pruning and augmentation operations: i) We observed that some of the most frequent verbs extracted by CAEVO were the so-called reporting verbs (Liu et al., 2018), like said , say , and told , which do not contribute to the underlying events. For example, said formed nearly 10% of all the verbs extracted by CAEVO as an event. To remove such noisy events, we remove the five verbs with the lowest inverse document frequencies, as well as an additional set of light and reporting verbs (Liu et al., 2018; Re-casens et al., 2010) 4 ii) To make event annotations richer, we follow (Chambers and Jurafsky, 2008), and prefix and suffix every verb with its 3 https://catalog.ldc.upenn.edu/LDC2008T19 4 The final list of verbs is: i) low idf: said, say, had, made, told, ii) light: appear, be, become, do, have, seem, get, give, go, have, keep, make, put, set, take, iii) reporting: argue, claim, say, suggest, tell. noun-phrase and object, respectively. This augmentation helps in adding a context to each verb, thus making events less ambiguous. For instance, given a sentence: A called B, after which B called C , CAEVO extracts AFTER ( called , called ) . With the proposed augmentation, the relation becomes AFTER ( A called B , B called C ) , clearly differentiating the two different called events. Our notion of events refers to such augmented verbs. Crucially, different from prior work, our system is trained to extract these augmented event phrases. We also drop all the verbs that do not have either a subject or an object. iii) We remove the relations extracted by the statistical sieves if they have a confidence score of less than 0.50 and retain the rule-based relations as those were shown to be extracted with a high precision by Chambers et al. (2014). Finally, we only retain event-event relations (dropping links between verbs and time expressions) and drop the vague relations as they typically do not play any role in improving the understanding of the temporal sequences in a document. As Table 1 shows, pruning noisy verbs and relations yields sparser and more informative graphs. We discovered that the (pruned) graph generated for a given document typically has several subgraphs that are either completely disconnected or have high intra-link density. Further, we found that each of these sub-graphs are grounded in different parts of the document. We exploit this phenomenon to map each sub-graph to its correct context, thus reducing the noise in the data. Relying merely on connectivity for creating subgraphs is still prone to noise, as largely unrelated sub-graphs are often connected via a single event. Instead, we propose a novel approach based on the detection of event communities to divide a graph into sub-graphs, such that the events in a sub-graph are more densely connected to each other. We learn these event communities using the concept of modularity, first introduced by (Newman and Girvan, 2004). We defer the derivation of modularity optimization to the Appendix. Datasets for Task 1 and Task 2 After running the pruning and clustering operations outlined above on 89k documents, we obtain a corpus of over 890,677 text-graph pairs, with an average of 120.31 tokens per document, and 3.33 events and 4.91 edges per graph. These text-graph pairs constitute the training data for Task 2. We derive the data for Task 1 from the original (undivided) 89k graphs (each document-graph pair contributes multiple examples for Task 1). In Task 1 data, nearly 80% of the queries ( C r , e q , r ) had a unique answer e t , and nearly 16% of the queries had two different true e t . We retain examples with multiple true e t in the training data because they help the model learn diverse temporal patterns that connect two events. For fairness, we retain such cases in the test set. Table 2 lists the statistics of the dataset. The splits were created using non-overlapping documents. We use language models to generate each graph as a sequence of tokens conditioned on the document, thus requiring that the graphs are represented as strings. We use DOT language (Gansner et al., 2006) to format each graph as a string. While our method does not rely on any specific graph representation format, we use DOT as it supports a wide variety of graphs and allows augmenting graphs with node, edge, and graph level information. Further, graphs represented in DOT are readily consumed by popular graph libraries like Net-workX (Hagberg et al., 2008), making it possible to use the graphs for several downstream applications. Figure 2 shows an example graph and the corresponding DOT code. The edges are listed in the order in which their constituent nodes appear in the document. This design choice was inspired by our finding that a vast majority of temporal links exist between events that are either in the same or in the adjoining sentence (this phenomenon was also observed by Ning et al. (2019a)). Thus, listing the edges in the order in which they appear in the document adds a simple inductive bias of locality for the auto-regressive attention mechanism, whereby the attention weights slide from left to right as the graph generation proceeds. Additionally, a fixed order makes the problem well defined, as the mapping between a document and a graph becomes deterministic. The training data X for both Tasks 1 and 2 comprises of tuples { ( x i , y i ) } Ni =1 . For task 1 (node generation), x i the concatenation of context, the source, node, and the relation. The target y i consists of the tokens of the target event. For task 2 (graph generation), x i is a document and y i is the corresponding temporal graph represented in DOT . We train a (separate) conditional language model to solve both the tasks. Specifically, given a training corpus of the form { ( x i , y i ) } , we aim to estimate the distribution p  ( y i | x i ) . Given a training example ( x i , y i ) we set u i = x i (cid:107) y i 5 . p  ( u i ) can then be factorized as a sequence of auto-regressive conditional probabilities using the chain rule: p  ( u i ) = (cid:81) nk =1 p ( u i,k | u i,<k ) , where u i,k denotes the k th token of the i th sequence, and u i,<k denotes the sequence of tokens { u 1 , u 2 , ..., u k  1 } . Language models are typically trained by minimizing a cross-entropy loss  log p  ( u i ) over each sequence u i in X . However, the cross-entropy loss captures the joint distribution p  ( x i , y i ) , and is not aligned with our goal of learning conditional distribution p  ( y i | x i ) . To 5 (cid:107) denotes concatenation Method Dataset BLEU MTR RG ACC Seq2Seq TG-Gen (-C) 20.20 14.62 31.95 19.68 Seq2Seq TG-Gen 21.23 16.48 35.54 20.99 GPT -2 TG-Gen (-C) 36.60 25.11 43.07 35.07 GPT -2 TG-Gen 62.53 43.78 69.10 61.35 Seq2Seq TB-Dense (-C) 11.55 9.23 21.87 10.06 Seq2Seq TB-Dense 16.68 12.69 27.75 13.97 GPT -2 TB-Dense (-C) 22.35 15.04 27.73 20.81 GPT -2 TB-Dense 52.21 35.69 57.98 47.91 Table 3: Node Generation (task 1) results. circumvent this, we train our model by masking the loss terms corresponding to the input x i , similar to Bosselut et al. (2019). Let m i be a mask vector for each sequence u i , set to 0 for positions corresponding to x i , and 1 otherwise i.e. m i,j = 1 if j > | x i | , else 0. We combine the mask vector with our factorization of p  ( u i ) to formulate a masked language modeling loss L , which is minimized over the training corpus X to estimate the optimal  : L ( X ) =  | X | (cid:88) i =1 | x i | + | y i | (cid:88) j =1 m i,j  log ( p  ( u i,j | u i,<j )) Note that the formulation of masked loss is opaque to the underlying architecture, and can be implemented with a simple change to the loss function. In practice, we use GPT -2 (Radford et al., 2019) based on transformer architecture (Vaswani et al., 2017) for our implementation. Having trained a p  for each task, we generate a node ( y ) given a query ( x ) (for Task 1), or a graph ( y ) given a document ( x ) (for Task 2) by drawing samples from the appropriate p  ( y | x ) using nucleus sampling (Holtzman et al., 2019). We provide more details of our training procedure and the architecture in the Appendix (C.1). We evaluate our method on two different datasets: i) TG-Gen : Test split of synthetically created dataset (Section 3), and ii) TB-Dense: A mixed-domain corpus, with human-annotated temporal annotations. We create TB-Dense from the test splits of TimeBank-Dense (Cassidy et al., 2014) by applying the same pre-processing operations as we did for TG-Gen. TB-Dense forms a very challenging dataset for our task because of domain mismatch; our system was trained on a corpus of terrorism-related events, whereas TB-Dense includes documents from a wide array of domains, forming a zero-shot evaluation scenario for our method. GPT -2: We use GPT -2 medium (355M parameters) for our experiments with 24-layers, a hidden size of 1024, and 16 self-attention heads. We build on the implementation by Wolf et al. (2019), using the default hyperparameters and a block size input sequence length after tokenization) of 512. For optimization, we use AdamW (Loshchilov and Hutter, 2017) with a learning rate of 5e-5, a batch size of 1, and no learning rate scheduling. We also experimented with a block size of 300 and a batch size of 2. We found the results (presented in the appendix) to be worse, underscoring the importance of using a larger block size for generating larger outputs. We generate samples using nucleus sampling using p = 0 . 9 , and set maximum output length to be 500 for graph generation and 60 foe node generation. All of our experiments were done on a single Nvidia GeForce RTX 2080 Ti. The models were initialized with the pre-trained weights provided by Radford et al. (2019), and fine-tuned for five epochs, with a run-time of 48 hours/epoch for Task 1 and 52 hours/epoch for Task 2. We use the last checkpoint (i.e., at the end of fine-tuning) for all experiments. Despite the higher perplexity on the development set, we found the overall performance of the last checkpoint to be better. We also experimented with GPT -2 without fine-tuning (i.e., by directly using pre-trained weights). The non-finetuned GPT -2 fared poorly for both the tasks across all the metrics, getting a BLEU score of near 0 for Task 1. This dismal performance underscores the importance of fine-tuning on the end task for large-scale pre-trained language models. Finally, we note that our method does not make any model-specific assumption, and can be used with any auto-regressive language model (i.e., a language model that generates a sequence of tokens from left-to-right). We use GPT -2 as a representative for large pre-trained language models. Seq2Seq: We train a bi-directional LSTM (Hochre-iter and Schmidhuber, 1997) based sequence-to-sequence model (Bahdanau et al., 2015) with global attention (Luong et al., 2015) and a hidden size of 500 as a baseline to contrast with GPT -2. The token embeddings initialized using 300-dimensional pre-trained Glove (Pennington et al., 2014). Paragraph: Mr. Grier, a former defensive lineman for the New York Giants who was ordained as a minister in 1986, testified on Dec. 9 that he had visited Mr. Simpson a month earlier Metrics Given a query ( C r , e q , r ) , with C r being the context (sentences containing events e q , e t and their neighboring sentences) and e q as the source event, Task 1 is to generate a target event e t such that r ( e q , e t ) . We format each query as In the context of C , what happens r e q ?. We found formatting the query in natural language to be empirically better. Let  e t be the system generated event. We compare e t vs.  e t using BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2011), and ROUGE (Lin, 2004) 6 , and measure the accuracy ( ACC ) as the fraction of examples where e t =  e t . Results on TG-Gen The results are listed in Table 3. Unsurprisingly, GPT -2 achieves high scores across the metrics showing that it is highly effective in generating correct events. To test the generative capabilities of the models, we perform an ablation by removing the sentence containing the target event e t from C r (indicated with -C ). Removal of context causes a drop in performance for both GPT -2 and Seq2Seq, showing that it is crucial for generating temporal events. However, GPT -2 obtains higher relative gains with context present, indicating that it uses its large architecture and pre-training to use the context more efficiently. GPT -2 also fares better as compared with Seq2Seq in terms of drop in performance for the out-of-domain TB-Dense dataset on metrics like accuracy (  21% vs.  33% ) and BLEU (  16% vs.  21% ), indicating that pre-training makes helps GPT -2 in generalizing across the domains. Human Evaluation To understand the nature of errors, we analyzed 100 randomly sampled incorrect generations. For 53% of the errors, GPT -2 generated a non-salient event which nevertheless 6 Sharma et al. (2017), https://github.com/ Maluuba/nlg-eval had the correct temporal relation with the query. Interestingly, for 10% of the events, we found that GPT -2 fixed the label assigned by CAEVO (Table 4), i.e., e t was incorrect but  e t was correct. 5.4 Task 2: Graph Generation Dataset BLEU MTR RG DOT % Seq2Seq TG-Gen 4.79 15.03 45.95 86.93 GPT -2 TG-Gen 37.77 37.22 64.24 94.47 Seq2Seq TB-Dense 2.61 12.76 28.36 89.31 GPT -2 TB-Dense 26.61 29.49 49.26 92.37 Table 5: Graph string metrics. Metrics Let G i ( V i , E i ) and G i ( V i , E i ) be the true and the generated graphs for an example i in the test corpus. Please recall that our proposed method generates a graph from a given document as a string in DOT . Let y i and  y i be the string representations of the true and generated graphs. We evaluate our generated graphs using three types of metrics: 1. Graph string metrics : To compare y i vs.  y i , we use BLEU , METEOR , and ROUGE , and also measure parse accuracy ( DOT %) as the % of generated graphs  y i which are valid DOT files. 2. Graph structure metrics To compare the structures of the graphs G i vs. G i , we use i) Graph edit distance ( GED ) (Abu-Aisheh et al., 2015) the minimum numbers of edits required to transform the predicted graph to the true graph by ad-dition/removal of an edge/node; ii) Graph isomorphism ( ISO ) (Cordella et al., 2001) a binary measure set to 1 if the graphs are isomorphic (without considering the node or edge attributes); iii) The average graph size ( | V i | , | E i | , | V i | , | E i | ) and the average degree (d ( V ) ). graphs. For every example i , we calculate node-set precision, recall, and F 1 score, and average them over the test set to obtain node precision ( v P ), recall ( v R ), and F 1 ( v F ). We evaluate the predicted edge set using temporal awareness (UzZaman and Allen, 2012; UzZaman et al., 2013). For an example i , we calculate e iP = | E  i  E i + | | E  i | , e iR = | E + i  E i  | | E i  | where symbol + denotes the temporal transitive closure (Allen, 1983) of the edge set. Similarly,  indicates the reduced edge set, obtained by removing all the edges that can be inferred from other edges transitively. The F 1 score e iF 1 is the harmonic mean of e iP and e iR , and these metrics are averaged over the test set to obtain the temporal awareness precision ( e P ), recall ( e R ), and F 1 score ( e F 1 ). Intuitively, the node metrics judge the quality of generated events in the graph, and the edge metrics evaluate the corresponding temporal relations. Results Tables 5, 7, and 6 present results for graph generation, and we discuss them next. GPT -2 vs. Seq2Seq GPT -2 outperforms Seq2Seq on all the metrics by a large margin in both fine-tuned (TG-Gen) and zero-shot settings (TB-Dense). GPT -2 generated graphs are closer to the true graphs in size and topology, as shown by lower edit distance and a higher rate of isomorphism in Table 7. Both the systems achieve high parsing rates ( DOT %), with GPT -2 generating valid DOT files 94.6% of the time. The high parsing rates are expected, as even simpler architectures like vanilla RNNs have been shown to generate syntactically valid complex structures like LA TEXdocuments with ease (Karpathy, 2015). GPT -2 vs. CAEVO We compare the graphs generated by GPT -2 with those extracted by CAEVO (Chambers et al., 2014) 7 from the TB-Dense documents. We remove all the vague 7 https://github.com/nchambers/caevo Top 10 Verbs: found, killed, began, called, want, took, came, used, trying, asked Randomly Sampled Verbs: shooting, caused, accused, took, conceived, visit, vowing, play, withdraw, seems Table 8: Verbs in GPT -2 generated graphs. edges and the light verbs from the output of CAEVO for a fair comparison. Please recall that CAEVO is the tool we used for creating the training data for our method. Further, CAEVO was trained using TB-Dense, while GPT -2 was not. Thus, CAEVO forms an upper bound over the performance of GPT -2. The results in Tables 6 and 7 show that despite these challenges, GPT -2 performs strongly across a wide range of metrics, including GED , ISO , and temporal awareness. Comparing the node-set metrics, we see that GPT -2 leads CAEVO by over eight precision points ( v P ), but loses on recall ( v R ) as CAEVO extracts nearly every verb in the document as a potential event. On temporal awareness (edge-metrics), GPT -2 outperforms both CAEVO and Seq2Seq in terms of average precision score e P and achieves a competitive e F 1 score. These results have an important implication: they show that our method can best or match a pipeline of specialized systems given reasonable amounts of training data for temporal graph extraction. CAEVO involves several sub-modules to perform part-of-speech tagging, dependency parsing, event extraction, and several statistical and rule-based systems for temporal extraction. In contrast, our method involves no hand-curated features, is trained end-to-end (single GPT -2), and can be easily scaled to new datasets. Node extraction and Edge Extraction The node-set metrics in Table 6 shows that GPT -2 avoids generating noisy events (high P ), and extracts salient events (high R ). This is confirmed by manual analysis, done by randomly sampling 100 graphs from the GPT -2 generated graphs and isolating the main verb in each node (Table 8). We provide several examples of generated graphs in the Appendix. We note from Table 6 that the relative difference between the e F 1 scores for GPT -2 and Seq2Seq (25.22 vs. 4.70) is larger than the relative difference between their v F 1 scores (66.34 vs. 28.11), showing that edge-extraction is the more challenging task which allows GPT -2 to take full advantage of its powerful architecture. We also observe that edge extraction ( e F 1 ) is highly sensitive Query ( C,e q ,r ) e t Explanation The suspected car bombings...turning busy streets...Which event happened before the suspected car bombings? to node extraction ( v F 1 ); for GPT -2, a 27% drop in v F 1 (66.34 on TG-Gen vs. 44.97 on TB-Dense) causes a 68% drop in e F 1 (25.22 on TG-Gen vs. 7.96 on TB-Dense). As each node is connected to multiple edges on average (Table 7), missing a node during the generation process might lead to multiple edges being omitted, thus affecting edge extraction metrics disproportionately. A benefit of our approach of using a pre-trained language model is that it can be used to generate an answer for open-ended temporal questions. Recently, Ning et al. (2020) introduced Torque, a temporal reading-comprehension dataset. Several questions in Torque have no answers, as they concern a time scope not covered by the passage (the question is about events not mentioned in the passage). We test the ability of our system for generating plausible answers for such questions out of the box (i.e., without training on Torque). Given a (passage, question) pair, we create a query ( C, e q , r ) , where C is the passage, and e q and r are the query event and temporal relation in the question. We then use our GPT -2 based model for node-generation trained without context and generate an answer e t for the given query. A human-judge rated the answers generated for 100 such questions for plausibility, rating each answer as being plausible , somewhat plausible , or incorrect . For each answer rated as either plausible or somewhat plausible , the human-judge wrote a short explanation to provide a rationale for the plausibility of the generated event. Out of the 100 questions, the human-judge rated 22 of the generated answers as plausible and ten as somewhat plausible, showing the promise of our method on this challenging task (Table 9). Current methods for generating event-level temporal graphs are developed with relatively small amounts of hand-labeled data. On the other hand, the possibility of using pre-trained language models for this task has not received sufficient attention. This paper addresses this open challenge by first developing a data generation pipeline that uses existing IE / NLP /clustering techniques for automated acquisition of a large corpus of document-graph pairs, and by proposing a new formulation of the graph generation task as a sequence-to-sequence mapping task, allowing us to leverage and fine-tune pre-trained language models. Our experiments strongly support the effectiveness of the proposed approach, which significantly outperforms strong baselines. We plan to explore techniques for adapting large-scale language models on unseen domains and at multiple granularity levels in the future. Thanks to Nathanael Chambers and Dheeraj Rajagopal for the helpful discussion, and to the anonymous reviewers for their constructive feedback. This material is based on research sponsored in part by the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
objective abstain abstain objective method result result result abstain other abstain abstain abstain abstain abstain abstain method abstain objective objective abstain method method result result objective result objective result other other other objective method objective method method method abstain method abstain method other other method other method objective other method abstain other objective other other method objective method abstain other other abstain other other other other other other other other other other other other method other other method method other other other method abstain other method other other other other other other other other abstain method abstain abstain abstain other method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain other method abstain method method method method abstain abstain abstain abstain abstain method method method method method abstain method method method method abstain method method method abstain abstain method method abstain method abstain method abstain abstain abstain method other method abstain abstain method abstain method method abstain abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain abstain method other abstain method abstain abstain method abstain method abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective objective objective other other other other