Event extraction (EE) has considerably ben-efited from pre-trained language models (PLMs) by fine-tuning. However, existing pre-training methods have not involved modeling event characteristics, resulting in the developed EE models cannot take full advantage of large-scale unsupervised data. To this end, we propose CLEVE, a contrastive pre-training framework for EE to better learn event knowledge from large unsupervised data and their semantic structures (e.g. AMR) obtained with automatic parsers. CLEVE contains a text encoder to learn event semantics and a graph encoder to learn event structures respectively. Specifically, the text encoder learns event semantic representations by self-supervised contrastive learning to represent the words of the same events closer than those unrelated words; the graph encoder learns event structure representations by graph contrastive pre-training on parsed event-related semantic structures. The two complementary representations then work together to improve both the conventional supervised EE and the unsupervised liberal EE, which requires jointly extracting events and discovering event schemata without any annotated data. Experiments on ACE 2005 and MAVEN datasets show that CLEVE achieves significant improvements, especially in the challenging unsupervised setting. The source code and pre-trained checkpoints can be obtained from https://github.com/THU-KEG/CLEVE . Event extraction (EE) is a long-standing crucial information extraction task, which aims at extracting event structures from unstructured text. As illustrated in Figure 1, it contains event detection task to identify event triggers (the word attack) and  indicates equal contribution  Correspondence to L.Hou (houlei@tsinghua.edu.cn) CNN's Kelly Wallace reports on today 's attack in Netanya . classify event types ( Attack ), as well as event argument extraction task to identify entities serving as event arguments (today and Netanya) and classify their argument roles ( Time-within and Place ) (Ahn, 2006). By explicitly capturing the event structure in the text, EE can benefit various downstream tasks such as information retrieval (Glavas and Snajder, 2014) and knowledge base population (Ji and Grishman, 2011). Existing EE methods mainly follow the supervised-learning paradigm to train advanced neural networks (Chen et al., 2015; Nguyen et al., 2016; Nguyen and Grishman, 2018) with human-annotated datasets and pre-defined event schemata. These methods work well in lots of public benchmarks such as ACE 2005 (Walker et al., 2006) and TAC KBP (Ellis et al., 2016), yet they still suffer from data scarcity and limited generalizability. Since annotating event data and defining event schemata are especially expensive and labor-intensive, existing EE datasets typically only contain thousands of instances and cover limited event types. Thus they are inadequate to train large neural models (Wang et al., 2020) and develop methods that can generalize to continually-emerging new event types (Huang and Ji, 2020). Inspired by the success of recent pre-trained language models (PLMs) for NLP tasks, some pioneering work (Wang et al., 2019a; Wadden et al., 2019) attempts to fine-tune general PLMs (e.g, BERT (Devlin et al., 2019)) for EE. Benefiting from the strong general language understanding ability learnt from large-scale unsupervised data, these PLM-based methods have achieved state-of-the-art performance in various public benchmarks. Although leveraging unsupervised data with pretraining has gradually become a consensus for EE and NLP community, there still lacks a pre-training method orienting event modeling to take full advantage of rich event knowledge lying in large-scale unsupervised data. The key challenge here is to find reasonable self-supervised signals (Chen et al., 2017; Wang et al., 2019a) for the diverse semantics and complex structures of events. Fortunately, previous work (Aguilar et al., 2014; Huang et al., 2016) has suggested that sentence semantic structures, such as abstract meaning representation (AMR) (Banarescu et al., 2013), contain broad and diverse semantic and structure information relating to events. As shown in Figure 1, the parsed AMR structure covers not only the annotated event ( Attack ) but also the event that is not defined in the ACE 2005 schema ( Report ). Considering the fact that the AMR structures of large-scale unsupervised data can be easily obtained with automatic parsers (Wang et al., 2015), we propose CLEVE, an event-oriented contrastive pre-training framework utilizing AMR structures to build self-supervision signals. CLEVE consists of two components, including a text encoder to learn event semantics and a graph encoder to learn event structure information. Specifically, to learn effective event semantic representations, we employ a PLM as the text encoder and encourage the representations of the word pairs connected by the ARG , time , location edges in AMR structures to be closer in the semantic space than other unrelated words, since these pairs usually refer to the trigger-argument pairs of the same events (as shown in Figure 1) (Huang et al., 2016). This is done by contrastive learning with the connected word pairs as positive samples and unrelated words as negative samples. Moreover, considering event structures are also helpful in extracting events (Lai et al., 2020) and generalizing to new event schemata (Huang et al., 2018), we need to learn transferable event structure representations. Hence we further introduce a graph neural network (GNN) as the graph encoder to encode AMR structures as structure representations. The graph encoder is contrastively pre-trained on the parsed AMR structures of large unsupervised corpora with AMR subgraph discrimination as the objective. By fine-tuning the two pre-trained models on downstream EE datasets and jointly using the two representations, CLEVE can benefit the conventional supervised EE suffering from data scarcity. Meanwhile, the pre-trained representations can also directly help extract events and discover new event schemata without any known event schema or annotated instances, leading to better generalizability. This is a challenging unsupervised setting named liberal event extraction (Huang et al., 2016). Experiments on the widely-used ACE 2005 and the large MAVEN datasets indicate that CLEVE can achieve significant improvements in both settings. Event Extraction. Most of the existing EE works follow the supervised learning paradigm. Traditional EE methods (Ji and Grishman, 2008; Gupta and Ji, 2009; Li et al., 2013) rely on manually-crafted features to extract events. In recent years, the neural models become mainstream, which automatically learn effective features with neural networks, including convolutional neural networks (Nguyen and Grishman, 2015; Chen et al., 2015), recurrent neural networks (Nguyen et al., 2016), graph convolutional networks (Nguyen and Grishman, 2018; Lai et al., 2020). With the recent successes of BERT (Devlin et al., 2019), PLMs have also been used for EE (Wang et al., 2019a,b; Yang et al., 2019; Wadden et al., 2019; Tong et al., 2020). Although achieving remarkable performance in benchmarks such as ACE 2005 (Walker et al., 2006) and similar datasets (Ellis et al., 2015, 2016; Getman et al., 2017; Wang et al., 2020), these PLM-based works solely focus on better fine-tuning rather than pre-training for EE. In this paper, we study pre-training to better utilize rich event knowledge in large-scale unsupervised data. Event Schema Induction. Supervised EE models cannot generalize to continually-emerging new event types and argument roles. To this end, Chambers and Jurafsky (2011) explore to induce event schemata from raw text by unsupervised clustering. Following works introduce more features like coreference chains (Chambers, 2013) and entities (Nguyen et al., 2015; Sha et al., 2016). Recently, Huang and Ji (2020) move to the semi-Parsed AMR Graphs Trigger-Argument Pair Discrimination CNN's Kelly Wallace report-01 attack-01 today Netanya ARG0 ARG1 ARG1 time Unsupervised Corpora CNN's Kelly Wallace reports on today's attack in Netanya. supervised setting allowing to use annotated data of known types. Following Huang et al. (2016), we evaluate the generalizability of CLEVE in the most challenging unsupervised liberal setting, which requires to induce event schemata and extract event instances only from raw text at the same time. Contrastive Learning. Contrastive learning was initiated by Hadsell et al. (2006) following an intuitive motivation to learn similar representations for neighboors and distinct representations for non-neighbors, and is further widely used for self-supervised representation learning in various domains, such as computer vision (Wu et al., 2018; Oord et al., 2018; Hjelm et al., 2019; Chen et al., 2020; He et al., 2020) and graph (Qiu et al., 2020; You et al., 2020; Zhu et al., 2020). In the context of NLP, many established representation learning works can be viewed as contrastive learning methods, such as Word2Vec (Mikolov et al., 2013), BERT (Devlin et al., 2019; Kong et al., 2020) and ELECTRA (Clark et al., 2020). Similar to this work, contrastive learning is also widely-used to help specific tasks, including question answering (Yeh and Chen, 2019), discourse modeling (Iter et al., 2020), natural language inference (Cui et al., 2020) and relation extraction (Peng et al., 2020). The overall CLEVE framework is illustrated in Figure 2. As shown in the illustration, our contrastive pre-training framework CLEVE consists of two components: event semantic pre-training and event structure pre-training, of which details are introduced in Section 3.2 and Section 3.3, respectively. At the beginning of this section, we first introduce the required preprocessing in Section 3.1, including the AMR parsing and how we modify the parsed AMR structures for our pre-training. CLEVE relies on AMR structures (Banarescu et al., 2013) to build broad and diverse self-supervision signals for learning event knowledge from large-scale unsupervised corpora. To do this, we use automatic AMR parsers (Wang et al., 2015; Xu et al., 2020) to parse the sentences in unsupervised corpora into AMR structures. Each AMR structure is a directed acyclic graph with concepts as nodes and semantic relations as edges. Moreover, each node typically only corresponds to at most one word, and a multi-word entity will be represented as a list of nodes connected with name and op (conjunction operator) edges. Considering pretraining entity representations will naturally benefits event argument extraction, we merge these lists into single nodes representing multi-word entities (like the CNN's Kelly Wallace in Figure 1) during both event semantic and structure pre-training. Formally, given a sentence s in unsupervised corpora, we obtain its AMR graph g s = (V s , E s ) after AMR parsing, where V s is the node set after word merging and E s denotes the edge set. E s = { ( u, v, r ) | ( u, v )  V s  V s , r  R} , where R is the set of defined semantic relation types. To model diverse event semantics in large unsupervised corpora and learn contextualized event semantic representations, we adopt a PLM as the text encoder and train it with the objective to discriminate various trigger-argument pairs. Like most PLMs, we adopt a multi-layer Transformer (Vaswani et al., 2017) as the text encoder since its strong representation capacity. Given a sentence s = { w 1 , w 2 , . . . , w n } containing n tokens, we feed it into the multi-layer Transformer and use the last layer's hidden vectors as token representations. Moreover, a node v  V s may correspond to a multi-token text span in s and we need a unified representation for the node in pre-training. As suggested by Baldini Soares et al. (2019), we insert two special markers [E1] and [/E1] at the beginning and ending of the span, respectively. Then we use the hidden vector for [E1] as the span representation x v of the node v . And we use different marker pairs for different nodes. As our event semantic pre-training focuses on modeling event semantics, we start our pre-training from a well-trained general PLM to obtain general language understanding abilities. CLEVE is agnostic to the model architecture and can use any general PLM, like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). We design trigger-argument pair discrimination as our contrastive pre-training task for event semantic pre-training. The basic idea is to learn closer representations for the words in the same events than the unrelated words. We note that the words connected by ARG , time and location edges in AMR structures are quite similar to the trigger-argument pairs in events (Huang et al., 2016, 2018), i.e., the key words evoking events and the entities participating events. For example, in Figure 1, Ne-tanya is an argument for the attack event, while the disconnected CNN's Kelly Wallace is not. With this observation, we can use these special word pairs as positive trigger-argument samples and train the text encoder to discriminate them from negative samples, so that the encoder can learn to model event semantics without human annotation. Let R p = { ARG , time , location } and P s = { ( u, v ) | ( u, v, r )  E s , r  R p } denotes the set of positive trigger-argument pairs in sentence s . For a specific positive pair ( t, a )  P s , as shown in Figure 2, we construct its corresponding negative samples with trigger replacement and argument replacement. Specifically, in the trigger replacement, we construct m t number of negative pairs by randomly sample m t number of negative triggers  t  V s and combine them with the positive argument a . A negative trigger  t must do not have a directed ARG , time or location edge with a , i.e., (cid:64) ( t, a, r )  E s , r  R p . Similarly, we construct m a more negative pairs by randomly sample m a number of negative arguments  a  V s satisfying (cid:64) ( t,  a, r )  E s , r  R p . As the example in Figure 2, (attack, reports) is a valid negative sample for the positive sample (attack, Ne-tanya), but (attack, today's) is not valid since there is a (attack, today's, time ) edge. To learn to discriminate the positive trigger-argument pair from the negative pairs and so that model event semantics, we define the training objective for a positive pair ( t, a ) as a cross-entropy loss of classifying the positive pair correctly: L t,a =  x (cid:62) t W x a + log (cid:16) exp (cid:16) x (cid:62) t W x a (cid:17) + m t (cid:88) i =1 exp (cid:16) x (cid:62)  t i W x a (cid:17) + m a (cid:88) j =1 exp (cid:16) x (cid:62) t W x  a j (cid:17) (cid:17) , (1) where m t , m a are hyper-parameters for negative sampling, and W is a trainable matrix learning the similarity metric. We adopt the cross-entropy loss here since it is more effective than other contrastive loss forms (Oord et al., 2018; Chen et al., 2020). Then we obtain the overall training objective for event semantic pre-training by summing up the losses of all the positive pairs of all sentences s in the mini batch B s : L sem (  ) = (cid:88) s  B s (cid:88) ( t,a )  P s L t,a , (2) where  denotes the trainable parameters, including the text encoder and W . Previous work has shown that event-related structures are helpful in extracting new events (Lai et al., 2020) as well as discovering and generalizing to new event schemata (Huang et al., 2016, 2018; Huang and Ji, 2020). Hence we conduct event structure pre-training on a GNN as graph encoder to learn transferable event-related structure representations with recent advances in graph contrastive pre-training (Qiu et al., 2020; You et al., 2020; Zhu et al., 2020). In CLEVE, we utilize a GNN to encode the AMR (sub)graph to extract the event structure information of the text. Given a graph g , the graph encoder represents it with an graph embedding g = G ( g, { x v } ) , where G (  ) is the graph encoder and { x v } denotes the initial node representations fed into the graph encoder. CLEVE is agnostic to specific model architectures of the graph encoder. Here we use a state-of-the-art GNN model, Graph Isomorphism Network (Xu et al., 2019), as our graph encoder for its strong representation ability. We use the corresponding text span representations { x v } produced by our pre-trained text encoder (introduced in Section 3.2) as the initial node representations for both pre-training and inference of the graph encoder. This node initialization also implicitly aligns the semantic spaces of event semantic and structure representations in CLEVE, so that can make them cooperate better. To learn transferable event structure representations, we design the AMR subgraph discrimination task for event structure pre-training. The basic idea is to learn similar representations for the subgraphs sampled from the same AMR graph by discriminating them from subgraphs sampled from other AMR graphs (Qiu et al., 2020). Given a batch of m AMR graphs { g 1 , g 2 , . . . , g m } , each graph corresponds to a sentence in unsupervised corpora. For the i -th graph g i , we randomly sample two subgraphs from it to get a positive pair a 2 i  1 and a 2 i . And all the subgraphs sampled from the other AMR graphs in the mini-batch serve as negative samples. Like in Figure 2, the two green (w/ attack) subgraphs are a positive pair while the other two subgraphs sampled from the purple (w/ solider) graph are negative samples. Here we use the subgraph sampling strategy introduced by Qiu et al. (2020), whose details are shown in Appendix C. Similar to event semantic pre-training, we adopt the graph encoder to represent the samples a i = G ( a i , x v ) and define the training objective as: L str (  ) =  m (cid:88) i =1 log exp (cid:0) a (cid:62) 2 i  1 a 2 i (cid:1) (cid:80) 2 mj =1 1 [ j (cid:54) =2 i  1] exp (cid:0) a (cid:62) 2 i  1 a j (cid:1) , (3) where 1 [ j (cid:54) =2 i  1]  { 0 , 1 } is an indicator function evaluating to 1 iff j (cid:54) = 2 i  1 and  is the trainable parameters of graph encoder. Before the detailed experiments, we introduce the pre-training setup of CLEVE in implementation. We adopt the New York Times Corpus (NYT) 1 (Sandhaus, 2008) as the unsupervised pretraining corpora for CLEVE. It contains over 1 . 8 million articles written and published by the New York Times between January 1, 1987, and June 19, 2007. We only use its raw text and obtain the AMR structures with a state-of-the-art AMR parser (Xu et al., 2020). We choose NYT corpus because (1) it is large and diverse, covering a wide range of event semantics, and (2) its text domain is similar to our principal evaluation dataset ACE 2005, which is helpful (Gururangan et al., 2020). To prevent data leakage, we remove all the articles shown up in ACE 2005 from the NYT corpus during pretraining. Moreover, we also study the effect of different AMR parsers and pre-training corpora in Section 5.2 and Section 5.3, respectively. For the text encoder, we use the same model architecture as RoBERTa (Liu et al., 2019), which is with 24 layers, 1024 hidden dimensions and 16 attention heads, and we start our event semantic pre-training from the released checkpoint 2 . For the graph encoder, we adopt a graph isomorphism network (Xu et al., 2019) with 5 layers and 64 hidden dimensions, and pre-train it from scratch. For the detailed hyperparameters for pre-training and fine-tuning, please refer to Appendix D. 4.2 Adaptation of CLEVE As our work focuses on pre-training rather than fine-tuning for EE, we use straightforward and common techniques to adapt pre-trained CLEVE to downstream EE tasks. In the supervised setting, we adopt dynamic multi-pooling mechanism (Chen et al., 2015; Wang et al., 2019a,b) for the text encoder and encode the corresponding local subgraphs with the graph encoder. Then we concate-1 https://catalog.ldc.upenn.edu/ LDC2008T19 2 https://github.com/pytorch/fairseq ED EAE Metric P R F1 P R F1 JointBeam 73 . nate the two representations as features and fine-tune CLEVE on supervised datasets. In the unsupervised liberal setting, we follow the overall pipeline of Huang et al. (2016) and directly use the representations produced by pre-trained CLEVE as the required trigger/argument semantic representations and event structure representations. For the details, please refer to Appendix A. 4.3 Supervised EE Dataset and Evaluation We evaluate our models on the most widely-used ACE 2005 English subset (Walker et al., 2006) and the newly-constructed large-scale MAVEN (Wang et al., 2020) dataset. ACE 2005 contains 599 English documents, which are annotated with 8 event types, 33 subtypes, and 35 argument roles. MAVEN contains 4 , 480 documents and 168 event types, which can only evaluate event detection. We split ACE 2005 following previous EE work (Liao and Grishman, 2010; Li et al., 2013; Chen et al., 2015) and use the official split for MAVEN. EE performance is evaluated with the performance of two subtasks: Event Detection (ED) and Event Argument Extraction (EAE). We report the precision (P), recall (R) and F1 scores as evaluation results, among which F1 is the most comprehensive metric. Baselines We fine-tune our pre-trained CLEVE and set the original RoBERTa without our event semantic pre-training as an important baseline. To do ablation studies, we evaluate two variants of CLEVE on both datasets: the w/o semantic model adopts a vanilla RoBERTa without event semantic pre-training as the text encoder, and the w/o structure only uses the event semantic representations ED Metric P R F1 DMCNN 66 . On ACE 2005, we set two more variants to investigate the effectiveness of CLEVE. The on ACE (golden) model is pre-trained with the golden trigger-argument pairs and event structures of ACE 2005 training set instead of the AMR structures of NYT. Similarly, the on ACE (AMR) model is pre-trained with the parsed AMR structures of ACE 2005 training set. We also compare CLEVE with various baselines, including: (1) feature-based method, the top-performing JointBeam (Li et al., 2013); (2) vanilla neural model DMCNN (Chen et al., 2015); (3) the model incorporating syntactic knowledge, dbRNN (Sha et al., 2018); (4) state-of-the-art models on ED and EAE respectively, including GatedGCN (Lai et al., 2020) and SemSynGTN (Pouran Ben Veyseh et al., 2020); (5) a state-of-the-art EE model RCEE ER (Liu et al., 2020), which tackle EE with machine reading comprehension (MRC) techniques. The last four models adopt PLMs to learn representations. On MAVEN, we compare CLEVE with the official ED baselines set by Wang et al. (2020), including DMCNN (Chen et al., 2015), BiLSTM (Hochreiter and Schmidhuber, 1997), BiL-STM+CRF , MOGANED (Yan et al., 2019), DMBERT (Wang et al., 2019a), BERT+CRF . The evaluation results are shown in Table 1 and Table 2. We can observe that: (1) CLEVE achieves significant improvements to its basic model RoBERTa on both ACE 2005 and MAVEN. The p-values under the t-test are 4  10  8 , 2  10  8 and 6  10  4 for ED on ACE 2005, EAE on ACE 2005, and ED on MAVEN, respectively. It also outperforms or achieves comparable results with ED EAE Metric (B-Cubed) P R F1 P R F1 LiberalEE 55 . all the baselines, including those using dependency parsing information (dbRNN, GatedGCN, SemSynGTN and MOGANED). This demonstrates the effectiveness of our proposed contrastive pre-training method and AMR semantic structure. It is noteworthy that RCEE ER outperforms our method in EAE since its special advantages brought by reformulating EE as an MRC task to utilize sophisticated MRC methods and large annotated external MRC data. Considering that our method is essentially a pre-training method learning better event-oriented representations, CLEVE and RCEE ER can naturally work together to improve EE further. (2) The ablation studies (comparisons between CLEVE and its w/o semantic or structure representations variants) indicate that both event semantic pre-training and event structure pre-training is essential to our method. (3) From the comparisons between CLEVE and its variants on ACE (golden) and ACE (AMR), we can see that the AMR parsing inevitably brings data noise compared to golden annotations, which results in a performance drop. However, this gap can be easily made up by the benefits of introducing large unsupervised data with pre-training. In the unsupervised setting, we evaluate CLEVE on ACE 2005 and MAVEN with both objective automatic metrics and human evaluation. For the automatic evaluation, we adopt the extrinsic clustering evaluation metrics: B-Cubed Metrics (Bagga and Baldwin, 1998), including B-Cubed precision, recall and F1. The B-Cubed metrics evaluate the quality of cluster results by comparing them to golden standard annotations and have been shown to be effective (Amigo et al., 2009). For the human evaluation, we invite an expert to check the outputs ED Metric (B-Cubed) P R F1 RoBERTa 32 . of the models to evaluate whether the extracted events are complete and correctly clustered as well as whether all the events in text are discovered. Baselines We compare CLEVE with reproduced LiberalEE (Huang et al., 2016), RoBERTa and RoBERTa+VGAE . RoBERTa here adopts the original RoBERTa (Liu et al., 2019) without event semantic pre-training to produce semantic representations for trigger and argument candidates in the same way as CLEVE, and encode the whole sentences to use the sentence embeddings (embed-dings of the starting token <s> ) as the needed event structure representations. RoBERTa+VGAE additionally adopts an unsupervised model Variational Graph Auto-Encoder (VGAE) (Kipf and Welling, 2016) to encode the AMR structures as event structure representations. RoBERTa+VGAE shares similar model architectures with CLEVE but is without our pre-training. Specially, for fair comparisons with LiberalEE, all the models in the unsupervised experiments adopt the same CAMR (Wang et al., 2015) as the AMR parser, including CLEVE pretraining. Moreover, we also study CLEVE variants as in the supervised setting. The w/o semantic variant replaces the CLEVE text encoder with a RoBERTa without event structure pre-training. The w/o structure variant only uses CLEVE text encoder in a similar way as RoBERTa . The on ACE (AMR) model is pre-trained with the parsed AMR structures of ACE test set. As shown in Huang et al. (2016), the AMR parsing is significantly superior to dependency parsing and frame semantic parsing on the unsupervised liberal event extraction task, hence we do not include baselines using other sentence structures in the experiments. The automatic evaluation results are shown in Table 3 and Table 4. As the human evaluation is laborious and expensive, we only do human ED EAE Metric (Human) P R F1 P R F1 LiberalEE 51 . evaluations for CLEVE and the most competitive baseline LiberalEE on ACE 2005, and the results are shown in Table 5. We can observe that: (1) CLEVE significantly outperforms all the baselines, which shows its superiority in both extracting event instances and discovering event schemata. (2) RoBERTa ignores the structure information. Although RoBERTa+VAGE encodes event structures with VGAE, the semantic representations of RoBERTa and the structure representations of VGAE are distinct and thus cannot work together well. Hence the two models even under-perform LiberalEE, while the two representations of CLEVE can collaborate well to improve liberal EE. (3) In the ablation studies, the discarding of event structure pre-training results in a much more significant performance drop than in the supervised setting, which indicates event structures are essential to discovering new event schemata. In this section, we study how the benefits of pretraining change along with the available supervised data size. We compare the ED performance on MAVEN of CLEVE, RoBERTa and a non-pre-training model BiLSTM+CRF when trained on different proportions of randomly-sampled MAVEN training data in Figure 3. We can see that the im-AMR 1.0 ACE 2005 MAVEN Parsing ED EAE ED Wang et al. (2015) 62 . provements of CLEVE compared to RoBERTa and the pre-training models compared to the non-pre-training model are generally larger when less supervised data available. It indicates that CLEVE is especially helpful for low-resource EE tasks, which is common since the expensive event annotation. CLEVE relies on automatic AMR parsers to build self-supervision signals for large unsupervised data. Intuitively, the performance of AMR parsers will influence CLEVE performance. To analyze the effect of different AMR parsing performance, we compare supervised EE results of CLEVE models using the established CAMR (Wang et al., 2016) and a new state-of-the-art parser (Xu et al., 2020) during pre-training in Table 6. We can see that a better AMR parser intuitively brings better EE performance, but the improvements are not so significant as the corresponding AMR performance improvement, which indicates that CLEVE is generally robust to the errors in AMR parsing. Pre-training on similar text domains may further improve performance on corresponding downstream tasks (Gururangan et al., 2020; Gu et al., 2020). To analyze this effect, we evaluate the supervised EE performance of CLEVE pre-trained on NYT and English Wikipedia in Table 7. We can see pre-training on a similar domain (NYT for ACE 2005, Wikipedia for MAVEN) surely benefits CLEVE on corresponding datasets. On ACE 2005, although Wikipedia is 2 . 28 times as large as NYT, CLEVE pre-trained on it underperforms CLEVE pre-trained on NYT (both in the news do-main). Moreover, we can see the in-domain benefits mainly come from the event semantics rather than structures in CLEVE framework (from the comparisons between the w/o semantic and w/o structure results). It suggests that we can develop domain adaptation techniques focusing on semantics for CLEVE, and we leave it to future work. In this paper, we propose CLEVE, a contrastive pre-training framework for event extraction to utilize the rich event knowledge lying in large unsupervised data. Experiments on two real-world datasets show that CLEVE can achieve significant improvements in both supervised and unsupervised liberal settings. In the future, we will (1) explore other kinds of semantic structures like the frame semantics and (2) attempt to overcome the noise in unsupervised data brought by the semantic parsers. This work is supported by the National Natural Science Foundation of China Key Project (NSFC No. U1736204), grants from Beijing Academy of Artificial Intelligence (BAAI2019ZD0502) and the Institute for Guo Qiang, Tsinghua University (2019GQB0003). This work is also supported by the Pattern Recognition Center, WeChat AI, Tencent Inc. We thank Lifu Huang for his help on the unsupervised experiments and the anonymous reviewers for their insightful comments. We discuss the ethical considerations and broader impact of the proposed CLEVE method in this section: (1) Intellectual property . NYT and ACE 2005 datasets are obtained from the linguistic data consortium (LDC), and are both licensed to be used for research. MAVEN is publicly shared under the CC BY-SA 4.0 license 3 . The Wikipedia corpus is obtained from the Wikimedia dump 4 , which is 3 https://creativecommons.org/licenses/ by-sa/4.0/ 4 https://dumps.wikimedia.org/ shared under the CC BY-SA 3.0 license 5 . The invited expert is fairly paid according to agreed working hours. (2) Intended use . CLEVE improves event extraction in both supervised and unsupervised settings, i.e., better extract structural events from diverse raw text. The extracted events then help people to get information conveniently and can be used to build a wide range of application systems like information retrieval (Glavas and Snajder, 2014) and knowledge base population (Ji and Grish-man, 2011). As extracting events is fundamental to various applications, the failure cases and potential bias in EE methods also have a significant negative impact. We encourage the community to put more effort into analyzing and mitigating the bias in EE systems. Considering CLEVE does not model people's characteristics, we believe CLEVE will not bring significant additional bias. (3) Misuse risk. Although all the datasets used in this paper are public and licensed, there is a risk to use CLEVE methods on private data without authorization for interests. We encourage the regulators to make efforts to mitigate this risk. (4) Energy and carbon costs. To estimate the energy and carbon costs, we present the computing platform and running time of our experiments in Appendix E for reference. We will also release the pre-trained checkpoints to avoid the additional carbon costs of potential users. We encourage the users to try model compression techniques like distillation and quantization in deployment to reduce carbon costs.
abstain abstain objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method abstain abstain abstain abstain abstain other other other other other other method other other other other other other method other other other method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain method method abstain abstain method abstain abstain abstain method abstain abstain method abstain method abstain other method other abstain method method abstain abstain method method abstain method abstain abstain method abstain abstain other method method abstain abstain method method method method method method method method other abstain method method abstain abstain method abstain method method method abstain abstain abstain method abstain method abstain result abstain abstain abstain abstain result result abstain result abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective abstain objective other other other abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain method method abstain method method method