Generating new events given context with correlated ones plays a crucial role in many event-centric reasoning tasks. Existing works either limit their scope to specific scenarios or overlook event-level correlations. In this paper, we propose to pre-train a general Correlation-aware context-to-Event Transformer (ClarET) for event-centric reasoning. To achieve this, we propose three novel event-centric objectives, i.e., whole event recovering, contrastive event-correlation encoding and prompt-based event locating, which highlight event-level correlations with effective training. The proposed ClarET is applicable to a wide range of event-centric reasoning scenarios, considering its versatility of (i) event-correlation types (e.g., causal, temporal, contrast), (ii) application formulations (i.e., generation and classification), and (iii) reasoning types (e.g., abductive, counterfactual and ending reasoning). Empirical fine-tuning results, as well as zeroand few-shot learning, on 9 benchmarks (5 generation and 4 classification tasks covering 4 reasoning types with diverse event correlations), verify its effectiveness and generalization ability. An event', usually a text span composed of a predicate and its arguments (Zhang et al., 2020b), is a fine-grained semantic unit to describe the state of entities/things (e.g., He looks very worried ) and how they act (e.g., I grab his arms ). Understanding events and modeling their correlations are fundamental to many reasoning tasks (Bhagavatula et al., 2020; Qin et al., 2019), e.g., abductive reasoning, story ending classification and generation, counterfactual reasoning, script reasoning. For instance, in the left example of Figure 1, to generate the missing event [E] in the given context, it is essential to understand that there are four events ( it tries  Work is done during internship at Microsoft. the knob ', [E] ,  the creature starts pounding on the door ', and  (the creature) to break it down '), and then predict [E] based on the other three events and its correlations to them (i.e., the contrast relation indicated by  but ' and the causal relation by  so '). Event-aware reasoning has gained much attention and achieved promising success in recent years (Lv et al., 2020; Ding et al., 2019). However, many algorithms are designed to solve only some specific tasks. For example, Qin et al. (2020) propose to improve unsupervised decoding for counterfactual and abductive reasoning; Huang et al. (2021) and Guan et al. (2019) advance story ending generation via incremental encoding and multi-level graph convolutional networks. Although these works show effectiveness in corresponding applications, they are limited to specific scenarios, and cannot generalize well to a broad scope of reasoning. Meanwhile, some pioneering works follow a recently arising paradigm to conduct event-based pretraining for those downstream reasoning tasks (Yu et al., 2020; Han et al., 2020a; Lin et al., 2020; Zhou et al., 2021b). However, these solutions have their own limitations: COMeT (Hwang et al., 2021) learns event correlations from a human-curated knowledge graph and thus limits its scalability. Han et al. (2020a) and Lin et al. (2020) only model temporal relations and cannot be expanded to other relations (e.g., causal, contrast). EventBERT (Zhou et al., 2021b) is proposed for event-based classifi-2559 cations and is thus inapplicable to generation tasks. In this work, we propose a general pre-training framework for event-centric reasoning by learning a C orre la tion-awa r e context-toE vent T ransformer (ClarET) from an event-rich text corpus. We propose three novel self-supervised objectives, dubbed as whole event recovering (WER), contrastive event-correlation encoding and prompt-based event locating, respectively. The first one aims to capture event correlation by recovering a whole event from its masked context. The second one enhances the representation of the masked event in WER by contrasting it with the gold event against the negative ones. The last one is a simplified WER task by providing hints in its prompt and thus facilitates effective learning for WER. ClarET explicitly models event correlations and contributes to various scenarios. From one aspect, it covers a variety of correlation types (e.g., causal, temporal, contrast) attributed to correlation type-agnostic objectives. From another aspect, it is applicable to both generation and classification task formulations by its unified structure. Lastly, it highlights event-level correlations and thus is more effective for diverse event-centric tasks, e.g., abductive, counterfactual and ending reasoning. To evaluate ClarET, we compare it with strong baselines on 9 diverse benchmarks. While ClarET is continually pre-trained from BART (Lewis et al., 2020) with very limited extra resources, i.e., training on a small subset of BART-used corpus (i.e., 200M out of 2.2T tokens) within 90 GPU hours (only 0.13% of 70,000h BART pre-training), it achieves state-of-the-art (SoTA) performance on all 5 generation benchmarks. It also outperforms all unified models on 4 classification benchmarks and achieves competitive, or even better, accuracy to strong discriminative baselines. We further exhibit that the ClarET provides a good initialization for downstream tasks by zeroand few-shot learning. Unified Pre-trained Model. A recent trend is to pre-train unified (a.k.a. universal or general) models to boost downstream generation and classification tasks, rather than masked language modeling (MLM) only. GPT (Radford et al., 2019) is based on auto-regressive language modeling but incompetent in classifications due to unidirectional contextualizing. To remedy this, BART (Lewis et al., 2020) trains seq2seq models as a text denoising autoencoder with mask-infilling, etc; UniLM (Dong et al., 2019) designs advanced self-attention masks in Transformer, leading to a partially autoregressive MLM; GLM (Du et al., 2021) proposes an auto-regressive blank-filling objective based on Transformer, achieved by bi-/uni-directional attention and 2D positional encoding. T5 (Raffel et al., 2020) pre-trains a text-to-text Transformer to recover the masked part of input by decoding. All these general-purpose pre-trained models focus on relatively short-span masking in random, whereas we focus on masking a whole semantic unit (i.e., event) and propose novel training objectives to circumvent problems in long-span event decoding. Besides, they are also vulnerable to pretrain-finetune inconsistency, leading to inferior event-centric performance. Task-specific Models for Event Reasoning. Many recent works present task-specific neural models for various event-centric reasoning types, including (1) abductive reasoning (Ji et al., 2020; Dong et al., 2021; Zhu et al., 2020), (2) counterfactual reasoning (Qin et al., 2019, 2020), (3) ending reasoning (Guan et al., 2019; Wang and Wan, 2019; Yao et al., 2019; Huang et al., 2021; Guan et al., 2020; Wang et al., 2017; Li et al., 2018; Ding et al., 2019; Zhou et al., 2021c; Chaturvedi et al., 2017; Srinivasan et al., 2018), (4) incoherence reasoning (Mori et al., 2020). However, these methods are designed for the specific reasoning scenarios based on task-specific models so hardly generalize to other scenarios. In contrast, we aim to pre-train a general event-centric model for generalizing to various scenarios. Event-centric Pre-training. With similar scopes, many works focus on event-centric pre-training to promote event-related tasks as event' is a self-contained semantic unit and also an entry of commonsense reasoning. One paradigm is to pre-train on corpora without human-labeling. Some methods focus on more specific aspects of events and their correlations. DEER (Han et al., 2020b) performs temporal and event masking predictions for temporal relations. Lin et al. (2021) propose to recover a temporally-disordered or event-missing sequence for temporal and causal relations. Wang et al. (2021) use AMR structure to design contrastive objectives for the event detection task. However, they are not general enough to various event reasoning tasks. In contrast, CoCoLM (Yu et al., 2020) learns 2560 an event-level MLM to generalize more. EventBERT (Zhou et al., 2021b) states the ineffectiveness of event-level MLM and exploits hard negatives via contrasting, contributing much to downstream multi-choice tasks. However, these methods are only competent in discriminative tasks. The other paradigm is based on supervised pre-training on similar tasks and then performs knowledge transfer, e.g., COMeT (Hwang et al., 2021), UnifiedQA (Khashabi et al., 2020) and UNICORN (Lourie et al., 2021), but they require human-curated data. Event-rich Corpus. Although raw corpora are viewed as off-the-shelf pre-training resources, a key question is how to mine event-rich examples. Here, event-rich' denotes that each example contains various events and entails adequate contexts to support event reasoning via either explicit or implicit event-correlation. This is crucial to learning event-correlations and reducing unnecessary overheads. Except for human-curated resources (e.g., ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017)), event-rich corpora are also mined via automatic schemes. ASER (Zhang et al., 2020b) builds an event-based graph, where each node is an event extracted from a text and the relation of an event pair is predicted by a PDTB model. In contrast, EventBERT (Zhou et al., 2021b) operates on pure text so filters out correlation-scarce contexts and extracts verb-rooted events. Besides, it offers event sampling methods for hard negatives. We adopt this data processing method as both pure-text examples and hard negatives are prerequisites of generic and robust pre-training. In this work, we directly adopt event-rich data mining and negative sampling methods from Zhou et al. (2021b) but focus our contributions on enlarging application scope of event-centric tasks and overcoming challenges raised in the new scope. Event-rich Data Mining. To mine event-rich data from raw corpus, we employ a story corpus, BOOKCORPUS (Zhu et al., 2015), and take a two-step procedural (i.e.,  filter ' and  extraction '). It filters out correlation-scarce paragraphs according to existence of connectives (i.e., discourse relation keywords, e.g., however , while ). Then, it highlights the event spans in the filtered paragraphs by extracting verb-rooted sub-trees in dependency trees of the paragraphs. With a filtered paragraph x , we build each example as ( x, e ) where e is an event mention in x . We obtain 200M tokens (out of 1B in BOOKCORPUS ) in 3.9M filtered paragraphs. For clear notations , we denote a text piece as a lower case letter (e.g., e ). It is tokenized into a sequence as a bold (e.g., e = [ e 1 , e 2 , . . . ] ), where a letter w/ subscript t is the t -th token in the sequence. Negative Event Sampling. Following Zhou et al. (2021b), we build a pool of events from the whole corpus and then retrieve negative events by three heuristic schemes. Given an event e in ( x, e ) , we sample its negative event,  e , in light of lexicon-based (20% time), PoS-based (60% time) or in-domain (20% time) retrieval. Consequently, given an event e , we sample M negative events, i.e., {  e } Mi =1 . Figure 1 (right) shows an integrated instance ( x, e, {  e } Mi =1 ) of the event-rich corpus 1 . We first present whole event recovering as a backbone pre-training objective in 3.2.1. After identifying incompetence of the simple backbone, we propose two other objectives in 3.2.2 and 3.2.3. An overview of the objectives is shown in Figure 2. For the objective of whole event recovering (WER), it is straightforward to leverage an encoder-decoder structure, where a masked context is passed into the encoder to generate the missing part by decoding. Specifically, given an event e in a paragraph x , we mask out e from x at the encoder side and then generate e at the decoder side, i.e., p ( e | x / { e } ;  ) = (cid:89) t p ( e t | e <t , x / { e } ;  ) , (1) where  denotes parameters and x / { e } denotes replacing e in x with one special token [M] . We estimate Eq. (1) by the Transformer sequence-to-sequence (seq2seq) structure (Vaswani et al., 2017). First, we apply the Transformer encoder to x / { m } for contextual embeddings for all tokens in x / { m } : H ( enc ) =Trans-Enc( x / { e } ;  ( enc ) )  R d  n , (2) where n is the number of tokens in x / { e } . y t = Trans-Dec( e <t , H ( enc ) ;  ( dec ) )  R |V| , (3) where V denotes token vocabulary and  y t is the predicted categorical distribution over V . Lastly, the training objective is defined as a maximum likelihood estimation. Its loss function is written as L ( wer ) =  (cid:88) ( x,e ) 1 | e | (cid:88) | e | t =1 log  y t [ y = e t ] , (4) where  y t [ y = e t ] ' denotes fetching the probability of the t -step gold token e t  e from  y t . This objective is similar to span recovering schema (Raffel et al., 2020; Joshi et al., 2020) but differs in that (i) each masked span is an event, i.e., an integrated semantic unit, so much longer (up to 22 tokens and see Figure 4 for length distribu-tion), and (ii) only one event is masked out from the context to facilitate event-correlation modeling between the event and its contexts. Intuitively, the success of Eq. (1) requires to capture correlations between the masked event and remaining contexts but two major problems arise due to WER with long event-level masking spans: (1) Implicit Event-correlation: The model recovers an event based solely on token-level concurrence as in a conditional language model (e.g., T5 and BART), regardless of the rich event-level correlations between the events in context x / { e } and the masked event e . Such a correlation-implicit model would achieve inferior performance on downstream event-centric correlation reasoning tasks. (2) Learning Difficulty: As the masked event is an integrated, self-contained, semantic unit, it is difficult for the conditional generation model to recover the whole event due to a lack of local contexts. As a result, the model cannot effectively learn from the long masked spans, which has been empirically proved in autoencoding MLM models. To alleviate the two problems above, we propose two other novel self-supervised objectives in the following. Briefly, we present contrastive event-correlation encoding to enhance correlations between contexts and events, and prompt-based event locating to reduce generation difficulty. For the implicit event-correlation problem, an intuitive solution is to explicitly highlight the correlation from the masked context to the missing event at the encoder side. To achieve this, we resort to contrastive learning to enhance the encoder-side representation of the masked event by contrasting it with the embedding of the gold event mention e against those of negative ones  e . Particularly, we first derive the embedding of e and  e independently via the Transformer encoder in Eq.(2), i.e., c = Pool(Trans-Enc( [CLS] + e ;  ( enc ) )) , (5)  c = Pool(Trans-Enc( [CLS] +  e ;  ( enc ) )) , (6) where [CLS] is a special token prefixed to each event mention, and Pool(  ) denotes using the contextual embedding of [CLS] to represent the whole event. Then, we enhance h [m] , the contextual representation of [M] in x / { e } from H ( enc ) in Eq.(2), by contrasting it with c against  c , i.e., L ( cee ) =max(0 , + d ( h [m] , c )  d ( h [m] ,  c )) , (7) where d (  ,  ) denotes a distance metric of two vectors, which is Euclidean distance in this work. As a result, the encoder-side correlation-aware representation h [m] also offers a straightforward pathway to transmit event-level information to decoding so mitigates the learning difficulty to some extent. As for learning difficulty problem, we also propose a prompt-based event locating objective to reduce generative difficulty by providing hints in the prompt. The basic idea is to simplify WER objective as an extractive generation task to locate and copy a candidate/hint from the prompt, which aims at improving learning effectiveness. To this end, we present two prompt-based generation schemas in the following. Correct Event Selection. Inspired by advances of prompt-based multi-choice question answering, we present correct event selection schema to select the gold event e against negative ones {  e } Mi =1 based on the contexts x / { e } . Given an event-masked paragraph x / { e } suffixed with several candidate events {  e } Mi =1 containing the gold masked one e , it aims to generate the masked event e back, i.e.,  x ( ces ) = x / { e } + Options: where [ e 1 , e 2 , . . . ] is a random permutation of [ e, {  e } Mi =1 ] in case of position bias. We use a random permutation as all candidates are assigned with distinct position embeddings during contextualizing, and a fixed permutation of gold events will result in a learning shortcut (position bias) to degrade the model. Thus, similar to Eq.(1), we can define its formula as p ( e |  x ( ces ) ;  ) . Wrong Event Tagging. The other schema is wrong event tagging to find the wrong event in a corrupted paragraph, similar to incoherence reasoning. Thus, we re-write the encoder input as  x ( wet ) = x / { e } & {  e } + Event: [M] is wrong , where x / { e } & {  e } denotes replacing the gold event e in x with a negative  e  {  e } Mi =1 . Thus, we can define the formula of this objective as p ( e |  x ( wet ) ;  ) . Based on the two formulas above, we define the prompt-based event locating objective as L ( pel ) = (cid:88) ( x,e )  1 | e | (cid:88) t log p ( e t | e <t ,  x ( ces ) ;  )  1 |  e | (cid:88) t log p ( e t |  e <t ,  x ( wet ) ;  ) , (8) where  = {  ( enc ) ,  ( dec ) } ,  e is sampled in {  e } Mi =1 . 3.3 Model Pre-training and Fine-tuning Self-supervised Pre-training. The final loss to pre-train our ClarET is a linear combination of the three losses above from Eq.(4, 7, 8), i.e., L = L ( wer ) + L ( cee ) + L ( pel ) . Supervised Downstream Fine-tuning. For generation tasks, we simply leverage the formula in Eq.(1) to establish fine-tuning objectives. For discriminative (e.g., multi-choice) tasks, we can either formulate all tasks into generation as in GPT/T5 or fine-tune with classifying heads as in BART. With pilot experiments, we found the latter one can achieve better performance and adopted it. While we adopt the same data processing in EventBERT (Zhou et al., 2021b) and share a similar motivation to learn an event-centric pre-trained model, we expand the scope from  discriminative-only ' in EventBERT into  unified ' by our context-to-event Transformer for a broad spectrum of scenarios. Such an expansion is non-trivial since new challenges arise in the unified formulation. Compared to the inefficient  event-backfilling and contextualizing ' paradigm in EventBERT, our model can explicitly and effectively learn event-level correlations between contexts and events by our novel contrastive and prompt-based objectives. Moreover, COMeT (Bosselut et al., 2019; Hwang et al., 2021) is also a conditional generation model but focuses on triple-level commonsense reasoning  given ( head event , relation ) to generate tail events , whose motivation, however, is orthogonal to ours. Therefore, we focus on a different motivation or scope, not to mention evaluation formulations. This section begins with descriptions of downstream datasets and experimental setups. Downstream Datasets. We conduct extensive evaluations on 9 datasets for 9 downstream tasks, i.e., 5 generation and 4 classification tasks. Generation tasks include abductive commonsense reasoning on ART (  NLG) (Bhagavatula et al., 2020), counterfactual story generation on TIMETRAVEL (Qin et al., 2019), story ending generation (Guan et al., 2019), commonsense story generation (Guan et al., 2020), and event process completion on APSI (Zhang et al., 2020a). Classification tasks include script reasoning on MCNC (Li et al., 2018), abductive commonsense reasoning on ART (  NLI) (Bhagavatula et al., 2020), narrative incoherence detection on ROCStories (Mori et al., 2020), and story cloze test (Mostafazadeh et al., 2016). Please refer to Appendix C for their details. Pre-training Setups. Instead of learning from scratch, we perform continual pre-training from BART-large (Lewis et al., 2020) due to limited computation resources. The batch size and number of training steps are 1152 and 160k. The model is trained by Adam (Kingma and Ba, 2015) w/ learning rate of 1e-5 and warmup proportion of 0.03. The gradient clip, dropout rate and weight decay are 1.0, 0.1 and 0.01. Notably, (i) BOOKCORPUS 2563 Abductive C.S. Reasoning CounterfactualStory Story Ending Generation C.S. Story Generation Event Process Completion Size B-4 R-L BERT B-4 R-L BERT B-1 B-2 B-1 B-2 B-1 B-2 Selected task-specific models with competitive performance GRF (Ji et al., 2020) -11.62 34.62 -----IE+MSA (Guan et al., 2019) ----24.40 7.80 --Plan&Write (Yao et al., 2019) ----24.40 8.40 30.80 12.60 -Fine-tuning with pre-trained unified (generative) model GPT2-S (Radford et al., 2019) 124M 2.23 22.83 48.74 69.27 65.72 60.53 39.23 13.08 32.20 14.10 35.25 11.75 GPT2-M (Radford et al., 2019) 335M --75.71 72.72 62.39 --45.43 14.81 BART (Lewis et al., 2020) 400M 16.47 38.73 56.36 82.91 76.44 79.50 54.22 18.07 54.22 18.07 56.25 18.75 GLM (Du et al., 2021) 335M 7.79 25.54 54.85 75.81 70.03 68.23 57.04 18.45 57.04 18.45 57.34 19.11 ClarET (ours) 400M 17.67 41.04 57.31 87.18 80.74 81.48 57.47 19.16 57.47 19.16 58.88 19.74 Table 1: Fine-tuning results on five generation benchmark datasets. has already been used by BART pre-training and our data processing is based on heuristics without human-curated resources; (ii) Our continual pretraining only needs 90 GPU hours on 200M tokens, i.e., 0.13% of BART that consumes 70K hours on 2.2T tokens (see Appendix B.1). Hence, ClarET with zero newly introduced corpus and relatively negligible computing overhead makes great lifts and preserves fair comparisons with baselines. Fine-tuning Setups. For finetuning, we train the model with an Adam w/ learning rate of 1e-5 and warmup proportion of 0.06. The dropout rate, batch size and weight decay are 0.1, 32 and 0.01. For generative downstream tasks, we take BLEUN (BN ) (Papineni et al., 2002), ROUGE-L (R-L) (Lin, 2004) and BERTScore (BERT) (Zhang et al., 2020c) as the evaluation metrics, while the accuracy (ACC) is taken for classification tasks. Each fine-tuning runs with seeds 2, 10 and 1234, and we evaluate the best dev model on the test set. Fine-tuning for Generation. As shown in Table 1, our proposed ClarET achieves SoTA performance across all generation tasks. For instance, ClarET increases the ROUGE-L score by 2.3 ab-solute value for abductive reasoning. The supe-rior performance of ClarET on the benchmarks demonstrates that it can model event-level corre-2564 Method B-4 R-L BERT GPT (Qin et al., 2019) 1.25 18.26 59.50 GPT2-S (Qin et al., 2019) 1.28 20.27 59.62 GPT2-M (Qin et al., 2019) 1.51 19.41 60.17 Zero-Shot-Ranked (Qin et al., 2020) 2.26 25.81 60.07 BART-large (Lewis et al., 2020) 7.08 30.60 61.58 DELOREAN (Qin et al., 2020) 21.35 40.73 63.36 ClarET (ours) 23.75 43.03 63.93 Table 3: Zero-shot results on generative Counterfactual Story. lation more effectively via few steps of continual pre-training and provide a general solution for a variety of event-centric correlation reasoning tasks. Fine-tuning for Classification. Table 2 lists results on 4 classification tasks. We find ClarET performs better than all task-specific models and unified pre-trained models with 2%-4% improvement. It achieves competitive accuracy to strong discriminative models, e.g., the gap between ClarET and EventBERT is  0.15 for narrative incoherence detection and story cloze test. However, EventBERT is a RoBERTa-based competitor using the identical pre-training corpus. Its pre-training follows event-backfilling and contextualizing (similar to multi-choice QA), which has a small gap to downstream classification tasks for strong performance but brings two drawbacks. Firstly, its pre-training is slow due to repeat contextualizing over paragraphs, leading to 5 . 6  longer GPU hours than ours. In addition, its discriminative paradigm limits it specifically to classifications, regardless of wide generation tasks. The results show ClarET is on par with the discriminative-only EventBERT on classifications. This is non-trivial given the large formulation gap between our generative pretraining objectives and downstream multi-choice-style classification tasks, and attributed to our effective event-correlation learning. In summary, these results show ClarET serves as a unified pre-trained model for event-centric generation and classification tasks. Zero-shot Learning. It is essential to verify if the targeted information was learned and retained by a pre-trained model. Compared to MLM, our generative recovering model is inherently applicable to event-centric multi-choice and generative formulations. For generation tasks , we apply Eq.(1) to generate answers. As shown in Table 3, ClarET achieves the best performance and outperforms DE Method ACC (%) Random 20.00 RoBERTa-large (Zhou et al., 2021b) 20.09 DeBERTa-xlarge (Zhou et al., 2021b) 20.31 BART-large (Lewis et al., 2020) 21.72 EventBERT (Zhou et al., 2021b) 30.79 ClarET (ours) 32.15 Table 4: Zero-shot results on discriminative Script Reasoning. LOREAN (which adapts auto-regression for counterfactual reasoning). For classification tasks , we apply Eq.(1) to each option for its perplexity and select the option with minimum. As shown in Table 4, ClarET surpasses previous models and beats the discriminative-only event-centric model, EventBERT. Besides, the general-purpose pre-trained models perform nearly random guesses due to their incompetence in long-span event discrimination. Few-shot Learning. Since our model reduces pretrain-finetune inconsistency for event-centric tasks and provides a good initialization for downstream fine-tuning, it is also interesting to see few-shot performance by scaling down training data. As shown in Figure 3, ClarET achieves similar performance to strong baselines with only 10%-30% of training data for fine-tuning. Ablation study. To measure the contribution of each objective to the final fine-tuning results, we conduct an ablation study on both generation and classification in Table 5. The first two ablations drop the two prompt schemas respectively in prompt-based event locating objective of Eq.(8), which verifies the effectiveness of reducing task difficulty. Then, the third ablation removes contrastive event-correlation encoding and shows a substantial drop, which verifies the significance of explicit event-correlation learning. Next, we keep only the prompt-based event locating objective to make our model a prompt-learning discrim-2565 Method Gen-CS Cls-SR B-4 R-L ACC ClarET (full, pre-trained by inative model (sharing more close methodology with EventBERT), however leading to a dramatic decrease. Lastly, when removing all the objectives, our model degenerates to BART-large. Comparison with Larger Model. A trend of pretraining models follows the law of larger models for better performance' but a crucial research question is how to perform competitively with fewer computation resources'. To answer, we show extra fine-tuning results on the five generation datasets in Table 7 to compare our ClarET (400M parameters) with T5-large (770M) and T5-base (220M). It is observed (i) with 3  scale, T5-large notably outperforms T5-base to support the above law and (ii) with almost half model size, our ClarET performs very competitively to T5-large (even better on 3 out of 5 tasks), verifying the significance of our objectives towards event-related knowledge. Difficulty of Event Generation. To exhibit the learning difficulty in pre-training (as stated in 3.2.1) and the effectiveness of our novel learning objectives, we conduct another ablation setting in Table 6. It is observed that ClarET achieves better event-level perplexity (ePPL), verifying the two novel objectives promote event generations and reduce difficulty of decoding. check if ClarET is more competitive on longer-span event generation, we compare it with BART-large and T5-base/-large by   log ' of Eq.(1). Different from recovering paradigm of others, we follow the denoising paradigm to implement BART and calculate its score by considering the masked part in decoding. Figure 4 shows that (1) Line Chart: the gap between ClarET and the others becomes larger with event length increasing as the general-purpose models only consider short-span masking in pretraining, leading to inferior event generation; and (2) Bar Chart: as for data distribution, although a majority of data falls into the 6-8 bin, there are still many examples with event length greater than nine. Natural Language Understanding (NLU). Our basic model, BART-large, is presented for general NLU tasks. To exhibit our minor event-centric continual pre-training would not interfere its NLU ability, we conduct fine-tuning experiments on GLUE benchmark (Wang et al., 2019) as in Figure 5. It is observed that, although slightly surpassed by the discriminative RoBERTa model, fine-tuning BART and ClarET achieve very comparable results, which verifies ClarET's retention of NLU capability. Case Study. As the first case in Figure 6, we conduct a case study on generative abductive reasoning task, where the fine-tuned ClarET generates an event semantically close to the gold reference, but the BART does not. BART only generates a part of the answer but ignores the event-correlations from  They were impressed with my phone ', while ClarET completely captures the correlations in the 2566 Figure 5: Fine-tuning results on GLUE dev, which verifies ClarET retains BART's natural language understanding ability. Context: I went to the store to buy a phone. [E] They were impressed with my phone. Reference of the Gold Event [E] : I bought the latest model of the phone I wanted, and showed it to my friends. Generation by ClarET: I bought a new phone and showed it to my friends. (BLEU-4: 34) Generation by BART: I bought a new phone. (BLEU-4: 0) Context: Cora was starting her job as a kindergarten teacher. [E] At the end of the day, they all told her how much they liked her! Reference of the Gold Event [E] : Cora was nervous, but knew the students were nervous too, so she tried to be extra friendly. Generation by ClarET: Cora spent the whole day with her students. (BLEU-4: 0) Error Analysis and Limitation. The second case in Figure 6 shows that our ClarET is ineffective when the gold event is very complicated. In detail, the model focus only on  at the end of the day ' to generate  ... spent the whole day ... ' but ignore very subtle contexts, e.g.,  starting her job ... teacher ' and  they liked her '. To expand, we found a problem in long-event decoding by pilot experiments. As shown in Figure 7, it is observed that the gap of token-level perplexity between ClarET and WER-only gradually diminishes. This is because the subsequent tokens in an event can be generated on the basis of previous generations on the decoder side, rather than context-aware representations from the encoder side. While a long span is masked, the model can see previous tokens in an event (i.e., e <t ) in decoding and incline to perform the t -th prediction based on e <t but not x / { e } , especially with a larger t . As a result, the model would cheat' in the generation but learn decoder-side language modeling rather than context-aware representations. In the future, we will exploit this problem. Besides, due to computation resources, we choose the model size with 400M and continual 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Position (top-percentage) of a token 0 1 2 3 T o k e n p e r p l e x i t y ClarET WER-Only Figure 7: Token-level perplexity w.r.t tokens' percentage positions in events on held-out dev set. We present a novel correlation-aware context-to-event Transformer to self-supervisedly learn event-correlation knowledge from text corpus and benefit various event-centric reasoning scenarios. Besides SoTA fine-tuning results on 5 generation and 4 classification tasks, we conduct zero-/few-shot learning and extensive ablation studies to exhibit our model's effectiveness. Lastly, we find our model is competitive to a twice larger general-purpose model, reduces learning difficulty for event generation, and retains NLU ability from its basic model. Although this work learns context-to-event knowledge, our self-supervised objectives are applicable to other semantically-meaningful text units besides events. For example, text units can be entities and concepts to learn relational and commonsense knowledge, which can benefit more downstream tasks. This work does not involve any sensitive data, but only public unlabeled corpora, i.e., BookCorpus (Zhu et al., 2015) pre-processed by Zhou et al. (2021b), and crowd-sourced datasets released in previous works, including ART (Bhagavatula et al., 2020), TIMETRAVEL (Qin et al., 2019), APSI (Zhang et al., 2020a), MCNC (Li et al., 2018), ROCStories (Mori et al., 2020).
abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain method other abstain method other other other other other objective other other other other objective other other other other other other other other other other other other other other other other other other other other method abstain abstain method abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain method abstain other abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method abstain abstain method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain other objective result objective objective abstain other