Within the context of event modeling and understanding, we propose a new method for neural sequence modeling that takes partially-observed sequences of discrete, external knowledge into account. We construct a sequential neural variational autoencoder, which uses Gumbel-Softmax reparametrization within a carefully defined encoder, to allow for successful backpropagation during training. The core idea is to allow semi-supervised external discrete knowledge to guide , but not restrict, the variational latent parameters during training. Our experiments indicate that our approach not only outperforms multiple baselines and the state-of-the-art in narrative script induction, but also converges more quickly. Event scripts are a classic way of summarizing events, participants, and other relevant information as a way of analyzing complex situations (Schank and Abelson, 1977). To learn these scripts we must be able to group similar-events together, learn common patterns/sequences of events, and learn to represent an event's arguments (Minsky, 1974). While continuous embeddings can be learned for events and their arguments (Ferraro et al., 2017; Weber et al., 2018a), the direct inclusion of more structured, discrete knowledge is helpful in learning event representations (Ferraro and Van Durme, 2016). Obtaining fully accurate structured knowledge can be difficult, so when the external knowledge is neither sufficiently reliable nor present, a natural question arises: how can our models use the knowledge that is present? Generative probabilistic models provide a framework for doing so: external knowledge is a random variable, which can be observed or latent, and the data/observations are generated (explained) from it. Knowledge that is discrete, sequential, or both Crush occurred at station <latexit sha1_base64="7Hk7rU2krBpGatMqW9ME3zVmSls=">AAAB6nicdVDLSgNBEOyNrxhfUY9eBoPgaZlNzK65Bb14jGhiIFnC7GQ2GTL7YGZWCCGf4MWDIl79Im/+jZOHoKIFDUVVN91dQSq40hh/WLmV1bX1jfxmYWt7Z3evuH/QUkkmKWvSRCSyHRDFBI9ZU3MtWDuVjESBYHfB6HLm390zqXgS3+pxyvyIDGIeckq0kW7CXrlXLGHb8bDr1BC2K27FrWFDPK9cdTFybDxHCZZo9Irv3X5Cs4jFmgqiVMfBqfYnRGpOBZsWupliKaEjMmAdQ2MSMeVP5qdO0YlR+ihMpKlYo7n6fWJCIqXGUWA6I6KH6rc3E//yOpkOz/0Jj9NMs5guFoWZQDpBs79Rn0tGtRgbQqjk5lZEh0QSqk06BRPC16fof9Iq207VxtdnpfrFMo48HMExnIIDHtThChrQBAoDeIAneLaE9Wi9WK+L1py1nDmEH7DePgFj543h</latexit> f 2 <latexit sha1_base64="mGIOp0nJe8b55O7mrOeWy9IWt7s=">AAAB6nicdVDJSgNBEK2JW4xb1KOXxiB4GnomCSa3oBePEc0CyRB6Oj1Jk56F7h4hDPkELx4U8eoXefNv7CyCij4oeLxXRVU9PxFcaYw/rNza+sbmVn67sLO7t39QPDxqqziVlLVoLGLZ9YligkespbkWrJtIRkJfsI4/uZr7nXsmFY+jOz1NmBeSUcQDTok20m0wKA+KJWzX67hSqSJsV7HrujVDcNmt1R3k2HiBEqzQHBTf+8OYpiGLNBVEqZ6DE+1lRGpOBZsV+qliCaETMmI9QyMSMuVli1Nn6MwoQxTE0lSk0UL9PpGRUKlp6JvOkOix+u3Nxb+8XqqDmpfxKEk1i+hyUZAKpGM0/xsNuWRUi6khhEpubkV0TCSh2qRTMCF8fYr+J23Xdqo2vqmUGperOPJwAqdwDg5cQAOuoQktoDCCB3iCZ0tYj9aL9bpszVmrmWP4AevtE16Djd0=</latexit> f 3 Train collided with train Killed passengers and injured <latexit sha1_base64="yIL6ypp3fIwMcYUAM5v+oQdlkmk=">AAAB+3icdVDLSgMxFM3UV62vWpdugkVwNWTGFttd0Y3gpoJ9QDuUTJq2oZkHyR1pGforblwo4tYfceffmD4EFT1w4XDOvcm9x4+l0EDIh5VZW9/Y3Mpu53Z29/YP8oeFpo4SxXiDRTJSbZ9qLkXIGyBA8nasOA18yVv++Grut+650iIK72Aacy+gw1AMBKNgpF6+0OkCn4Bm6Y2Q5pHhzOvli8SuVkmpVMbELhPXdSuGkHO3UnWwY5MFimiFei//3u1HLAl4CExSrTsOicFLqQLBJJ/luonmMWVjOuQdQ0MacO2li91n+NQofTyIlKkQ8EL9PpHSQOtp4JvOgMJI//bm4l9eJ4FBxUtFGCfAQ7b8aJBIDBGeB4H7QnEGcmoIZUqYXTEbUUUZmLhyJoSvS/H/pOnaTtkmt6Vi7XIVRxYdoxN0hhx0gWroGtVRAzE0QQ/oCT1bM+vRerFel60ZazVzhH7AevsEDtaVHQ==</latexit> [ Killing ] Figure 1: An overview of event modeling, where the observed events (black text) are generated via a sequence of semi-observed random variables. In this paper, we provide a successful approach for incorporating partially-observed, discrete, sequential external knowledge in a neural generative model. We specifically examine event sequence modeling augmented by semantic frames . Frames (Minsky, 1974, i.a.) are a semantic representation designed to capture the common and general knowledge we have about events, situations, and things. They have been effective in providing crucial information for modeling and understanding the meaning of events (Peng and Roth, 2016; Ferraro et al., 2017; Padia et al., 2018; Zhang et al., 2020). Though we focus on semantic frames as our source of external knowledge, we argue this work is applicable to other similar types of knowledge. We examine the problem of modeling observed event tuples as a partially observed sequence of semantic frames . Consider the following three events, preceded by their corresponding bracketed frames, from Fig. 1: [ EVENT ] crash occurred at station. frames can help predict the word killed in the third event; the frames summarize the events and can be used as guidance for the latent variables to represent the data. On the other hand, words like crash , station and killed from the first and third events play a role in predicting [ IMPACT ] in the second event. Overall, to successfully represent events, beyond capturing the event to event connections, we propose to consider all the information from the frames to events and frames to frames. In this work, we study the effect of tying discrete, sequential latent variables to partially-observable, noisy (imperfect) semantic frames. Like Weber et al. (2018b), our semi-supervised model is a bidirectional auto-encoder, with a structured collection of latent variables separating the encoder and decoder, and attention mechanisms on both the encoder and decoder. Rather than applying vector quantization, we adopt a Gumbel-Softmax (Jang et al., 2017) ancestral sampling method to easily switch between the observed frames and latent ones, where we inject the observed frame information on the Gumbel-Softmax parameters before sampling. Overall, our contributions are:  We demonstrate how to learn a VAE that contains sequential, discrete, and partially-observed latent variables. We show that adding partially-observed, external, semantic frame knowledge to our structured, neural generative model leads to improvements over the current state-of-the-art on recent core event modeling tasks. Our approach leads to faster training convergence. We show that our semi-supervised model, though developed as a generative model, can effectively predict the labels that it may not observe. Additionally, we find that our model outperforms a discriminatively trained model with full supervision. Our work builds on both event modeling and latent generative models. In this section, we outline relevant background and related work. Generative latent variable models learn a mapping from the low-dimensional hidden variables f to the observed data points x , where the hidden representation captures the high-level information to where  represents the model parameters. Since in practice maximizing the log-likelihood is intractable, we approximate the posterior by defining q ( f | x ;  ) and maximize the ELBO (Kingma and Welling, 2013) as a surrogate objective: L , = E q ( f | x ;  ) log p ( x , f ;  ) q ( f | x ;  ) . In this paper, we are interested in studying a spe-cific case; the input x is a sequence of T tokens, we have M sequential discrete latent variables z = { z m } Mm =1 , where each z m takes F discrete values. While there have been effective proposals for unsupervised optimization of  and  , we focus on learning partially observed sequences of these variables. That is, we assume that in the training phase some values are observed while others are latent. We incorporate this partially observed, external knowledge to the  parameters to guide the inference. The inferred latent variables later will be used to reconstruct to the tokens. Kingma et al. (2014) generalized VAEs to the semi-supervised setup, but they assume that the dataset can be split into observed and unobserved samples and they have defined separate loss functions for each case; in our work, we allow portions of a sequence to be latent. Teng et al. (2020) characterized the semi-supervised VAEs via sparse latent variables; see Mousavi et al. (2019) for an in-depth study of additional sparse models. Of the approaches that have been developed for handling discrete latent variables in a neural model (Vahdat et al., 2018a,b; Lorberbom et al., 2019, i.a.), we use the Gumbel-Softmax reparametrization (Jang et al., 2017; Maddison et al., 2016). This approximates a discrete draw with logits  as softmax(  + g  ) , where g is a vector of Gumbel(0 , 1) draws and  is an annealing temperature that allows targeted behavior; its ease, customizability, and efficacy are big advantages. Sequential event modeling, as in this paper, can be viewed as a type of script or schema induction (Schank and Abelson, 1977) via language modeling techniques. Mooney and DeJong (1985) provided an early analysis of explanatory schema <latexit sha1_base64="OAXIudvJ7tf0NU68IKHuiNr2LT0=">AAAB8XicdVBNSwMxEJ2tX7V+VT16CRbB05JdW2xvRS89KlgrtkvJptk2NJtdkqxQlv4LLx4U8eq/8ea/Ma0VVPTBwOO9GWbmhang2mD87hSWlldW14rrpY3Nre2d8u7etU4yRVmbJiJRNyHRTHDJ2oYbwW5SxUgcCtYJx+czv3PHlOaJvDKTlAUxGUoecUqMlW57rWzI8qjvTfvlCnYbDVyt1hB2a9j3/bol+MSvNzzkuXiOCixw0S+/9QYJzWImDRVE666HUxPkRBlOBZuWeplmKaFjMmRdSyWJmQ7y+cVTdGSVAYoSZUsaNFe/T+Qk1noSh7YzJmakf3sz8S+vm5moHuRcpplhkn4uijKBTIJm76MBV4waMbGEUMXtrYiOiCLU2JBKNoSvT9H/5Np3vZqLL6uV5tkijiIcwCEcgwen0IQWXEAbKEi4h0d4crTz4Dw7L5+tBWcxsw8/4Lx+AMUwkP4=</latexit> f 1 <latexit sha1_base64="d8TrulWQx4qqnM6MhIhcgGBMTh0=">AAAB8XicdVBNS8NAEN3Ur1q/qh69LBbBU0lLS9Nb0UuPFewHtqFstpN26WYTdjdCCf0XXjwo4tV/481/46aNoKIPBh7vzTAzz4s4U9q2P6zcxubW9k5+t7C3f3B4VDw+6akwlhS6NOShHHhEAWcCupppDoNIAgk8Dn1vfp36/XuQioXiVi8icAMyFcxnlGgj3Y3a8RQSf1xdjoslu2zbjlNv4JSkMKTZrDWdGq5kSgll6IyL76NJSOMAhKacKDWs2JF2EyI1oxyWhVGsICJ0TqYwNFSQAJSbrC5e4gujTLAfSlNC45X6fSIhgVKLwDOdAdEz9dtLxb+8Yax9x02YiGINgq4X+THHOsTp+3jCJFDNF4YQKpm5FdMZkYRqE1LBhPD1Kf6f9KrlSr1s39RKrassjjw6Q+foElVQA7VQG3VQF1Ek0AN6Qs+Wsh6tF+t13ZqzsplT9APW2yfC/ZD9</latexit> f 2 <latexit sha1_base64="aipUtqXOYJJNUZBDtiGm3Yk5JoM=">AAAB6nicdVDJSgNBEK2JW4xb1KOXxiB4GnrGBJNb0IsXIaJZIBlCT6cnadKz0N0jhCGf4MWDIl79Im/+jZ1FUNEHBY/3qqiq5yeCK43xh5VbWV1b38hvFra2d3b3ivsHLRWnkrImjUUsOz5RTPCINTXXgnUSyUjoC9b2x5czv33PpOJxdKcnCfNCMox4wCnRRroN+tf9YgnbtRoulysI2xXsum7VEHzmVmsOcmw8RwmWaPSL771BTNOQRZoKolTXwYn2MiI1p4JNC71UsYTQMRmyrqERCZnysvmpU3RilAEKYmkq0miufp/ISKjUJPRNZ0j0SP32ZuJfXjfVQdXLeJSkmkV0sShIBdIxmv2NBlwyqsXEEEIlN7ciOiKSUG3SKZgQvj5F/5OWazsVG9+US/WLZRx5OIJjOAUHzqEOV9CAJlAYwgM8wbMlrEfrxXpdtOas5cwh/ID19gmF6433</latexit> f M Sample Frame Frames Lookup Emb Extracted Frames Train collided with train Gumbel-Softmax Params Impact Event and Frames Alignment RNN Hidden States <latexit sha1_base64="A7mnOCRVU3hPWuanN9QnGogYoxA=">AAAB8nicbVBNS8NAEN3Ur1q/qh69LBbBiyURRY9FLx4rWFtIQ9hsJ+3S3WzY3Qgl5Gd48aCIV3+NN/+N2zYHbX0w8Hhvhpl5UcqZNq777VRWVtfWN6qbta3tnd29+v7Bo5aZotChkkvVi4gGzhLoGGY49FIFREQcutH4dup3n0BpJpMHM0khEGSYsJhRYqzkQ5jHYS7OvKII6w236c6Al4lXkgYq0Q7rX/2BpJmAxFBOtPY9NzVBTpRhlENR62caUkLHZAi+pQkRoIN8dnKBT6wywLFUthKDZ+rviZwIrScisp2CmJFe9Kbif56fmfg6yFmSZgYSOl8UZxwbiaf/4wFTQA2fWEKoYvZWTEdEEWpsSjUbgrf48jJ5PG96l033/qLRuinjqKIjdIxOkYeuUAvdoTbqIIokekav6M0xzovz7nzMWytOOXOI/sD5/AEwG5Ew</latexit> e f m  1 Context Frame and Events Alignment Sample Token Context External Information ENCODER DECODER Figure 2: Our encoder (left) and decoder (right). generating system to process narratives, and Pichotta and Mooney (2016) applied an LSTM-based model to predict the event arguments. Modi (2016) proposed a neural network model to predict randomly missing events, while Rudinger et al. (2015) showed how neural language modeling can be used for sequential event prediction. Weber et al. (2018a) and Ding et al. (2019) used tensor-based decomposition methods for event representation. Weber et al. (2020) studied causality in event modeling via a latent neural language model. Previous work has also examined how to incorporate or learn various forms of semantic representations while modeling events. Cheung et al. (2013) introduced an HMM-based model to explain event sequences via latent frames. Materna (2012), Chambers (2013) and Bamman et al. (2013) provided structured graphical models to learn event models over syntactic dependencies; Ferraro and Van Durme (2016) unified and generalized these approaches to capture varying levels of semantic forms and representation. Kallmeyer et al. (2018) proposed a Bayesian network based on a hierarchical dependency between syntactic dependencies and frames. Ribeiro et al. (2019) provided an analysis of clustering predicates and their arguments to infer semantic frames. Variational autoencoders and attention networks (Kingma and Welling, 2013; Bahdanau et al., 2014), allowed Bisk et al. (2019) to use RNNs with attention to capture the abstract and concrete script representations. Weber et al. (2018b) came up with a recurrent autoencoder model (HAQAE), which used vector-quantization to learn hierarchical dependencies among discrete latent variables and an observed event sequence. Kiyomaru et al. (2019) suggested generating next events using a conditional VAE-based model. In another thread of research, Chen et al. (2018) utilized labels in conjunction with latent variables, but unlike Weber et al. (2018b), their model's latent variables are conditionally independent and do not form a hierarchy. Snderby et al. (2016) proposed a sequential latent structure with Gaussian latent variables. Livin et al. (2019), similar to our model structure, provided an analysis of hierarchical relaxed categorical sampling but for the unsupervised settings. Our focus in this paper is modeling sequential event structure. In this section, we describe our variational autoencoder model, and demonstrate how partially-observed external knowledge can be injected into the learning process. We provide an overview of our joint model in 3.1 and Fig. 2. Our model operates on sequences of events: it consists of an encoder (3.2) that encodes the sequence of events as a new sequence of frames (higher-level, more abstract representations), and a decoder (3.3) that learns how to reconstruct the original sequence of events from the representation provided by the encoder. During training (3.4), the model can make use of partially-observed sequential knowledge to enrich the representations produced by the encoder. In 3.5 we summarize the novel aspects of our model. We define each document as a sequence of M events. In keeping with previous work on event representation, each event is represented as a lexi-calized 4-tuple: the core event predicate (verb), two main arguments (subject and object), and event modifier (if applicable) (Pichotta and Mooney, 2016; Weber et al., 2018b). For simplicity, we can write each document as a sequence of T words w = { w t } Tt =1 , where T = 4 M and each w t is from a vocabulary of size V . 1 Fig. 1 gives an example of 3 events: during learning (but not testing) our model would have access to some, but not all, frames to lightly guide training (in this case, the first two frames but not the third). While lexically rich, this 4-tuple representation is limited in the knowledge that can be directly encoded. Therefore, our model assumes that each document w can be explained jointly with a collection of M random variables f m : f = { f m } Mm =1 . The joint probability for our model factorizes as p ( w , f ) = T (cid:89) t =1 p ( w t | f , w <t ) M (cid:89) m =1 p ( f m | f m  1 ) . For event modeling, each f m represents a semantic frame. We assume there are F unique frames and let f m be a discrete variable indicating which frame, if any, was triggered by event m . 2 In the general case, f is completely unobserved, and so inference for this model requires marginal-izing over f : when F (cid:29) 1 , optimizing the likelihood is intractable. We follow amortized variational inference (Kingma and Welling, 2013) as an alternative approach and use an ancestral sampling technique to compute it. We define q ( f | w ) as the variational distribution over f , which can be thought of as stochastically encoding w as f . Our method is semi-supervised, so we follow Kingma et al. (2014), Chen et al. (2018) and Ye et al. (2020) and optimize a weighted variant of the evidence lower bound (ELBO), L = Reconstruction term (cid:122) (cid:125)(cid:124) (cid:123) E q ( f | w ) log p ( w | f ) + KL term (cid:122) (cid:125)(cid:124) (cid:123)  q E q ( f | w ) log p ( f ) q ( f | w ) , + Supervised classification term (cid:122) (cid:125)(cid:124) (cid:123)  c L c ( q ( f | w )) , (4) where L c ( q ( f | w )) is a classification objective that encourages q to predict the frames that actually were observed, and  q and  c are empirically-set to give different weight to the KL vs. classification terms. We define L c in 3.4. The reconstruc-1 Table 4 in the Appendix provides all the notations. 2 Our model is theoretically adaptable to making use of multiple frames, though we assume each event triggers at most one frame. tion term learns to generate the observed events w across all valid encodings f , while the KL term uses the prior p ( f ) to regularize q . Optimizing Eq. (4) is in general intractable, so we sample S chains of variables f (1) , . . . , f ( S ) from q ( f | w ) and approximate Eq. (4) as L  1 S (cid:88) s (cid:34) log p ( w | f ( s ) ) +  q log p ( f ( s ) ) q ( f ( s ) | w )+  c L c ( q ( f ( s ) | w )) (cid:105) . (5) As our model is designed to allow the injection of external knowledge I , we define the variational distribution as q ( f | w ; I ) . In our experiments, I m is a binary vector encoding which (if any) frame is observed for event m . 3 For example in Fig. 1, we have I 1 = 1 and I 3 = 0 . We define q ( f | w ; I ) = M (cid:89) m =1 q ( f m | f m  1 , I m , w ) . We broadly refer to Eq. (6) as our encoder ; we detail this in 3.2. In 3.3 we describe how we compute the reconstruction term, and in 3.4 we provide our semi-supervised training procedure. The reconstruction term relies on the frame samples given by the encoder. As discussed above though, we must be able to draw chains of variables f , by iteratively sampling f m  q ( | f m  1 , w ; I ) , in a way that allows the external knowledge I to guide , but not restrict, f . This is a deceptively difficult task, as the encoder must be able to take the external knowledge into account in a way that neither prevents nor harms back-propagation and learning. We solve this problem by learning to compute a good representation  m for each event, and sampling the current frame f m from a Gumbel-Softmax distribution (Jang et al., 2017) parametrized by  m . Alg. 1 gives a detailed description of our encoder. We first run our event sequence through a recurrent network (like an RNN or bi-LSTM); if w is T tokens long, this produces T hidden representations, each of size d h . Let this collection be H  RT  d h . Our encoder proceeds iteratively over each of the M events as follows: given the previous sampled frame f m  1 , the encoder first computes a weighted embedding e f m  1 of this previous frame (line 1). Algorithm 1 Encoder: The following algorithm shows how we compute the next frame f m given the previous frame f m  1 . We compute and return a hidden frame representation  m , and f m via a continuous Gumbel-Softmax reparametrization. Next, it calculates the similarity between e f m  1 and RNN hidden representations and all recurrent hidden states H (line 2). After deriving the attention scores, the weighted average of hidden states ( c m ) summarizes the role of tokens in influencing the frame f m for the m th event (line 3). We then combine the previous frame embedding e f m  1 and the current context vector c m to obtain a representation  (cid:48) m for the m th event (line 4). While  (cid:48) m may be an appropriate representation if no external knowledge is provided, our encoder needs to be able to inject any provided external knowledge I m . Our model defines a chain of variablessome of which may be observed and some of which may be latentso care must be taken to preserve the gradient flow within the network. We note that an initial strategy of solely using I m instead of f m (whenever I m is provided) is not sufficient to ensure gradient flow. Instead, we incorporate the observed information given by I m by adding this information to the output of the encoder logits before drawing f m samples (line 5). This remedy motivates the encoder to softly in-crease the importance of the observed frames during the training. Finally, we draw f m from the Gumbel-Softmax distribution (line 7). For example, in Fig. 1, when the model knows that [ IMPACT ] is triggered, it increases the value of [ IMPACT ] in  m to encourage [ IMPACT ] to be sampled, but it does not prevent other frames from being sampled. On the other hand, when a frame is not observed in training, such as for the third event ([ KILLING ]),  m is not adjusted. Since each draw f m from a Gumbel-Softmax is a simplex vector, given learnable frame embeddings EF , we can obtain an aggregate frame representation e m by calculating e m = f m (cid:62) EF . This can be thought of as roughly extracting row m from EF for low entropy draws f m , and using many frames in the representation for high entropy draws. Via the temperature hyperparameter, the Gumbel-Softmax allows us to control the entropy. Our decoder (Alg. 2) must be able to reconstruct the input event token sequence from the frame representations f = ( f 1 , . . . , f M ) computed by the encoder. In contrast to the encoder, the decoder is relatively simple: we use an auto-regressive (left-to-right) GRU to produce hidden representations z t for each token we need to reconstruct, but we enrich that representation via an attention mechanism over f . Specifically, we use both f and the same learned frame embeddings EF from the encoder to compute inferred, contextualized frame embeddings as EM = f EF  RM  d e . For each output token (time step t ), we align the decoder GRU hidden state h t with the rows of EM (line 1). After calculating the scores for each frame embedding, we obtain the output context vector c t (line 2), which is used in conjunction with the hidden state of the decoder z t to generate the w t token (line 5). In Fig. 1, the collection of all the three frames and the tokens from the first event will be used to predict the Train token from the second event. We now analyze the different terms in Eq. (5). In our experiments, we have set the number of samples S to be 1. From Eq. (6), and using the sampled sequence of frames f 1 , f 2 , . . . f M from our encoder, we approximate the reconstruction term as L w = (cid:80) t log p ( w t | f 1 , f 2 , . . . f M ; z t ) , where z t is the decoder GRU's hidden representation after having reconstructed the previous t  1 words in the event sequence. Looking at the KL-term, we define the prior frame-to-frame distribution as p ( f m | f m  1 ) = 1 /F , and let the variational distribution capture the dependency between the frames. A similar type of strategy has been exploited successfully by Chen et al. (2018) to make computation simpler. We see the computational benefits Algorithm 2 Decoder: To (re)generate each token in the event sequence, we compute an attention   over the sequence of frame random variables f (from Alg. 1). This attention weights each frame's contribution to generating the current word. Input: EM  RM  d e (computed as f EF ) decoder's current hidden state z t  R d h Parameters:  W in  R d e  d h ,  W out  RV  d e , EF Output: w t 1:    Softmax( EM  W in z t ) (cid:46) Attn. Scores 2: c t  EM (cid:62)   (cid:46) Context Vector 3: g   W out ( tanh(  W in z t ) + tanh( c t )) 4: p ( w t | f ; z t )  exp( g ) 5: w t  p ( w t | f ; z t ) of a uniform prior: splitting the KL term into E q ( f | w ) log p ( f )  E q ( f | w ) log q ( f | w ) allows us to neglect the first term. For the second term, we normalize the Gumbel-Softmax logits  m , i.e.,  m = Softmax(  m ) , and compute L q =  E q ( f | w ) log q ( f | w )   (cid:88) m  (cid:62) m log  m . L q encourages the entropy of the variational distribution to be high which makes it hard for the encoder to distinguish the true frames from the wrong ones. We add a fixed and constant regularization coefficient  q to decrease the effect of this term (Bowman et al., 2015). We define the classification loss as L c =  (cid:80) I m > 0 I m (cid:62) log  m , to encourage q to be good at predicting any frames that were actually observed. We weight L c by a fixed coefficient  c . Summing these losses together, we arrive at our objective function: L = L w +  q L q +  c L c . A number of efforts have leveraged frame induction for event modeling (Cheung et al., 2013; Chambers, 2013; Ferraro and Van Durme, 2016; Kallmeyer et al., 2018; Ribeiro et al., 2019). These methods are restricted to explicit connections between events and their corresponding frames; they do not capture all the possible connections between the observed events and frames. Weber et al. (2018b) proposed a hierarchical unsupervised attention structure (HAQAE) that corrects for this. HAQAE uses vector quantization (Van Den Oord et al., 2017) to capture sequential event structure via tree-based latent variables. Our model is related to HAQAE (Weber et al., 2018b), though with important differences. While HAQAE relies on unsupervised deterministic inference, we aim to incorporate the frame information in a softer, guided fashion. The core differences are: our model supports partially-observed frame sequences (i.e., semi-supervised learning); the linear-chain connection among the event variables in our model is simpler than the tree-based structure in HAQAE; while both works use attention mechanisms in the encoder and decoder, our attention mechanism is based on addition rather than concatenation; and our handling of latent discrete variables is based on the Gumbel-Softmax reparametrization, rather than vector quantization. We discuss these differences further in Appendix C.1. We test the performance of our model on a portion of the Concretely Annotated Wikipedia dataset (Ferraro et al., 2014), which is a dump of English Wikipedia that has been annotated with the outputs of more than a dozen NLP analytics; we use this as it has readily-available FrameNet annotations provided via SemaFor (Das et al., 2014). Our training data has 457k documents, our validation set has 16k documents, and our test set has 21k documents. More than 99% of the frames are concentrated in the first 500 most common frames, so we set F = 500 . Nearly 15% of the events did not have any frame, many of which were due to auxiliary/modal verb structures; as a result, we did not include them. For all the experiments, the vocabulary size ( V ) is set as 40 k and the number of events ( M ) is 5; this is to maintain comparability with HAQAE. For the documents that had more than 5 events, we extracted the first 5 events that had frames. For both the validation and test datasets, we have set I m = 0 for all the events; frames are only observed during training. Documents are fed to the model as a sequence of events with verb, subj, object and modifier elements. The events are separated with a special separating < TUP > token and the missing elements are represented with a special NONE token. In order to facilitate semi-supervised training and examine the impact of frame knowledge, we introduce a user-set value (cid:15) : in each document, for event m , the true value of the frame is preserved in I m with probability (cid:15) , while with probability 1  (cid:15) we set I m = 0 . This (cid:15) is set and fixed prior to each experiment. For Model (cid:15) PPL Valid Test RNNLM -61.34  2 . Setup We represent words by their pretrained Glove 300 embeddings and used gradient clipping at 5.0 to prevent exploding gradients. We use a two layer of bi-directional GRU for the encoder, and a two layer uni-directional GRU for the decoder (with a hidden dimension of 512 for both). See Appendix B for additional computational details. 4 Baselines In our experiments, we compare our proposed methods against the following methods:  RNNLM : We report the performance of a sequence to sequence language model with the same structure used in our own model. A Bidirectional GRU cell with two layers, hidden dimension of 512, gradient clipping at 5 and Glove 300 embeddings to represent words. RNNLM+ROLE (Pichotta and Mooney, 2016): This model has the same structure as RNNLM, but the role for each token (verb, subject, object, modifier) as a learnable embedding vector is concatenated to the token embeddings and then it is fed to the model. The embedding dimension for roles is 300. HAQAE (Weber et al., 2018b) This work is the most similar to ours. For fairness, we seed HAQAE with the same dimension GRUs and pretrained embeddings. To measure the effectiveness of our proposed model for event representation, we first report the perplexity and Inverse Narrative Cloze metrics. Perplexity We summarize our per-word perplexity results in Table 1a, which compares our event chain model, with varying (cid:15) values, to the three baselines. 5 Recall that (cid:15) refers to the (average) percent of frames observed during training . During evaluation no frames are observed; this ensures a fair comparison to our baselines. As clearly seen, our model outperforms other baselines across both the validation and test datasets. We find that increasing the observation probability (cid:15) consistently yields performance improvement. For any value of (cid:15) we outperform RNNLM and RNNLM+ROLE. HAQAE outperforms our model for (cid:15)  0 . 5 , while we outperform HAQAE for (cid:15)  { 0 . 7 , 0 . 8 , 0 . 9 } . This suggests that while the tree-based latent structure can be helpful when external, semantic knowledge is not sufficiently available during training, a simpler linear structure can be successfully guided by that knowledge when it is available. Finally, recall that the external frame annotations are automatically provided, without human curation: this suggests that our model does not require perfectly, curated annotations. These observations support the hypothesis that frame observations, in conjunction with latent variables, provide a benefit to event modeling. Inverse Narrative Cloze This task has been proposed by Weber et al. (2018b) to evaluate the ability of models to classify the legitimate sequence of events over detractor sequences. For this task, we have created two Wiki-based datasets from our validation and test datasets, each with 2k samples. Each sample has 6 options in which the first events are the same and only one of the options represents the actual sequence of events. All the options have a fixed size of 6 events and the one that has the lowest perplexity is selected as the correct one. We also consider two NYT inverse narrative cloze datasets that are publicly available. 6 All the models are trained on the Wiki dataset and then classifications are done on the NYT dataset (no NYT training data was publicly available). Table 1b presents the results for this task. Our method tends to achieve a superior classification score over all the baselines, even for small (cid:15) . Our model also yields performance improvements on the NYT validation and test datasets. We observe that the inverse narrative cloze scores for the NYT datasets is almost independent from the (cid:15) . We suspect this due to the different domains between training (Wikipedia) and testing (newswire). Note that while our model's perplexity improved monotonically as (cid:15) increased, we do not see monotonic changes, with respect to (cid:15) , for this task. By examining computed quantities from our model, we observed both that a high (cid:15) resulted in very low entropy attention and that frames very often attended to the verb of the eventit learned this association despite never being explicitly directly to. While this is a benefit to localized next word prediction (i.e., perplexity), it is detrimental to inverse narrative cloze. On the other hand, lower (cid:15) resulted in slightly higher attention entropy, suggesting that less peaky attention allows the model to capture more of the entire event sequence and improve global coherence. To illustrate the effectiveness of our proposed attention mechanism, in Table 2 we show the most likely frames given tokens (  enc ) , and tokens given frames (  dec ). We define  enc = W out tanh( H (cid:62) ) and  dec =  W out tanh( EM (cid:62) ) where  enc  RF  T provides a contextual token-to-frame soft clustering matrix for each document and analogously  dec  RV  M provides a frame-to-word soft-clustering contextualized in part based on the inferred frames. We argue that these clusters are useful for analyzing and interpreting the model and its predictions. Our experiments demonstrate that the frames in the encoder (Table 2, top) mostly attend to the verbs and similarly the decoder utilizes expected and reasonable frames to predict the next verb. Note that we have not restricted the frames and tokens connection: the attention mechanism makes the ultimate decision for these connections. We note that these clusters are a result of our attention mechanisms. Recall that in both the encoder and decoder algorithms, after computing the context vectors, we use the addition of two tanh(  ) functions with the goal of separating the GRU hidden states and frame embeddings (line 3). This is a different computation from the bi-linear attention mechanism (Luong et al., 2015) that applies the tanh(  ) function over concatenation. Our additive approach was inspired by the neural topic modeling method from Dieng et al. (2016), which similarly uses additive factors to learn an expressive and predictive neural component and the classic topics (distributions/clusters over words) that traditional topic models excel at finding. While theoretical guarantees are beyond our scope, qualitative analyses suggests that our additive attention lets the model learn reasonable soft clusters of tokens into frame-based topics. See Table 6 in the Appendix for an empirical comparison and validation of our use of addition rather than concatenation in the attention mechanisms. Though we develop a generative model, we want to make sure the latent nodes are capable of leveraging the frame information in the decoder. We examine this assessing the ability of one single latent node to classify the frame for an event. We repurpose the Wikipedia language modeling dataset into a new training data set with 1,643,656 samples, validation with 57,261 samples and test with 75903 samples. We used 500 frame labels. Each sample is a single event. We fixed the number of latent nodes to be one. We use RNNLM and RNNLM+ROLE as baselines, adding a linear classifier layer followed by the softplus function on top of the bidirectional GRU last hidden vectors and a dropout of 0.15 on the logits. We trained all the models with the aforementioned training dataset, and tuned the hyper parameters on the validation dataset. We trained the RNNLM and RNNLM+ROLE baselines in a purely supervised way, whereas our model mixed supervised (discriminative) and unsupervised (generative) training. The baselines observed all of the frame labels in the training set; our model only observed frame values in training with probability (cid:15) , which it predicted from  m . The parameters leading to the highest accuracy were chosen to evaluate the classification on the test dataset. The results for this task are summarized in Table 3. Our method is an attention based model which captures all the dependencies in each event to construct the latent representation, but the baselines are autoregressive models. Our encoder acts like a discriminative classifier to predict the frames, where they will later be used in the decoder to construct the events. We expect the model performance to be comparable to RNNLM and RNNLM+ROLE in terms of classification when (cid:15) is high. Our model with larger (cid:15) tends to achieve better performance in terms of macro precision and macro F1-score. Our experiments show that our proposed approach converges faster than the existing HAQAE model. For fairness, we have used the same data-loader, batch size as 100 , learning rate as 10  3 and Adam optimizer. In Fig. 3, on average each iteration takes 0.2951 seconds for HAQAE and 0.2958 seconds for our model. From Fig. 3 we can see that for sufficiently high values of (cid:15) our model is converging both better, in terms of negative log-likelihood (NLL), and fasterthough for small (cid:15) , our model still converges much faster. The reasons for this can be boiled down to utilizing Gumbel-Softmax rather than VQ-VAE, and also injection information in the form of frames jointly. We showed how to learn a semi-supervised VAE with partially observed, sequential, discrete latent variables. We used Gumbel-Softmax and a modi-fied attention to learn a highly effective event language model (low perplexity), predictor of how an initial event may progress (improved inverse narrative cloze), and a task-based classifier (outper-forming fully supervised systems). We believe that future work could extend our method by incorporating other sources or types of knowledge (such as entity or commonsense knowledge), and by using other forms of a prior distribution, such as plug-and-play priors (Guo et al., 2019; Moham-madi et al., 2021; Laumont et al., 2021). suggestions. Some experiments were conducted on the UMBC HPCF, supported by the National Science Foundation under Grant No. CNS-1920079. We'd also like to thank the reviewers for their comments and suggestions. This material is based in part upon work supported by the National Science Foundation under Grant Nos. IIS-1940931 and IIS-2024878. This material is also based on research that is in part supported by the Air Force Research Laboratory (AFRL), DARPA, for the KAIROS program under agreement number FA8750-19-2-1003. The U.S.Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either express or implied, of the Air Force Research Laboratory (AFRL), DARPA, or the U.S. Government.
objective method abstain result abstain abstain abstain result abstain other result objective method abstain method result abstain abstain abstain objective method abstain result objective result abstain objective result method abstain other other abstain method abstain abstain abstain other method other abstain other abstain method other other other other other other other other other other other other other other method method method method method abstain method method abstain method method abstain method method abstain method abstain method method method method abstain method abstain abstain method abstain method method method method method method method abstain method abstain method abstain method method abstain method method method abstain abstain method method method method method abstain method abstain abstain method abstain method method method method method method abstain method abstain method abstain method method abstain method abstain abstain method method abstain method method method method abstain abstain abstain abstain method method method method abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result result abstain other other other other other other other other other