Neural Assimilation

We introduce a new neural network for Data Assimilation (DA). DA is the approximation of the true state of some physical system at a given time obtained combining time-distributed observations with a dynamic model in an optimal way. The typical assimilation scheme is made up of two major steps: a prediction and a correction of the prediction by including information provided by observed data. This is the so called prediction-correction cycle. Classical methods for DA include Kalman filter (KF). KF can provide a rich information structure about the solution but it is often complex and time-consuming. In operational forecasting there is insufficient time to restart a run from the beginning with new data. Therefore, data assimilation should enable real-time utilization of data to improve predictions. This mandates the choice of an efficient data assimilation algorithm. Due to this necessity, we introduce, in this paper, the Neural Assimilation (NA), a coupled neural network made of two Recurrent Neural Networks trained on forecasting data and observed data respectively. We prove that the solution of NA is the same of KF. As NA is trained on both forecasting and observed data, after the phase of training NA is used for the prediction without the necessity of a correction given by the observations. This allows to avoid the prediction-correction cycle making the whole process very fast. Experimental results are provided and NA is tested to improve the prediction of oxygen diffusion across the Blood-Brain Barrier (BBB).

Introduction and Motivations
The current approach to forecasting modelling consists of simulating explicitly only the largest-scale phenomena, while taking into account the smaller-scale ones by means of "physical parameterisations". All numerical models introduce uncertainty through the selection of scales and parameters. Additionally, any computational methodology contributes to uncertainty due to discretization, finite precision and accumulation of round-off errors. Finally the ever growing size of the computational domains leads to increasing sources of uncertainties. Taking into account these uncertainties is essential for the acceptance of any numerical simulation. Numerical forecasting models often use Data Assimilation methods for the uncertainty quantification in the medium to long-term analysis.
Data Assimilation (DA) is the approximation of the true state of some physical system at a given time by combining time-distributed observations with a dynamic model in an optimal way. DA can be classically approached in two ways: as variational DA [16] and as filtering [5]. In both cases we seek an optimal solution. The most popular filtering approach for data assimilation is the Kalman Filter (KF) [15]. Statistically, KF seeks a solution with minimum variance. Variational methods seek a solution that minimizes a suitable cost function. In certain cases, the two approaches are identical and provide exactly the same solution [16]. However, the statistical approach, though often complex and time-consuming, can provide a richer information structure, i.e. an average and some characteristics of its variability (probability distribution). During the last 20 years hybrid approaches [11,18] have become very popular as they combine the two approaches into a single taking advantage of the relative rapidity and robustness of variational approaches, and at the same time, obtaining an accurate solution [2] thanks to the statistical approach. In this paper, in order to achieve the accuracy of the KF solution and reduce the execution time, we use Recurrent Neural Networks (RNN). Today the computational power of RNN is exploited for several application in different fields. Any non-linear dynamical system can be approximated to any accuracy by a Recurrent Neural Network, with no restrictions on the compactness of the state space, provided that the network has enough sigmoidal hidden units. This is what the Universal Approximation Theorem [12,20] claims. Only during the last few years, the DA community is starting to approach machine learning models to improve the efficiency of DA models. In [17], the authors combined Deep Learning and Data Assimilation to predict the production of gas from mature gas wells. They used a modified deep LSTM model as their prediction model in the EnKF framework for parameter estimation. Even if the prediction phase is speed up due to the introduction of Deep Learning, this only partially affects the whole prediction-correction cycle which is still time-consuming. In [9], the authors presented an approach for employing artificial neural networks (NNs) to emulate the local ensemble transform Kalman filter (LETKF) as a method of data assimilation. Even if the Feed Forward NN they implemented is able to emulate the DA process for the time window they fixed, when they need to assimilate observations in new time steps, it still needs the prediction-correction cycle and this affects the execution time which is just 90 times faster than the reference DA model. To further speed up the process, in [8] the authors combined the power of Neural Networks and High Performance Computing to assimilate meteorological data. These studies, alongside others discussed in conferences and still under publication, highlight the necessity to avoid the prediction-correction cycle by developing a Neural Network able to completely emulate the whole Data Assimilation process. In this context, we developed a Neural Assimilation (NA) as a Coupled Neural Network made of two RNNs. NA captures the features of a Data Assimilation process by interleaving the training of the two component RNNs on the forecasting data and the observed data. That is, the two component RNNs are trained on forecasting and observed data respectively with additional inputs provided by the interaction of these two. This NA network emulates the KF and runs much faster than the KF prediction-correction cycle for data assimilation. In this paper we develop the NA architecture and proved its equivalence to the KF. The equivalence between NA and KF is independent from the structure on the RNNs. In this paper we show results we obtained employing two Long short-term memory (LSTM) architectures for the two RNNs. Then we employ the NA model to a practical problem in predicting of oxygen (and drugs) diffusion across the Blood-Brain Barrier (BBB) [1] to justify its correctness and efficiency.
This paper is structured as follows. In Sect. 2 the Data Assimilation problem is described. The Neural Assimilation is introduced in Sect. 3, where we investigate the accuracy of the introduced method and we present a theorem demonstrating that the novel model is consistent with the KF result. Experimental results are provided in Sect. 4. Conclusions and future works are summarised in Sect. 5.

Data Assimilation
Data Assimilation (DA) is the approximation of the true state of some physical system at a given time by combining time-distributed observations o(t) with a dynamic modelẋ = M(x, t) in an optimal way. DA can be classically approached in two ways: as variational DA [3] and as filtering. One of the best known tools for filtering approach is the Kalman filter (KF) [15]. We seek to estimate the state x(t) of a discrete-time dynamic process that is governed by the linear difference equation Note that M and H are discrete operators. The random vectors w t and v t represent the modeling and the observation errors respectively. They are assumed to be independent, white-noise processes with normal probability distributions where B t and R t are covariance matrices of the modeling and observation errors respectively. All these assumptions about unbiased and uncorrelated errors (in time and between each other) are not limiting, since extensions of the standard KF can be developed should any of these not be valid [5]. The KF problem can be summarised as follows: given a background estimate x(t), of the system state at time t, what is the best analysis z(t) based on the current available observation o(t)?
The typical assimilation scheme is made up of two major steps: a prediction step and a correction step. At time t we have the result of the previous forecast, x(t) and the result of an ensemble of observations o(t). Based on these two vectors, we perform an analysis that produces z(t). We then use the evolution model to obtain a prediction of the state at time t + 1. The result of the forecast at the prediction step is denoted with x(t + 1) and becomes the background for the next correction time step: We observe that, in case the observed data are defined in the same space of the state variable, the operator H t in (2) is the identity matrix and the Eqs. (6)-(7) can be simplified becoming: Due to the high computational cost in updating the covariance matrices B t by Eq. (5), it in operational DA, is often used to assume B t = B t+1 ∀t. This assumption leads to a model which is also called Optimal Interpolation [16].
Statistically, KF seeks a solution with minimum variance. This approach, though often complex and time-consuming, can provide a rich information structure (often richer than information provided by variational DA), such as an average and some characteristics of its variability (probability distribution). In order to maintain the accuracy of the KF solution and reduce the execution time, we introduce, in the next section, a Neural Assimilation (NA) which is a network representing KF but much faster than a KF prediction-correction cycle.

Neural Assimilation
For a fixed time window [t 0 , t 1 ] and a fixed discretization time step Δt, let x(t) still denote the forecasting result at each time step t ∈ [t 0 , t 1 ]. Let o(t) denotes an observation of the state value (Fig. 1). As it does not affect the generality of our study, we are assuming here the observed data defined in the same space of the state variable, i.e. the operator H t in (2)   A fundamental feature of each network is that it contains a feedback connection, so the activations can flow round in a loop. That enables the networks to do temporal processing and learn sequences with temporal prediction. The form of NA is a RNN with the previous set of hidden unit activations feeding back into the network along with the inputs.

Fig. 2. Neural Assimilation
Note that the time t is discretized, with the activations updated at each time step. The time scale might correspond to any time step of size appropriate for the given problem. A delay unit given by the network NN F needs to be introduced to hold activations in NN O until they are processed at the next time step and vice versa. As for simple architectures and deterministic activation functions, learning will be achieved using similar gradient descent procedures to those leading to the back-propagation algorithm for feed forward networks.
The NA scheme is made up of two major steps: a pre-processing step and a training step. During the pre-processing step, the data set is normalized considering the information we have about the error estimations and the error covariance matrices introduced in (3). We consider, to normalise, the inverse of the error covariance matrices so that, data with big covariance/variance are assumed with a small weight [5,16]. We posē The computed vectorsx(t) andō(t) are the data used in the training step: where the vectorsx(t − 1) are the inputs, the matrices W IH , W HH , W HOF and W HOO are the four connection weight matrices, and f H , f OF and f OO are the hidden and outputs unit activation functions. The state of the dynamical system is a set of values that summarizes all the information about the past behaviour of the system that is necessary to provide a unique description of its future behaviour, apart from the effect of any external factors. In this case the state is defined by the set of hidden unit activations h(t). The Back propagation Through Time for this algorithm is a natural extension of standard back propagation that performs gradient descent on a complete unfolded network ( [21], Chapter 5 of [6]). If the NA training sequence starts at time t 0 and ends at time t 1 , the total cost function is simply the sum over time of the standard error function C(t) at each time-step: where and n is the total number of training samples. The gradient descent weight updates have contributions from each time-step [19]: where η is the learning rate [14]. The constituent partial derivatives ∂C(t) ∂wij have contributions from the multiple instances of each weight w ij ∈ {W IH , W HH , W HOO , W HOF } and depend on the inputs and hidden unit activations at previous time steps. The errors now have to be back-propagated through time as well as through the network [23].
We prove that the output function h(t) of the NA model corresponds to the solution of Kalman filter with fixed covariance matrices, i.e. in its Optimal Interpolation version [16]. The following result held.

Theorem 1. Let h(t) be the solution of NA given by Eqs. (10)-(16) and let z(t)
denote the solution of the KF algorithm as defined in (9). We have Proof: Due to the definition of the L 2 norm, the loss function in (15) can be written as then, from Eq. (1), and except for the numerical errors that will be introduced later as already included in the data sets, the (18) can be written as: From the properties of the L 2 norm, the (19) can be written as To minimise this loss function, we compute the gradient where M T denotes the Adjoint operator of the linear operator M [7] and we pose ∇ h(t−1) C(t) = 0, then we have: From the definition ofx andō in (10), the (22) gives: Then, adding and subtracting the quantity B t−1 x(t − 1) and merging the common factors, the (23) become

the (24) gives:
which is the expression of the KF solution z(t − 1) in (9) for the time step t − 1 and for the case of observed data defined in the same space of the state variable (i.e. H = I and I is the identity matrix). Q t−1 is the Kalman gain matrix in (8).
The Eq. (25) in Theorem 1 represents a condition to assume that NA is consistent with KF.
In Sect. 4, we validate the results provided in this section. We also show that the employment of NA alleviates the computational cost making the running less expensive.

Experimental Results
In this section we provide experimental results that demonstrate the applicability and efficiency of NA. In our experiment, the NA is implemented by adopting Long short-term memory (LSTM) architecture for the two RNNs. The reason we use LSTMs is that they are suitable to contain information outside the normal flow of the recurrent network so it is easier to plug two networks together. Also, LSTMs allow to preserve the error that can be backpropagated through time and layers which is a very important point for discrete forecasting models. A description of the NA we implemented is provided in Fig. 3.

Fig. 3. Implementation of Neural Assimilation
The test case we consider is a numerical model to predict the oxygen diffusion across the Blood-Brain Barrier (BBB). Nevertheless the model can be used for any drugs by replacing the diffusion constant and the initial and boundary conditions [1]. The Blood-Brain Barrier protects the central nervous system, controls the entry of compounds into the brain by restricting access for blood borne compounds and facilitates access for nutrients. This protection makes it difficult to provide therapeutic compounds to brain cells when they are affected by brain diseases as Alzheimer, Autism [13]. The BBB is composed of endothelial cells connected by tight junctions. The main mechanisms allowing the transport of drugs across the membrane are passive transport, carrier-mediated transport, receptor-mediated transcytosis, and adsorption-mediated transcytosis [22]. The passive transport mechanism is the easiest method of drug transport for lipophilic and low molecular size molecules. It means a simple diffusion across any membrane without application of energy and carrier proteins. Opioids and steroids are examples of drugs which can be passively diffused [4]. Assuming that the main transport mechanism is through passive diffusion, the initial threedimensional space problem can be reduced to a one-dimensional space problem. In fact, passive diffusion involves many simplifications as no reaction term, uniform movement in all directions and an overall diffusion constant. Therefore, a 1D partial differential equation (PDE) as (26) with one initial condition and two boundary conditions is an accurate model for this problem [22] where 0 corresponds to the location at which the blood meets the Blood-Brain Barrier and L = 400 nm is the real average thickness of the Blood-Brain Barrier.
where t ∈ [0, 10 ms] (ms denotes microsecond) and y ∈ [0, L]. We consider that, at time 0 there is no oxygen, then x 0,y = 0. Moreover, for our boundary conditions, we consider that we have a constant concentration of oxygen in the bloodstream and that at the interface of the barrier and the brain tissue all oxygen will be consumed x t,0 = 0.02945 L/L blood and x t,L = 0. We assume the diffusivity of oxygen through the Blood-Brain Barrier to be 3.24 * 10 −5 cm 2 /s [1]. Equation (26) is discretised by a second order central finite difference in space with Δy = 8 nm and a backward Euler method in time with Δt = 0.1 ms: where F = D Δt Δy 2 , i = 1, . . . , 50 and n = 1, . . . , 100. As we know that it does not affect the generality of our study, in this paper we show results of NA using observed data o(t) provided in [1] by the analytical solution of (26) for the oxygen diffusion. The model can be used for any drugs by replacing the diffusion constant and the initial and boundary conditions. Data sets for observed data can be found in http://cheminformatics.org/datasets/. The NA code and the pre-processed data can be downloaded using the link https://drive.google.com/ drive/folders/1C O-rk5wyqFsG5U-T7 vugBOddTPmOlY?usp=sharing.
The NA network has been trained using the 85% of the data and tested on the remaining 15%. Figure 4 shows the value of the Loos function for training and testing the forecasting network.
NA has been compiled as a sequential neural network with just one LSTM layer of 48 units using as loss function the mean squared error one and as optimiser the Adam one. Weights are automatically initialised by Keras using: -Glorot uniform for the kernel weights matrix for the linear transformation of the inputs; -Orthogonal for the linear transformation of the recurrent state.   Figure 5 shows the temporal evolution of the concentration at y = 12 nm (Fig. 5a) and y = 35 nm (Fig. 5b). The accuracy of the NA results is evaluated by the absolute error e NA (t, y) = |z(t, y) − h(t, y)| and the mean squared error where z(t, y) is the solution of KF performed at each time step. Table 1 shows values of absolute error computed every 10 time steps. We can see that the order of magnitude of the error is between e−07 and e−04. The corresponding values of mean squared error are MSE(h(t, y)) = 1.31e−07 for y = 12 nm and MSE(h(t, y)) = 8.16e−08 for y = 35 nm where t ∈ [0, 0.10 ms]. Figure 6 shows the comparison of the KF result and the NA result for the temporal evolution of the concentration at each point of the BBB we are modelling. Values of execution time are provided in Table 2. The values are computed as mean of execution times from 100 runnings. We can observe that the time for prediction in NA is 1000 faster than the prediction with KF. Finally, Table 3 shows the values of mean square forecasting error: and mean square assimilation error: computed with respect observations o(t, y). The values of the errors in the assimilation results present a reduction of approximately one order of magnitude with respect to the error in forecasting data.

Conclusions and Future Works
We introduced a new neural network for Data Assimilation (DA) that we named Neural Assimilation (NA). We proved that the solution of NA is the same of KF. We tested the validity of the provided theoretical results showing values of misfit between the solution of NA and the solution of KF for the same test case. We provided experimental results on a realistic test case studying oxygen diffusion across the Blood-Brain Barrier. NA is trained on both forecasting and observed data and it is used for predictions without needing a correction given by the information provided by observations. This allows to avoid the prediction-correction cycle of a Kalman filter, and it makes the assimilation process very fast. We show that the time for prediction in NA is 1000 faster than the prediction with KF. An implementation of NA to emulate variational DA [10] will be developed as future work. In particular, we will focus on a 4D variational (4DVar) method [5]. 4DVar is a computational expensive method as it is developed to assimilate several observations (distributed in time) for each time step of the forecasting model. We will develop an extended version of NA able to assimilate set of distributed observations for each time step and, then, able to emulate 4DVar.


Event Presence Prediction Helps Trigger Detection Across Languages

The task of event detection and classification is central to most information retrieval applications. We show that a Transformer based architecture can effectively model event extraction as a sequence labeling task. We propose a combination of sentence level and token level training objectives that significantly boosts the performance of a BERT based event extraction model. Our approach achieves a new state-of-the-art performance on ACE 2005 data for English and Chinese. We also test our model on ERE Spanish, achieving an average gain of 2 absolute F1 points over prior best performing model.

Introduction and Prior Work
Event Extraction (EE) is an important subfield in Information Extraction. It usually comprises two subtasks: (1) Event Detection (ED) which extracts from text nuggets of information that signals occurrence of an event; (2) Event Argument Extraction (EAE) which extracts the participants of an event. The task is challenging as there are various ways to express the same event, and the same phrase can express different events. For example, pay could mean event Transaction, as in figure 1, or could mean Justice, as in "...pay a fine of...".
Events have been used by applications like Question Answering (Yang et al., 2003), Information Retrieval (Basile et al., 2014), Narrative Schema extraction (Chambers and Jurafsky, 2009), Reading Comprehension (Cheng and Erk, 2018), and to build knowledge resources.
The best performing EE models rely on advanced architectures or language specific features. Here, we propose a Transformer based model called sepBERT which uses multiple objectives to improve performance. The idea of learning additional objectives that exploit labels that are available with the data is used successfully by Mikolov et al. (2013); Devlin et al. (2019); Logeswaran and Lee (2018), etc.
We focus on Event Detection (ED) and model the ED problem as a sequence labeling task. We introduce a simple and intuitive sentence level objective that greatly improves model performance. We show that for ED: • Adding an auxiliary objective function helps achieving the state-of-the-art performance.
• Highly contextualized Transformer models can be effectively used for ED without additional features or layers on top, performing better than more elaborate Transformer based models.
• Our model can be applied to different languages and can still achieve the state-of-theart performance.

Event Extraction Task
Event Extraction is defined for Automatic Content Extraction (ACE) evaluation (Walker et al., 2006) as follows: • Event Trigger is a phrase that most clearly expresses the occurrence of an event. It is denoted by a type and subtype.
• Event Arguments are the mentions (entity, value or time) that serve as a participant or attribute with a specific role in the event. They are denoted by a role.
As shown in figure 1, pay is an event trigger of type Transaction and subtype Transfer-Ownership with two arguments The company with role Buyer and Yukos with role of Artifact. We focus on Event Trigger Extraction, also referred to as Event Detection (ED) in the literature.

Methodology
Most of the previous work on Trigger Extraction has focused on one word triggers (Nguyen and Grishman, 2018; Li et al., 2013;Chen et al., 2015), etc. Such methods classify each word of the sentence as a trigger, ignoring multi-word triggers. Lin et al. (2020) predicts the start and end positions of the triggers to capture multi-word triggers. We treat the task as a sequence labelling task to implicitly capture multi-word triggers, as is done by (M'hamdi et al., 2019). Figure 2 shows the architecture of our system, which builds on top of a pre-trained BERT model. We model trigger extraction as a token sequence labeling task. We combine the token-level classification loss L t with a novel sentence-level event presence prediction loss L sep :

Trigger Extraction
For the token-level component of our model, we use the IOB2 encoding (Sang and Veenstra, 1999) where each token is labeled with its trigger label l BERT Transformer Model and an indicator of whether it starts or continues a trigger. We introduce a linear classification layer on top of the token-level BERT output vectors. The parameters of the layer are trained via cross entropy loss, a standard approach for BERT based sequence labeling models (Devlin et al., 2019). This is equivalent to minimizing the negative log-likelihood of the true labels, Although each token is classified independently, the BERT transformer layers help capture the context across all tokens via a multi-head self-attention mechanism.
In addition, we introduce a simple yet effective sentence-level objective that serves as an auxiliary task. This is a binary classification objective, where truth y = 1 if the sentence contains an event and y = 0 otherwise. We model this by adding a classification layer over the BERT output of the [CLS] token -a token added to the beginning of all BERT inputs, L sep = − log(P (y)) As many sentences do not have events, this loss acts as a discriminator. It offers the system a global representation of when events show up in sentences. We call this loss sentence event presence (SEP) loss and our model sepBERT.

Dataset
For English and Chinese we conduct our experiments on ACE (Walker et al., 2006), a popular event extraction dataset. ACE defines 8 event types and 33 subtypes.
As ACE does not have an official test split, for English we follow the work of (Li et al., 2013) and use their pre-defined split of the documents to create a test set with the same 40 newswire documents, development set of 30 mixed documents and training set of the remaining 529. For Chinese, we follow the split proposed by (Feng et al., 2018), and randomly split the data into 60 development, 60 test, and the remaining 513 training documents.
For our Spanish language experiments we use the Entities, Relations and Events(ERE) corpus from LDC2015E107 catalog. ERE defines 9 event types, and 38 subtypes. We follow (Lin et al., 2020)'s split to split the 154 documents into 10 dev, 10 test and 134 train documents.

Experimental Setup
Details of our setup are mentioned below. For contextualized word embedding, we use the Hug-gingFace PyTorch implementation of Transformers (Wolf et al., 2019). We train monolingual models on three languages, English, Spanish and Chinese, using the same method as described in 3. We denote the language-specific models as sepBERT lang . We use the out-of-the-box pre-trained transformer models, and fine-tune them with the event data, updating all layers. Our English model uses bertbase-uncased and the Chinese and Spanish model uses bert-base-multilingual-cased with the standard BERT hyperparameters; we ran 20 epochs with 5 seeds each, dropout of 0.3, learning rate of 3 · 10 −5 or 5 · 10 −5 , and training batch sizes of 30 for English, and 15 for Arabic and Spanish. We train our models on various seeds and learning rate combinations, and select the model that performs the best on the development set and report the test score for that model. Our sepBERT model takes about 2 minutes per epoch to train on a k80 GPU.

Metrics
We adopt the formal ACE evaluation criteria to evaluate our model: A trigger is correctly classified if its offsets, event type, and subtype match those of a reference trigger. We use Precision, Recall and F 1 -Measure to score the performance.

Baselines
We create an ablation baseline, stdBERT, where we formulate the task as sequence labeling, but train without the SEP objective. This is a standard out-of-the-box sequence labeling model. We compare our model with the following external English baselines: Feature-based: Li et al. (2013) is the best performing feature-based model.
Neural net, pre-transformer: JRNN (Nguyen et al., 2016) is a joint RNN based system. BiL-STM+GAN (Hong et al., 2018) is an ED only NN model. GCN-ED (Nguyen and Grishman, 2018) is a joint model that uses graph CNN to model dependency trees to capture event information.
Transformer-based: DMBERT+Boot (Wang et al., 2019) uses bert-base-uncased and models GANs to generate more training data for ED. They also have a vanilla baseline DMBERT, which does not use additional training data. BERT-CRF (M'hamdi et al., 2019) is an ED only transformer based model that uses a CRF layer on top of bertbase model. ONEIE (Lin et al., 2020) is a joint transformer based model, that extracts mentions, events and relations at the same time. They use bert-large-cased for the English model. We report their score on what they call ACE-E+ dataset, as that matches our data.
For Chinese, we compare our model with the following external baselines: Language dependent: Rich-C (Chen and Ng, 2012) is a feature-based model that adds language specific features to improve performance. NPN (Lin et al., 2018) is a Chinese only Neural Net model that learns a hybrid representation for each character from both characters and words.
Language independent: HNN (Feng et al., 2018) is a BiLSTM based Neural Net model. ONEIE (Lin et al., 2020) is a transformer based model that uses bert-base-multilingual-cased.
*BERT-CRF* (M'hamdi et al., 2019) although we have listed BERT-CRF Chinese language scores, their numbers are not comparable to ours as they "limit the maximum sequence length of sentences to 128, padding or cutting otherwise." This leads to reduction in the ground truth making their total event count 2521, when there are 3333 total events in the Chinese data set. 1 Our Spanish baseline is ONEIE.   Table 1 shows our results. The sepBERT model consistently achieves state-of-the-art results across languages. Some baselines systems, annotated with J in Table 1 are joint event extraction model, optimizing a global objective over event triggers and arguments. Our model outperforms them and the Event Detection only models like DMBERT, BERT-CRF, BiLSTM+GAN with an average gain more than 2F points. On English, our ablation baseline stdBERT en , performs better than the feature-based model and the JRNN model by a good margin. It performs at par with the GCN-ED and BiLSTM+GAN, but does worse than the tranformer based baselines. Our full model sepBERT en performs 1F point better than BERT-CRF, and 1.8F points better than DMBERT, both of which are transformer based methods. It performs 1.2F points better than DM-BERT+Boot, system that augments the training data, and more than 3F points better than the more elaborate neural net models GCN-ED and BiLSTM-GAN. It is 3.5F points better than ONEIE, which is a larger model built using bert-large.

Results and Discussion
Similarly, on Chinese our ablation baseline stdBERT zh does better than all other models, except transformer based ONEIE. sepBERT zh is again the best model, performing 1.1F points better than ONEIE, also based on bert-base-multilingual, and 1.9+F points better than language specific models like NPN and Rich-C. It is also 3.7F points better than the other language independent baseline HNN. *As mentioned in 4.3 BERT-CRF is not comparable with our model.
On Spanish sepBERT es is 5.2F points better than ONEIE. The stdBERT es also scores 3.8F higher than ONEIE.
Results show that the stdBERT baseline has a lower precision than sepBERT. By comparing their output we see that stdBERT en produces 30% more false positives than sepBERT en . This suggests that the added event presence objective helps make sepBERT en more precise. To further investigate, we measure the sentence level event prediction accuracy of both systems, i.e. we measure ratio of number of times system is correct in predicting presence of an event with total number of times a system predicts any event in a sentence. We find for all languages, sepBERT is more accurate than stdBERT, by as much as 3 points, suggesting the SEP loss works as desired.
We also find that both sepBERT en and stdBERT en mis-classify around 5% of the events, i.e., they label a trigger of type x as y, where both x,y are valid trigger types and not "O". This suggests that the added objective does not make the model more accurate on the identified events.

Conclusion
In this paper, we proposed a method for event trigger extraction (sepBERT), which formulates the task as sequence labeling to capture multi-word triggers. With the help of an added auxiliary objective, sepBERT achieves the state-of-the-art performance. We show that our model effectively uses the pre-trained Transformer without additional features or layers on top, performing better than more elaborate Transformer based models. Moreover, our model can be applied to different languages and can still achieve the state-of-the-art performance in those languages, attaining an average gain of 2 absolute F 1 points over past best performing models.