We present a study on leveraging multilingual pre-trained generative language models for zero-shot cross-lingual event argument extraction (EAE). By formulating EAE as a language generation task, our method effectively encodes event structures and captures the dependencies between arguments. We design language-agnostic templates to represent the event argument structures, which are compatible with any language, hence facilitating the cross-lingual transfer. Our proposed model finetunes multilingual pre-trained generative language models to generate sentences that fill in the language-agnostic template with arguments extracted from the input passage. The model is trained on source languages and is then directly applied to target languages for event argument extraction. Experiments demonstrate that the proposed model outperforms the current state-of-the-art models on zero-shot cross-lingual EAE. Comprehensive studies and error analyses are presented to better understand the advantages and the current limitations of using generative language models for zero-shot cross-lingual transfer EAE. Event argument extraction (EAE) aims to recognize the entities serving as event arguments and identify their corresponding roles. As illustrated by the English example in Figure 1, given a trigger word destroyed for a Conflict:Attack event, an event argument extractor is expected to identify commando , Iraq , and post as the event arguments and predict their roles as At-tacker , Place , and Target , respectively. Zero-shot cross-lingual EAE has attracted considerable attention since it eliminates the requirement of labeled data for constructing EAE models in low-resource languages (Subburathinam et al., 2019; Ahmad et al., 2021; Nguyen and Nguyen,  The authors contribute equally. 2021). In this setting, the model is trained on the examples in the source languages and directly tested on the instances in the target languages. Recently, generation-based models 1 have shown strong performances on monolingual structured prediction tasks (Yan et al., 2021; Huang et al., 2021b; Paolini et al., 2021), including EAE (Li et al., 2021; Hsu et al., 2021). These works fine-tune pre-trained generative language models to generate outputs following designed templates such that the final predictions can be easily decoded from the outputs. Compared to the traditional classification-based models (Wang et al., 2019; Wadden et al., 2019; Lin et al., 2020), they better capture the structures and dependencies between entities, as the templates provide additional declarative information. Despite the successes, the designs of templates in prior works are language-dependent, which makes it hard to be extended to the zero-shot cross-lingual transfer setting (Subburathinam et al., 2019; Ahmad et al., 2021). Naively applying such models trained on the source languages to the target languages usually generates code-switching outputs, yielding poor performance for zero-shot 1 We use pre-trained generative language models to refer to pre-trained models with encoder-decoder structure, such as BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and mBART (Liu et al., 2020). For models adapting these pretrained generative models to generate texts for downstream applications, we denote them as generation-based models. cross-lingual transfer, 2 as we will empirically show in Section 5.4. How to design language-agnostic generation-based models for zero-shot cross-lingual structured prediction problems is still an open question. In this work, we present a study that leverage multilingual pre-trained generative models for zero-shot cross-lingual event argument extraction and propose X-GEAR ( Cross -lingual G enerative E vent A rgument extracto R ). Given an input passage and a carefully designed prompt that contains an event trigger and the corresponding language-agnostic template, X-GEAR is trained to generate a sentence that fills in a language-agnostic template with arguments. X-GEAR inherits the strength of generation-based models that captures event structures and the dependencies between entities better than classification-based models. Moreover, the pre-trained decoder inherently identifies named entities as candidates for event arguments and does not need an additional named entity recognition module. The language-agnostic templates prevents the model from overfitting to the source language's vocabulary and facilitates cross-lingual transfer. We conduct experiments on two multilingual EAE datasets: ACE-2005 (Doddington et al., 2004) and ERE (Song et al., 2015). The results demonstrate that X-GEAR outperforms the state-of-the-art zero-shot cross-lingual EAE models. We further perform ablation studies to justify our design and present comprehensive error analyses to understand the limitations of using multilingual generation-based models for zero-shot cross-lingual transfer. Our code is available at https: //github.com/PlusLabNLP/X-Gear 2 Related Work Zero-shot cross-lingual structured prediction. Zero-shot cross-lingual learning is an emerging research topic as it eliminates the requirement of labeled data for training models in low-resource languages (Ruder et al., 2021; Huang et al., 2021a). Various structured prediction tasks have been studied, including named entity recognition (Pan et al., 2017; Huang et al., 2019; Hu et al., 2020), dependency parsing (Ahmad et al., 2019b,a; Meng 2 For example, TANL (Paolini et al., 2021) is trained to generate  [Two soldiers|target] were attacked  to represent Two soldiers being a target argument. When directly applying it to Chinese, the ground truth for TANL becomes  [     |target]    , which is a sentence alternating between Chinese and English. et al., 2019), relation extraction (Zou et al., 2018; Ni and Florian, 2019), and event argument extraction (Subburathinam et al., 2019; Nguyen and Nguyen, 2021; Fincke et al., 2021). Most of them are classification-based models that build classi-fiers on top of a multilingual pre-trained masked language models. To further deal with the discrepancy between languages, some of them require additional information, such as bilingual dictionaries (Liu et al., 2019; Ni and Florian, 2019), translation pairs (Zou et al., 2018), and dependency parse trees (Subburathinam et al., 2019; Ahmad et al., 2021; Nguyen and Nguyen, 2021). However, as pointed out by previous literature (Li et al., 2021; Hsu et al., 2021), classification-based models are less powerful to model dependencies between entities compared to generation-based models . Generation-based structured prediction. Several works have demonstrated the great success of generation-based models on monolingual structured prediction tasks, including named entity recognition (Yan et al., 2021), relation extraction (Huang et al., 2021b; Paolini et al., 2021), and event extraction (Du et al., 2021; Li et al., 2021; Hsu et al., 2021; Lu et al., 2021). Yet, as mentioned in Section 1, their designed generating targets are language-dependent. Accordingly, directly applying their methods to the zero-shot cross-lingual setting would result in less-preferred performance. Prompting methods. There are growing interests recently to incorporate prompts on pre-trained language models in order to guide the models' behavior or elicit knowledge (Peng et al., 2019; Sheng et al., 2020; Shin et al., 2020; Schick and Schtze, 2021; Qin and Eisner, 2021; Scao and Rush, 2021). Following the taxonomy in (Liu et al., 2021), these methods can be classified depending on whether the language models' parameters are tuned and on whether trainable prompts are introduced. Our method belongs to the category that fixes the prompts and tunes the language models' parameters. Despite the flourish of the research in prompting methods, there is only limited attention being put on multilingual tasks (Winata et al., 2021). We focus on zero-shot cross-lingual EAE. Given an input passage and an event trigger, an EAE 4634 Multilingual Generative Model Input Passage <SEP> Prompt Five Iraqi civilians, including a woman, were killed Monday when their houseswere hit by a missilefired by the US -led coalitionwarplanes, witnesses said. model identifies arguments and their corresponding roles. More specifically, as illustrated by the training examples in Figure 2, given an input passage x and an event trigger t ( killed ) belonging to an event type e ( Life:Die ), an EAE model predicts a list of arguments a = [ a 1 , a 2 , ..., a l ] ( coalition , civilians , woman , missile , houses ) and their corresponding roles r = [ r 1 , r 2 , .., r l ] ( Agent , Victim , Victim , Instrument , Place ). In the zero-shot cross-lingual setting, the training set X train = { ( x i , t i , e i , a i , r i ) } Ni =1 belongs to the source languages while the testing set X test = { ( x i , t i , e i , a i , r i ) } Mi =1 are in the target languages. Similar to monolingual EAE, zero-shot cross-lingual EAE models are expected to capture the dependencies between arguments and make structured predictions. However, unlike monolingual EAE, zero-shot cross-lingual EAE models need to handle the differences (e.g., grammar, word order) between languages and learn to transfer the knowledge from the source languages to the target languages. We formulate zero-shot cross-lingual EAE as a language generation task and propose X-GEAR , a Cross -lingual G enerative E vent A rgument extracto R that is illustrated in Figure 2. There are two challenges raised by this formulation: (1) The input language may vary during training and testing; (2) The generated output strings need to be easily parsed into final predictions. Therefore, the output strings have to reflect the change of the input language accordingly while remaining well-structured. We address these challenges by designing language-agnostic templates . Specifically, given an input passage x and a designed prompt that contains the given trigger t , its event type e , and a language-agnostic template , X-GEAR learns to generate an output string that fills in the language-agnostic template with information extracted from input passage. The language-agnostic template is designed in a structured way such that parsing the final argument predictions a and role predictions r from the generated output is trivial. Moreover, since the template is language-agnostic, it facilitates cross-lingual transfer. X-GEAR fine-tunes multilingual pre-trained generative models, such as mBART-50 (Tang et al., 2020) or mT5 (Xue et al., 2021), and augments them with a copy mechanism to better adapt to input language changes. We present its details as follows, including the language-agnostic templates, the target output string, the input format, and the training details. We create one language-agnostic template T e for each event type e , in which we list all possible associated roles 3 and form a unique HTML-tag-style template for that event type e . For example, in Figure 2, the Life:Die event is associated with four roles: Agent , Victim , Instrument , and Place . Thus, the template for Life:Die events is designed as: 3 The associated roles can be obtained by skimming training data or directly from the annotation guideline if provided. For ease of understanding, we use English words to present the template. However, these tokens ( [None] , <Agent>, </Agent>, <Victim>, etc.) are encoded as special tokens 4 that the pre-trained models have never seen and thus their representations need to be learned from scratch. Since these special tokens are not associated with any language and are not pre-trained, they are considered as language-agnostic . X-GEAR learns to generate target output strings that follow the form of language-agnostic templates. To compose the target output string for training, given an instance ( x , t , e , a , r ) , we first pick out the language-agnostic template T e for the event type e and then replace all  [None]  in T e with the corresponding arguments in a according to their roles r . If there are multiple arguments for one role, we concatenate them with a special token  [and] . For instance, the training example in Figure 2 has two arguments ( civilians and woman ) for the Victim role, and the corresponding part of the output string would be < Victim > civilians [and] woman < /Victim > . If there are no corresponding arguments for one role, we keep  [None]  in T e . By applying this rule, the full output string for the training example in Figure 2 becomes < Agent > coalition < /Agent >< Victim > civilians[and] woman < /Victim >< Instrument > missile < /Instrument > < Place > houses < /Place > . Since the output string is in the HTML-tag style, we can easily decode the argument and role predictions from the generated output string via a simple rule-based algorithm. As we mentioned previously, the key for the generative formulation for zero-shot cross-lingual EAE is to guide the model to generate output strings in the desired format. To facilitate this behavior, we feed the input passage x as well as a prompt to X-GEAR , as shown by Figure 2. The prompt contains all 4 In fact , the special tokens can be replaced by any other format, such as <token1> or </token1>. Here, we use <Agent> and </Agent> to highlight that arguments between these two special tokens are corresponding to the Agent role. valuable information for the model to make predictions, including a trigger t and a language-agnostic template T e . Notice that we do not explicitly include the event type e in the prompt because the template T e implicitly contains this information. In Section 6.1, we will show the experiments on explicitly adding event type e to the prompt and discuss its influence on the cross-lingual transfer. To enable X-GEAR to generate sentences in different languages, we resort multilingual pre-trained generative model to be our base model, which models the conditional probability of generating a new token given the previous generated tokens and the input context to the encoder c , i.e, Copy mechanism. Although the multilingual pre-trained generative models can generate sequences in many languages, solely relying on them may result in generating hallucinating arguments (Li et al., 2021). Since most of the tokens in the target output string appear in the input sequence, 5 we augment the multilingual pre-trained generative models with a copy mechanism to help X-GEAR better adapt to the cross-lingual scenario. Specifically, we follow See et al. (2017) to decide the conditional probability of generating a token t as a weighted sum of the vocabulary distribution computed by multilingual pre-trained generative model P gen and copy distribution P copy PX-GEAR ( x i = t | x <i , c ) = w copy  P copy ( t )+(1  w copy )  P gen ( x i = t | x <i , c ) where w copy  [0 , 1] is the copy probability computed by passing the decoder hidden state at time step i to a linear layer. As for P copy , it refers to the probability over input tokens weighted by the cross-attention that the last decoder layer computed (at time step i ). Our model is then trained end-to-end with the following loss: L =  log (cid:88) i PX-GEAR ( x i | x <i , c ) . We consider two commonly used event extraction datasets: ACE-2005 and ERE. We consider En-5 Except for the special tokens [and] and [None] . glish, Arabic, and Chinese annotations for ACE-2005 (Doddington et al., 2004) and follow the preprocessing in Wadden et al. (2019) to keep 33 event types and 22 argument roles. ERE (Song et al., 2015) is created by the Deep Exploration and Filtering of Test program. We consider its English and Spanish annotations and follow the preprocessing in Lin et al. (2020) to keep 38 event types and 21 argument roles. Detailed statistics and preprocessing steps about the two datasets are in Appendix A. Notice that prior works working on the zero-shot cross-lingual transfer of event arguments mostly focus on event argument role labeling (Subburathi-nam et al., 2019; Ahmad et al., 2021), where they assume ground truth entities are provided during both training and testing. In their experimental data splits, events in a sentence can be scattered in all training, development, and test split since they treat each event-entity pair as a different instance. In this work, we consider event argument extraction (Wang et al., 2019; Wadden et al., 2019; Lin et al., 2020), which is a more realistic setting. We follow previous work (Lin et al., 2020; Ahmad et al., 2021) and consider the argument classification F1 score to measure the performance of models. An argument-role pair is counted as correct if both the argument offsets and the role type match the ground truth. Given the ground truth arguments a , ground truth roles r , predicted arguments  a , and predicted roles  r , the argument classification F1 score is defined as the F1 score between the set { ( a i , r i ) } and the set { ( a j ,  r j ) } . For every model, we experiment with three different random seeds and report the average results. We compare the following models and their implementation details are listed in Appendix B. OneIE (Lin et al., 2020), the state-of-the-art for monolingual event extraction, is a classification-based model trained with multitasking, including entity extraction, relation extraction, event extraction, and event argument extraction . We simply replace its pre-trained embedding with XLM-RoBERTa-large (Conneau et al., 2020) to fit the zero-shot cross-lingual setting. Note that the multi-task learning makes OneIE require additional annotations , such as named entity annotations and relation annotations. CL-GCN (Subburathinam et al., 2019) is a classification-based model for cross-lingual event argument role labeling (EARL). It considers dependency parsing annotations to bridge different languages and use GCN layers (Kipf and Welling, 2017) to encode the parsing information. We follow the implementation of previous work (Ahmad et al., 2021) and add two GCN layers on top of XLM-RoBERTa-large. Since CL-GCN focuses on EARL tasks, which assume the ground truth entities are available during testing, we add one name entity recognition module jointly trained with CL-GCN. GATE (Ahmad et al., 2021), the state-of-the-art model for zero-shot cross-lingual EARL, is a classification-based model which considers dependency parsing annotations as well. Unlike CL-GCN, it uses a Transformer layer (Vaswani et al., 2017) with modified attention to encode the parsing information. We follow the original implementation and add two GATE layers on top of pre-trained multilingual language models. 6 Similar to CL-GCN, we add one name entity recognition module jointly trained with GATE. TANL (Paolini et al., 2021) is a generation-based model for monolingual EAE. Their predicted target is a sentence that embeds labels into the input passage, such as [Two soldiers|target] were attacked , which indicates that  Two soldiers  is a  target  argument. To adapt TANL to zero-shot cross-lingual EAE, we change its pre-trained generative model from T5 (Raffel et al., 2020) to mT5-base (Xue et al., 2021). X-GEAR is our proposed model. We consider three different pre-trained generative language models: mBART-50-large (Tang et al., 2020), mT5-base, and mT5-large (Xue et al., 2021). Table 1 and Table 2 list the results on ACE-2005 and ERE, respectively, with all combinations of source languages and target languages. Note that all the models have similar numbers of parameters 6 To better compare our method with this strong baseline, we consider three different pre-trained multilingual language models for GATE  (1) XLM-RoBERTa-large (2) mBART-50-large (3) mT5-base. For mBART-50-large and mT-base, we follow BART's recipe (Lewis et al., 2020) to extract features for EAE predictions. Specifically, the input passage is fed into both encoder and decoder, and the final token representations are elicited from the decoder output. Comparison to prior generative models. We first observe that TANL has poor performance when transferring to different languages. The reason is that its language-dependent template makes TANL easily generate code-switching outputs, 7 which is a case that pre-trained generative model rarely seen, leading to poor performance. In contrast, X-GEAR considers the language-agnostic templates and achieves better performance for zero-shot cross-lingual transfer. Comparison to classification models. X-GEAR with mT5-base outperforms OneIE, CL-GCN, and GATE on almost all the combinations of the source language and the target language. This suggests that our proposed method is indeed a promising approach for zero-shot cross-lingual EAE. It is worth noting that OneIE, CL-GCN, and GATE require an additional pipeline named entity recognition module to make predictions. Moreover, CL-GCN and GATE need additional dependency 7 Such as the example shown in footnote 2. parsing annotations to align the representations of different languages. On the contrary, X-GEAR is able to leverage the learned knowledge from the pre-trained generative models, and therefore no additional modules or annotations are needed. language models. Interestingly, using mT5-base is more effective than using mBART-50-large for X-GEAR , although they have a similar amount of parameters. We conjecture that the use of special tokens leads to this difference. mBART-50 has different begin-of-sequence (BOS) tokens for different languages. During generation, we have to specify which BOS token we would like to use as the start token. We guess that this language-specific BOS token makes mBART-50 harder to transfer the knowledge from the source language to the target language. Unlike mBART-50, mT5 does not have such language-specific BOS tokens. During generation, mT5 uses the padding token as the start token to generate a sequence. This design is more general and benefit zero-shot cross-lingual transfer. Larger pre-trained models are better. Finally, we demonstrate that the performance of X-GEAR can be further boosted with a larger pre-trained generative language model. As shown by Table 1 and Table2, X-GEAR with mT5-large achieves the best scores on most of the cases. Copy mechanism. We first study the effect of the copy mechanism. Table 3 lists the performance of X-GEAR with and without copy mechanism. It shows improvements in adding a copy mechanism 4638 Model en  xx ar  xx zh  xx xx  en xx  ar xx  zh avg mBART-50-large 51.6 39.8 47.2 48.2 43.2 47.2 46.2 w/o copy 50.9 42.2 49.6 50.6 43.5 48.7 47.6 mT5-base 54.3 41.4 51.4 49.4 46.7 51.0 49.1 w/o copy 52.1 39.5 47.6 48.1 42.7 48.5 46.4 mT5-large 56.7 44.8 52.6 53.0 48.9 52.1 51.3 w/o copy 55.1 45.0 51.5 52.0 46.3 53.2 50.5 Table 3: Ablation study on copy mechanism for ACE-2005. when using mT5-large and mT-base. However, interestingly, adding a copy mechanism is not effective for mBART-50. We conjecture that this is because the pre-trained objective of mBART-50 is denoising autoencoding (Liu et al., 2020), and it has already learned to copy tokens from the input. Therefore, adding a copy mechanism is less useful. In contrast, the pre-trained objective of mT5 is to only generate tokens been masked out, resulting in lacking the ability to copy input. Thus, the copy mechanism becomes beneficial for mT5. Including event type in prompts. In Section 4, we mentioned that the designed prompt for X-GEAR consists of only the input sentence and the language-agnostic template. In this section, we discuss whether explicitly including the event type information in the prompt is helpful. We consider three ways to include the event type information:  English tokens . We put the English version of the event type in the prompt even if we are training or testing on non-English languages, for example, using Attack for the event type Attack . Translated tokens . For each event type, we prepare the translated version of that event type token. For example, both Attack and   represents the Attack event type. During training or testing, we decide the used token(s) according to the language of the input passage. Since all the event types are written in English in ACE-2005 and ERE, we use an off-the-self machine translation tool to perform the translation. Special tokens . We create a special token for every event type and let the model learn the representations of the special tokens from scratch. For instance, we use <-attack-> to represent the Attack event type. Table 4 shows the results. In most cases, including event type information in the prompt decreases Model en  xx ar  xx zh  xx xx  en xx  ar xx  zh avg X-GEAR (mT5-base) 54.3 41.4 51.4 49.4 46.7 51.0 49.1 w/ English Tokens 53.3 39.3 52.3 49.2 46.5 49.2 48.3 w/ Translated Tokens 51.7 40.4 52.2 49.8 45.6 48.8 48.1 w/ Special Tokens 52.3 39.7 51.8 49.0 45.4 49.3 47.9 Table 4: Ablation study on including event type information in prompts for ACE-2005. the performance. One reason is that one word in a language can be mapped to several words in an-other language. For example, the Life event type is related to Marry , Divorce , Born , and Die four sub-event types. In English, we can use just one word Life to cover all four sub-event types. However, In Chinese, when talking about Marry and Divorce , Life should be translated to    ; when talking about Born and Die , Life should be translated to    . This mismatch may cause the performance drop when considering event types in prompts. We leave how to efficiently use event type information in the cross-lingual setting as future work. Influence of role order in templates. The order of roles in the designed language-agnostic templates can potentially influence performance. When designing the templates, we intentionally make the order of roles close to the order in natural sentences. 8 To study the effect of different orders, we train X-GEAR with templates with different random orders and report the results in Table 5. X-GEAR with random orders still achieve good performance but slightly worse than the original order. It suggests that X-GEAR is not very sensitive to different templates while providing appropriate order of roles can lead to a small improvement. to facilitate the cross-lingual transfer. To further validate the effectiveness of the language-agnostic template. We conduct experiments using English tokens as the templates. Specifically, we set format Agent: [None] < SEP > Victim: [None] < SEP > Instrument: [None] < SEP > Place: [None] to be the template for Life:Die events. Hence, for non-English instances, the targeted output string is a code-switching sequence. Table 6 lists the results. We can observe that applying language-agnostic templates bring X-GEAR 2.3 F1 scores improvements in average. We perform error analysis on X-GEAR (mT5-base) when transferring from Arabic to English and transferring from Chinese to English. For each case, we sample 30 failed examples and present the distribution of various error types in Figure 3. Errors on both monolingual and cross-lingual models. We compare the predicted results from X-GEAR (ar  en) with X-GEAR (en  en), or from X-GEAR (zh  en) with X-GEAR (en  en). If their predictions are similar and both of them are wrong when compared to the gold output, we classify the error into this category. To overcome the errors in this category, the potential solution is to improve monolingual models for EAE tasks. Over-generating. Errors in this category happen more often in X-GEAR (ar  en). It is likely because the entities in Arabic are usually much longer than that in English when measuring by the number of sub-words. Based on our statistics, the average entity span length is 2.85 for Arabic and is 2.00 for English (length of sub-words). This leads to the natural for our X-GEAR (ar  en) to overly generate some tokens even though they have captured the correct concept. An example is that the model predicts The EU foreign ministers , while the ground truth is ministers . Label disagreement on different language splits. The annotations for the ACE dataset in different language split contain some ambiguity. For example, given sentence He now also advocates letting in U.S. troops for a war against Iraq even though it is a fellow Muslim state. and the queried trigger war , the annotations in English tends to label Iraq as the Place where the event happen, while similar situations in other languages will mark Iraq as the Target for the war. Grammar difference between languages. An example for this category is ... Blackstone Group would buy Vivendi's theme park division, including Universal Studios Hollywood ... and the queried trigger buy . We observe that X-GEAR (ar  en) predicts Videndi as the Artifact been sold and division is the Seller , while X-GEAR (en  en) 4640 can correctly understand that Videndi are the Seller and division is the Artifact . We hypothesize the reason being the differences between the grammar in Arabic and English. The word order of the sentence Vivendi's theme park division in Arabic is reversed with its English counterpart, that is, theme park division will be written before Vivendi in Arabic. Such difference leads to errors in this category. Generating words not appearing in the passage. In X-GEAR (zh  en), we observe several cases that generate words not appearing in the passage. There are two typical situations. The first case is that X-GEAR (zh  en) mixes up singular and plural nouns. For example, the model generates stu-dios as prediction while only studio appears in the passage. This may be because Chinese does not have morphological inflection for plural nouns. The second case is that X-GEAR (zh  en) will generate random predictions in Chinese. Generating correct predictions but in Chinese. This is a special case of Generating words not appearing in the passage . In this category, we observe that although the prediction is in Chinese (hence, a wrong prediction), it is correct if we translate the prediction into English. Among all the errors, we highlight two specific categories  Generating words not appearing in the passage and Generating correct predictions but in Chinese . These errors can be resolved by applying constrained decoding (Cao et al., 2021) to force all the generated tokens to appear input. Table 7 presents the result of X-GEAR with constrained decoding. We observe that adapting such constraints indeed helps the cross-lingual transferability, yet it also hurts the performance in some monolingual cases. We conduct a qualitative inspection of the predictions. The observation is that constrained decoding algorithm although guarantees all generated tokens appearing in the input, the coercive method breaks the overall sequence distribution that learned. Hence, in many monolingual examples, once one of the tokens is corrected by constrained decoding, its following generated sequence changes a lot, while the original predicted suffixed sequence using beam decoding are actually correct. This leads to a performance decrease. 9 9 Indeed, a similar situation happens to cross-lingual cases; Model monolingual cross-lingual average all X-GEAR (mBART-50-large) 63.9 37.4 46.2 w/ constrained decoding 62.4 37.6 45.9 X-GEAR (mT5-base) 67.8 39.7 49.1 w/ constrained decoding 67.0 39.9 48.9 X-GEAR (mT5-large) 69.7 42.2 51.3 w/ constrained decoding 68.8 43.1 51.6 Table 7: Results of applying constrained decoding. We present the first generation-based models for zero-shot cross-lingual event argument extraction. To overcome the discrepancy between languages, we design language-agnostic templates and propose X-GEAR , which well capture output dependencies and can be used without additional named entity extraction modules. Our experimental results show that X-GEAR outperforms the current state-of-the-art, which demonstrates the potential of using a language generation framework to solve zero-shot cross-lingual structured prediction tasks. We thank anonymous reviewers for their helpful feedback. We thank the UCLA PLUSLab and UCLA-NLP group for the valuable discussions and comments. We also thank Steven Fincke, Shantanu Agarwal, and Elizabeth Boschee for their help on data preparation in Arabic. This work is supported in part by the Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, and research awards sponsored by CISCO and Google. Our proposed models are based on the multilingual pre-trained language model that is trained on a large text corpus. It is known that the pre-trained language model could capture the bias reflecting the training data. Therefore, our models can potentially generate offensive or biased content learned by the pre-trained language model. We suggest carefully examining the potential bias before deploying our model in any real-world applications. however, since the original performance for cross-lingual transfer is not high enough, the benefits of correcting tokens are more significant than this drawback.
method method method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method result abstain objective abstain abstain abstain abstain method abstain method other abstain other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain method method abstain method abstain method method method abstain method abstain method method method abstain abstain abstain other abstain method method method abstain abstain method abstain abstain method method abstain abstain abstain method abstain method abstain abstain abstain method method abstain abstain method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain method abstain method method abstain other abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other objective objective objective other other other other abstain abstain method abstain abstain