Zero-shot sequence labeling aims to build a sequence labeler without human-annotated datasets. One straightforward approach is utilizing existing systems (source models) to generate pseudo-labeled datasets and train a target sequence labeler accordingly. However, due to the gap between the source and the target languages/domains, this approach may fail to recover the true labels. In this paper, we propose a novel unified framework for zero-shot sequence labeling with minimum risk training and design a new decomposable risk function that models the relations between the predicted labels from the source models and the true labels. By making the risk function trainable, we draw a connection between minimum risk training and latent variable model learning. We propose a unified learning algorithm based on the expectation maximization (EM) algorithm. We extensively evaluate our proposed approaches on cross-lingual/domain sequence labeling tasks over twenty-one datasets. The results show that our approaches outperform state-of-the-art baseline systems. Sequence labeling is an important task in natural language processing. It has many applications such as Part-of-Speech Tagging (POS) (DeRose, 1988; Toutanova et al., 2003) and Named Entity Recognition (NER) (Ratinov and Roth, 2009; Ritter et al., 2011; Lample et al., 2016; Ma and Hovy, 2016; Hu et al., 2020). Approaches to sequence labeling are mostly based on supervised learning, which relies heavily on labeled data. However, the labeled data is generally expensive and hard to obtain (for low-resource languages/domains), which means that these supervised learning approaches fail in many cases. Learning knowledge from imperfect predictions from other rich-resource sources (such as cross-lingual, cross-domain transfer) (Yarowsky and Ngai, 2001; Guo et al., 2018; Huang et al., 2019; Hu et al., 2021) is a feasible and efficient way to tackle the low-resource problem. It transfers knowledge from rich-resource languages/domains to low-resource ones. One typical approach to this problem is utilizing existing systems to provide predicted results for the zero-shot datasets. However, due to the gap between the source and the target languages/domains, this approach may fail to recover the true labels. Several previous approaches try to alleviate this problem by relying heavily on cross-lingual information (e.g., parallel text (Wang and Manning, 2014; Ni et al., 2017)), labeled data in source languages (Chen et al., 2019), and prior domain knowledge (Yang and Eisenstein, 2015) for different kinds of zero-shot scenarios. However, these approaches are designed to be specific, and might not be generalizable to other kinds of settings where the required resources are expensive to obtain or not available due to data privacy (Wu et al., 2020). Instead, we want a learning framework that can address the zero-shot learning problem in a unified perspective. In this work, we consider two widely explored settings in which we have access to: 1) the imperfect hard predictions (Rahimi et al., 2019; Lan et al., 2020); 2) the imperfect soft predictions (Wu et al., 2020), produced by one or more source models on target unlabeled data , and propose two novel approaches. We start by introducing a novel approach based on the minimum risk training framework. We design a new decomposable risk function parameterized by a fixed matrix that models the relations between the noisy predictions from the source models and the true labels. We then make the matrix trainable, which leads to further expressiveness and connects minimum risk training to learning latent variable models. We propose a learning algorithm based on the EM algorithm, which alternates between updating a posterior distribution and optimizing model parameters. To empirically evaluate our proposed approaches, we extensively conduct experiments on four sequence labeling tasks of twenty-one datasets. Our two proposed approaches, especially the latent variable model, outperform several strong baselines. Given a sentence x = x 1 , . . . , x n , its word representations are extracted from the pre-trained embeddings and passed into a sentence encoder such as BiLSTM, Convolutional Neural Networks (CNN) and multilingual BERT (Devlin et al., 2019) to obtain a sequence of contextual features. Without considering the dependencies between predicted labels, the Softmax layer computes the conditional probability as follows, P  ( y | x ) = n (cid:89) i =1 P  ( y i | x ) Given the gold sequence y  = y  1 , . . . , y  n , the general training objective is to minimize the negative log-likelihood of the sequence, J (  ) =  log P  ( y  | x ) =  n (cid:88) i =1 log P  ( y  i | x ) For simplicity, throughout this paper, we assume that all the sequence labelers are based on the Softmax method. Supervised models fail when labeled data are absent. Learning from imperfect predictions from rich-resource sources is a viable approach to tackle the problem. Generally speaking, there are two settings to obtain the imperfect predictions from: single source and multi source. The simplest single-source approach is to train a single-source model on one source language/domain and use the source model to directly predict labels on the target test data. We name this approach as direct single-source transfer (DT). Another single-source approach is to use the predictions of the source model on a set of unlabeled target data to supervise the training of a target model. With imperfect hard predictions from the source model, the corresponding objective function is the cross-entropy loss between the imperfect hard predictions and the target model's soft predictions, J (  ) =  log P  ( y | x ) =  n (cid:88) i =1 log P  ( y i | x ) where y denotes the pseudo label sequence of x predicted by the source model and  y i is the pseudo label for position i . With imperfect soft predictions from the source model, the corresponding objective function is the KL-divergence (KL) or mean square error (MSE) loss between the imperfect soft predictions and the target model's soft predictions (knowledge distillation, KD) (Wu et al., 2020). For multi-source setup, a simple approach contains the following two steps. The first step is to apply DT with each source language to produce predictions on unlabeled target data. The second step is to mix the predictions from all the source models and perform supervised learning of a target model on the mixed pseudo-labeled dataset. However, the mixed pseudo-labeled dataset can be very noisy because predictions from different source models may contradict each other. Similar to single-source setting, a more effective way is aggregating the soft predictions from multiple sources and doing KD (Wu et al., 2020). In supervised learning, minimum risk training aims to minimize the expected error (risk) concerning the conditional probability, where R ( y  , y ) is the risk function that measures the distance between the gold sequence y  and the candidate sequence y , and Y ( x ) denotes the collection of all the possible label sequences given the sentence x . The risk function can be defined in many ways depending on specific applications, such as the BLEU score in machine translation (Shen et al., 2016). However, in our setting, there are no gold labels to compute R ( y  , y ) . Instead, we assume there are multiple pretrained source models which can be used to predict hard labels, and we define the risk function as R ( y , y ) to measure the difference between pseudo label sequence  y predicted by source models and the candidate sequence y . The objective function becomes, J (  ) = EP  ( y | x ) [ R ( y , y )] = (cid:88) y Y ( x ) P  ( y | x ) R ( y , y ) Conventional minimum risk training is intractable which is mainly due to the combination of two reasons: first, the set of candidate label sequences Y ( x ) is exponential in size and intractable to enumerate; second, the risk function is hard to decompose (or indecomposable). To tackle the problem, we define the risk function as a negative probability  P ( y | y ) that can be fully decomposed by position. The objective function becomes, J (  ) = (cid:88) y Y ( x ) P  ( y | x ) R ( y , y ) =  (cid:88) y Y ( x ) P  ( y | x ) P  (  y | y ) (1) =  n (cid:89) i =1 (cid:88) y i P  ( y i | x ) P  ( y i | y i ) We introduce a matrix   RK  K to model P  ( y i | y i ) , where K is the number of labels. Notice that  here is a fixed matrix that does not change in training. In the general imperfect predictions learning, it is often implicitly assumed that the prediction from a source model is generally better than uniformly selecting a candidate label at random. Given this prior knowledge, we require P  ( y i = k | y i = k ) > 1 K . Therefore, we empirically define matrix  as,  ij = (cid:40)  if i = j , 1   K  1 if i (cid:54) = j where > 1 K is a hyper-parameter. In the implementation, for convenience, we multiply an identity matrix by a hyper-parameter  and then apply Softmax operation to every column to obtain the matrix  . To further explain  , we give an example from the perspective of prediction in Table 1. Given a sentence x = I cried, a label distribution P  ( y | x ) for the sentence, a pseudo label sequence  y = { Pron , Adj } predicted by the source model, and two settings  1 = 0 . 4 and  2 = 1 for  (1) and  (2) respectively, we compute P  ( y i | x )  P  ( y i | y i ) as shown in the table. Since  (2) is an identity matrix, it predicts the label with the largest value at each position. It assigns the wrong label Adj to the word cried as a consequence. On the contrary,  (1) introduces some uncertainties by providing smoothing over the pseudo labels. As a result, it correctly predicts the word cried as Verb . From the perspective of training, which minimizes J (  ) , if  is an identity matrix, then it is a supervised model with  y as the supervision signal; on the other hand, if  is a uniform matrix, then the supervision signal becomes random and training becomes meaningless. Extending to Leverage Soft Predictions Previous works shows that the soft predictions from source models can provide more information than the hard predictions (Hinton et al., 2015; Wu et al., 2020). Our novel approach can also easily leverage this information by simply replacing the one-hot pseudo labels with soft probability distributions from source models. The training objective becomes, J (  )=  n (cid:89) i =1 (cid:88) y i P  ( y i | x ) (cid:88)  y i P s ( y i | x ) P  ( y i | y i ) where P s is the source model's soft predictions. For simplicity, in the rest of this section, we introduce our approaches based on the setup of using one-hot pseudo labels, but all the approaches can be extended to leverage soft predictions in a similar way. In this subsection, we instead use a trainable matrix  to model P  ( y | y ) . We initialize  in the same way as  . Assuming that conditioning on y , x and  y are independent with each other, we find that the non-negative term of equation (1) is a conditional marginal probability defined by a latent variable model in which y is the latent variable. In latent variable model training, we generally optimize the negative conditional log-likelihood, and the objective function becomes, Interpolation In practice, given a pre-defined hyper-parameter  , we combine the fixed P  ( y i | y i ) with the trainable P  ( y i | y i ) to get a new probability, where   [0 , 1] is a hyper-parameter,  is the combined matrix. If  = 1 , it denotes the minimum risk training. Otherwise, it denotes the latent variable model. By modeling the joint distribution over the pseudo labels which are predicted by U source models on the target unlabeled data, we can easily extend our latent variable model to the multi-source setting. The objective function becomes, J ( , )=  n (cid:88) i =1 log (cid:88) y i P  ( y i | x ) U (cid:89) u =1 P  ( y ( u ) i | y i , u ) Our overall architecture of the latent variable model is depicted in Figure 1. 3.4 Optimization In this section, we propose a unified optimization scheme, which is based on the EM algorithm (Dempster et al., 1977) 1 , to learn the parameters 1 Another approach is to perform direct gradient descent optimization, which we find weaker results. We have a discussion on that in the analysis section. of the two proposed approaches. The EM algorithm is widely applied to learn parameters in a large family of models with latent variables such as the Gaussian mixture models. It is an iterative approach that has two steps in every iteration, which are the E-step and the M-step. In the E-step, it optimizes a posterior distribution of the latent variables. In the M-step, it estimates the parameters of the latent variable model according to the posterior distribution. As the single-source setup can be seen as a special case, we focus on the multi-source setup to derive the equations. We first introduce Q ( y ) = (cid:81) i Q ( y i ) as a distribution over the latent variable y , and then we derive the upper bound of J ( , ) as follows, J ( , ) =  n (cid:88) i =1 log (cid:88) y i P  ( y i | x ) U (cid:89) u =1 P  ( y ( u ) i | y i , u ) =  n (cid:88) i =1 log (cid:88) y i Q ( y i ) P  ( y i | x ) U (cid:81) u =1 P  ( y ( u ) i | y i , u ) Q ( y i )  n (cid:88) i =1 (cid:88) y i Q ( y i )log P  ( y i | x ) U (cid:81) u =1 P  ( y ( u ) i | y i , u ) Q ( y i ) (2) =  n (cid:88) i =1 EQ ( y i ) log P  ( y i | x ) U (cid:89) u =1 P  ( y ( u ) i | y i , u )+ C where C is a residual term, and Q ( y i ) stands for Q ( y i = y i ) . The inequation above is derived from Jensen's inequality. To make the bound tight for particular  and  , we derive Q ( y i ) as, Q ( y i )  P  ( y i | x ) U (cid:89) u =1 P  ( y ( u ) i | y i , u ) (3) We sketch our strategy of parameter update in the t -th iteration as follows,  E step , we compute Q ( y i ) using parameters  and  from the ( t  1) -th iteration;  M step , we update parameters  and  together using a gradient-based approach by minimizing the upper bound above. Q ( y i ) is fixed in this step and hence we minimize  n (cid:88) i =1 EQ ( y i ) log P  ( y i | x ) U (cid:89) u =1 P  ( y ( u ) i | y i , u ) we repeat the two steps alternately until convergence. We give an overall process for multi-source setup with unlabeled target data in Algorithm 1. Algorithm 1 Multi-source transfer with latent variable model 1: Input : unlabeled dataset of target T , U pretrained source models {M = M (1) , . . . , M ( U ) } , U trainable matrices {  =  (1) , . . . , ( U ) } and U fixed matrices {  =  (1) , . . . , ( U ) } , hyper-parameter  and  , maximal iterations E for the EM algorithm. 2: Initialize : initialize  and  with the same hyper-parameter  . Initialize {  =  (1) , . . . , ( U ) } using  ,  and  . Initialize an empty pseudo label list  Y , an upper bound loss l m = +  , and an overall loss l e = +  . 3: for u = 1 , . . . , U do 4: Use M ( u ) to obtain the hard/soft label sequence of the unlabeled data T and append the predictions to the list of pseudo label sequences. Y . 5: end for 6: Concatenate the unlabeled data T with all pseudo label collections  Y to form a new training dataset  T . For inference, we use Q ( y ) to obtain lowing Wu et al. (2020), the source model are previously trained on its corresponding training data. We use the BIO scheme for CoNLL and OntoNotes NER tasks and Aspect Extraction. We run each model three times and report the average accuracy for the POS tagging task and F1-score for the other tasks. Cross-Lingual Sequence Labeling We choose three tasks to conduct the cross-lingual sequence labeling task, which are POS tagging, NER, and Aspect Extraction. For the POS tagging task, we use Universal Dependencies treebanks (UD) v2.4 4 and randomly select five anguages together with the English dataset. The whole datasets are English (En), Catalan (Ca), Indonesian (Id), Hindi (Hi), Finnish (Fi), and Russian (Ru). For the Aspect Extraction task, we select the restaurant domain over subtask 1 in the SemEval-2016 shared task (Pon-tiki et al., 2016). For the NER task, we evaluate our models on the CoNLL 2002 and 2003 shared tasks (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003). Cross-Domain Sequence Labeling We use English portion of the OntoNotes (v5) (Hovy et al., 2006), which contains six domains: broadcast conversation (bc), broadcast news (bn), magazine (mz), newswire (nw), and web (wb). Single-source Setup The following approaches are applicable for single-source setup, DT: we use the pre-trained source model to directly predict the pseudo labels on the target unlabeled data. Hard: we use the pseudo labels from DT on the target unlabeled data to train a new model. Hard-Cat: we apply DT with all the source languages/domains, mix the resulting pseudo labels from all the sources on the unlabeled target data, and train a new model. Hard-Vote: we do majority voting at the token level on the pseudo labels from DT with each source and train a new model. 4 https://universaldependencies.org/ CONLL NER ASPECTEXTRACTION English German Dutch Spanish Avg. Both Setups The following approaches are applicable for both single-/multi-source setups,  KD-re: to fairly compare with the the KD approach (Wu et al., 2020) in the same settings (such as source model's cross-lingual ability), we re-implement the KD approach and adapt it to all tasks. MRT: our minimum risk training approach with a fixed matrix  with soft or hard predictions. LVM: our latent variable model with parameter  (containing the fixed matrix  and the trainable matrix  ) with soft or hard predictions. We also provide the reported results from existing approaches for reference. Due to different experiment configuration reasons, directly comparing our approaches to their reported results is generally not fair. For the CoNLL NER tasks, we provide the reported results from Wu et al. (2020). For the cross-domain sequence labeling tasks, we provide the reported results from Lan et al. (2020) who learns a consensus network to aggregate predictions from multiple sources. Hyper-parameter selection in transfer learning is difficult as no labeled dataset is available for the target language. We select the hyper-parameters only on the development set over the English language and directly use the selected hyper-parameters for the other languages. This may result in sub-optimal SINGLESOURCEMULTI-SOURCECAIDHIFIRU Avg. performance but is more realistic. In latent variable model training, the latent variable is generally very flexible, which may result in sub-optimal performance. Therefore, the initialization of the latent variable is very crucial. In practice, we find that the best strategy is to initialize  of  with a large value (e.g., 0 . 9 ) and  of  with a small value (e.g., 0 . 3 ), and anneal  from 1 to 0 . At the early stage of training, this initialization offers a strong prior for the encoder which can keep the encoder from going in a bad direction; and at later stages of training, the warmed-up encoder can better guide the training of  and vice versa. In this way, the encoder and  can achieve a good balance during training. More details of the hyper-parameters can be found in the Appendix A.2. For the single-source setting, we use English as the source language and the others as the unlabeled target languages. In the multi-source setting, we repeat our experiments multiple times, each time with a language as the target and the others as the sources. We evaluate all approaches on the CoNLL, Aspect Extraction, OntoNotes, and POS tagging. We report the results in Table 2, 3 and 4 5 . Observation #1 Our two approaches outperform several strong baselines on all the tasks and all the scenarios (single-/multi-source scenarios with soft/hard predictions), especially the multi-source scenario, which demonstrates the effectiveness of the two proposed approaches. It shows that modeling this kind of relation is fairly important, which 5 We utilize almost stochastic dominance (ASD) test (Dror et al., 2019) to compare the best score of our approaches and the score of the best performing baselines. We mark the the highest score as bold if its superiority is significant ( p < 0 . 05 ) and underline otherwise. helps to recover the true labels from noisy data. Meanwhile, introducing uncertainties for the relations between the predicted labels from the source models and the true labels in both training and prediction processes significantly benefit our approaches. Observation #2 Our LVM approach achieves overall improvements over the MRT approach on all tasks. It suggests that our LVM approach learns the relations between predicted labels from the source models and true labels better than MRT. Other Minor Observations First, all the approaches that use unlabeled target data for training outperform DT. It suggests that leveraging the unlabeled target data (which may contain knowledge of the target language/domain) in training for zero-shot transfer learning does help. Comparing the approaches that leverage soft instead of hard predictions from sources, the former generally outperform the latter. It suggests that soft predictions can still provide useful knowledge for samples with incorrect hard predictions. The reported results from Lan et al. (2020) are significantly worse. We speculate the reason is that they leverage poor embeddings and different encoders (BiLSTM-CRF). KD-re outperforms our approaches on Ca and Id of POS tagging task on the single-source setting, but its advantage is not statistically significant. We conduct the analysis on the multi-source setting with soft predictions from sources for its better performance. Big Data Performance We experiment with our two models and the KD-re baseline on big target training data on the POS tagging task. We ran-86.5 87.5 88.5 1000 10000 100000 Acc KD MRT LVM Figure 2: The multi-source performance of Ca datasets by varying different sizes on the POS tagging task. We randomly select 1000, 10000, and 100000 sentences to train these three approaches, evaluate on the UD test set for each of the three languages respectively, and show the results in Figure 2. It shows that our latent variable model outperforms the other two approaches over all the settings. Though KD outperform MRT with less than 10000 sentences, but MRT has comparable result with enough unlabeled data. Besides, with more unlabeled data used for training, each model further gains a considerable boost. Our two proposed approaches can also be optimized directly by any gradient-based approach, such as the AdamW optimizer (Loshchilov and Hut-ter, 2018). We use the two proposed approaches to compare the performance of the direct gradient-based training strategy and the EM algorithm. We conduct the experiments on our two proposed approaches on CoNLL NER task on the multi-source setting. We show the results in Table 5. It shows that the EM algorithm outperforms direct gradient-based training for our approaches, which is slightly different from previous findings (Berg-Kirkpatrick et al., 2010). Comparison to Hard EM In this part, we compare our optimization strategy (soft-EM) with the hard-EM approach. Instead of computing a dense vector for Q ( y i ) , hard-EM computes a one-hot vector. We conduct the experiments on our two proposed approaches on the CoNLL NER task on the multi-source setting. The results are shown in Table 6. It shows that soft-EM gains slightly improvement over hard-EM on the MRT approach, but differs significantly from hard-EM on our LVM approach. Impact of Matrix  We analyze the relation between the performance and different initialization of  . We experiment with the MRT approach in the single-source setup with soft predictions on NER tasks and Figure 3 shows the results. The best value of  is 2 for De and 3 for the others (resulting in  = 0 . 43 and 0 . 67 respectively 6 ), which shows that the uncertainties introduced by a smooth  can effectively boost the model's performance. On the other hand, setting  to a nearly identity matrix with  = 10 leads to worse scores. Cross-lingual/domain Sequence Labeling Recent works on cross-lingual transfer mainly have two scenarios: the single-source cross-lingual transfer (Yarowsky and Ngai, 2001; Wang and Manning, 2014; Huang et al., 2019) and the multi-source cross-lingual transfer (Tackstrom et al., 2012; Guo et al., 2018; Rahimi et al., 2019; Hu et al., 2021). Wu et al. (2020) propose a knowledge distillation approach to further leveraging unlabeled target data and achieve the state-of-the-art results. Hu et al. (2021) propose a multi-view framework to selectively transfer knowledge from multiple sources by utilizing a small amount of labeled dataset. Cross-domain adaption is widely studied (Steedman et al., 6 The CoNLL NER datasets have 11 labels (9 entity labels, a padding label and an ending label). 2003). Existing works include bootstrapping approaches (Ruder and Plank, 2018), mixture-of-experts (Guo et al., 2018; Wright and Augenstein, 2020), and consensus network (Lan et al., 2020). Other previous work (Kim et al., 2017; Guo et al., 2018; Huang et al., 2019) utilized labeled data in the source domain to learn desired information. However, our proposed approaches do not require any source labeled data or parallel texts. Contextual Multilingual Embeddings Embeddings like mBERT (Devlin et al., 2019), XLM (CONNEAU and Lample, 2019) and XLM-R (Con-neau et al., 2020) which are trained on many languages, make great progress on cross-lingual learning for multiple NLP tasks. Recent works (Wu and Dredze, 2019; Pires et al., 2019) show the strong cross-lingual ability of the contextual multilingual embeddings. In this paper, we propose two approaches to the zero-shot sequence labeling problem. Our MRT approach uses a fixed matrix to model the relations between the predicted labels from the source models and the true labels. Our LVM approach uses trainable matrices to model these label relations. We extensively verify the effectiveness of our approaches on both single-source and multi-source transfer over both cross-lingual and cross-domain sequence labeling problems. Experiments show that MRT and LVM generally bring significant improvements over previous state-of-the-art approaches on twenty-one datasets. by Alibaba Group through Alibaba Innovative Research Program.
abstain abstain abstain objective abstain objective objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain objective objective objective abstain other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method other method other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method method abstain method method method method other method method method method method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain other other other other other other other objective other other objective method method result abstain other