We focus on improving name tagging for low-resource languages using annotations from related languages. Previous studies either directly project annotations from a source language to a target language using cross-lingual representations or use a shared encoder in a multitask network to transfer knowledge. These approaches inevitably introduce noise to the target language annotation due to mismatched source-target sentence structures. To effectively transfer the resources, we develop a new neural architecture that leverages multilevel adversarial transfer: (1) word-level adversarial training, which projects source language words into the same semantic space as those of the target language without using any parallel corpora or bilingual gazetteers, and (2) sentence-level adversarial training, which yields language-agnostic sequential features. Our neural architecture outperforms previous approaches on CoNLL data sets. Moreover, on 10 low-resource languages, our approach achieves up to 16% absolute F-score gain over all high-performing baselines on cross-lingual transfer without using any target-language resources. 1 1 Introduction Low-resource language name tagging is an important but challenging task. An effective solution is to perform cross-lingual transfer, by leveraging the annotations from high-resource languages. Most of these efforts achieve cross-lingual annotation projection based on bilingual parallel corpora combining with automatic word alignment (Yarowsky et al., 2001; Wang et al., 2013; Fang and Cohn, 2016; Ehrmann et al., 2011; Ni et al., 2017), bilingual gazetteers (Feng et al., 2017; Zirikly 1 Our programs will be released at https://github. com/wilburOne/AdversarialNameTagger and Hagiwara, 2015), cross-lingual word embedding (Fang and Cohn, 2017; Wang et al., 2017; Huang et al., 2018), or cross-lingual Wikifica-tion (Kim et al., 2012; Nothman et al., 2013; Tsai et al., 2016; Pan et al., 2017), but these resources are still only available for dozens of languages. Recent efforts on multi-task learning model each language as one single task while all the tasks share the same encoding layer (Yang et al., 2016, 2017; Lin et al., 2018). These methods can transfer knowledge via the shared encoder without using bilingual resources. However, different languages usually have different underlying sequence structures, as shown in Figure 1. Without an explicit constraint, the encoder is not guaranteed to extract language-independent sequential features. Moreover, when the size of annotated resources is not balanced, the encoder is likely to be biased toward the resource-dominant language. NED: ENG: ESP: The European Union 5 ' s competition policy 3 has been of central importance 4 since European integration 2 began 1 . La poltica de competencia 3 de la Unin Europea 5 ha sido de central importancia 4 desde que se inici 1 la integracin europea 2 . Sedert het begin 1 van de Europese integratie 2 is het mededingingsbeleid 3 van groot belang 4 voor de Europese Unie 5 . Considering these challenges, we develop a new neural architecture which can effectively transfer resources from source languages to improve target language name tagging. Our neural architecture is built upon a state-of-the-art sequence tagger: bi-directional long short-term memory as input to conditional random fields (Bi-LSTM-CRF) (Lam-ple et al., 2016; Huang et al., 2015; Ma and ... ... Target Language Source Language ... Linear Projection Word Discriminator Sequence Feature Encoder ContextEncoder CRF Name Tagger SequenceDiscriminator C onvo l u ti on a l N e u r a l N e t w o r k s B-PER I-PER O ... O B-GPE Figure 2: Architecture overview. Hovy, 2016), integrated with multi-level adversarial transfer: (1) word level adversarial transfer, similar to Conneau et al. (2017), applying a projection function on the source language and a discriminator to distinguish each word of the target language from that of the source language, resulting in a bilingual shared semantic space; (2) sentence-level adversarial transfer, where a discriminator is trained to distinguish each sentence of the target language from that of the source language, 2 and a sequence encoder is applied to each sentence of both languages to prevent the discriminator from correctly predicting the source of each sentence, yielding language-agnostic sequential features. These features can better facilitate the resource transfer from the source language to the target language. Our contributions are twofold: (1) without requiring any parallel corpora or bilingual gazetteers, the multi-level adversarial approach can efficiently transfer annotated resources from the source language to the target language and improve target language name tagging; (2) In addition to outperforming previous high-performing baselines on CoNLL data sets, we also evaluate cross-lingual name tagging on 10 low-resource languages and achieve up to 16% absolute F-score gain over all baselines when there is no annotated resource for the target language. Figure 2 shows the overview of our neural architecture. It consists of three components: 2 For the name tagging task, sequence' always means sentence.' Cross-lingual word embedding learning with adversarial training: Given pre-trained monolingual word embeddings for a target language t and a source language s , we first apply a mapping function to each word representation from s , then feed both the projected source word representations and the target word representations to a word discriminator to predict the language of each word. If the discriminator cannot distinguish the language of t from the projection of s , then we consider t and the projection of s to be in a shared space. Language-agnostic sequential feature extraction: For each sentence of t and s , we apply a sequence encoder to extract sequential features, and a Convolutional Neural Network (CNN) (Krizhevsky et al., 2012) based sequence discriminator to predict the language source of each sentence. The sequence encoder is trained to prevent the sequence discriminator from correctly predicting the language of each sentence, such that it finally extracts language-agnostic sequential features. Language-independent name tagger The language-agnostic sequential features from both t and s are further fed into a context encoder to better capture and refine contextual information and a conditional random field (CRF) (Lafferty et al., 2001) based name tagger. To better leverage the resources from the source language, our first step is to construct a shared semantic semantic space where the words from the source and target languages are semantically aligned. Without requiring any bilingual gazetteers, recent efforts (Zhang et al., 2017b; Conneau et al., 2017; Chen and Cardie, 2018) explore unsupervised approaches to learn cross-lingual word embeddings and achieve comparable performance to supervised methods. Following these studies, we perform word-level adversarial training to automatically align word representations from s and t . Formally, assume we are given pretrained monolingual word embeddings V t = f v t 1 ; v t 2 ; :::; v tN g 2 RN (cid:2) d t for t , and V s = f v s 1 ; v s 2 ; :::; v sM g 2 RM (cid:2) d s for s , where v ti and v sj are the vector representations of words w ti and w si from t and s , N and M denote the vocabulary sizes, d t and d s denote the embedding dimensionality of t and s respectively. We then apply a mapping function f to project s into the same semantic space as t : e V s = f ( V s ) = V s U (1) where U 2 R d s (cid:2) d t is the transformation matrix. e V s 2 RM (cid:2) d t are the projected word embeddings for s , and (cid:2) f = f (cid:18) f g denotes the set of parameters to be optimized for f . Similar to Xing et al. (2015), Conneau et al. (2017), and Chen and Cardie (2018), we constrain the transformation matrix U to be orthogonal with singular value decomposition (SVD) to reduce the parameter search space: U = AB  ; with A (cid:6) B  = SVD ( e V s V  s ) (2) To automatically optimize the mapping function f without using extra bilingual signals, we introduce a multi-layer perceptron D as a word discriminator, which takes word embeddings of t and projected word embeddings of s as input features and outputs a single scalar. D ( w (cid:3) i ) represents the probability of w (cid:3) i coming from t . The word discriminator is trained by minimizing the binary cross-entropy loss: L wdis = (cid:0) 1 I t ; s (cid:1) I t ; s  i =0 ( y i (cid:1) log ( D ( w (cid:3) i )) + (1 (cid:0) y i ) (cid:1) log (1 (cid:0) D ( w (cid:3) i )) ) ; y i = (cid:14) i (1 (cid:0) 2  ) +  ; where (cid:14) i = 1 when w (cid:3) i is from t and (cid:14) i = 0 otherwise. I t ; s represents the number of words sampled from the vocabulary of t and s together. is a smoothed value added to the positive and negative labels. (cid:2) dis = f (cid:18) D g is the parameter set. The mapping function f and word discriminator D are two adversarial players, thus we flip the word labels and optimize f by minimizing the following loss: L wf = (cid:0) 1 I t ; s (cid:1) I t ; s  i =0 ( (1 (cid:0) y i ) (cid:1) log ( D ( w (cid:3) i )) + y i (cid:1) log (1 (cid:0) D ( w (cid:3) i )) ) ; y i = (cid:14) i (1 (cid:0) 2  ) +  Following the standard training procedures of deep adversarial networks (Goodfellow et al., 2014), we train the word discriminator and the mapping function successively with stochastic gradient descent (SGD) (Bottou, 2010) to minimize L wdis and L wf . Similar to Conneau et al. (2017), after word-level adversarial training, we also adopt a refinement step to construct a bilingual dictionary for the topk most frequent words in the source language 3 based on e V s and V t , and further optimize U with Equation 2 in a supervised way. Once s is projected into the same semantic space as t , we can regard both sentences as coming from one unified language and directly project annotations from s to t . However, name tagging not only relies on word level features, but also on sequential contextual features for entity type classification. Without constraints, the sequence encoder can only extract sequential features for both t and s based on their final training signals while these features are not necessarily beneficial to the target language. Thus, we further design sentence level adversarial transfer to encourage the encoder to extract language-agnostic sequential features. Given a sentence x t = f w t 1 ; w t 2 ; ::: g from t and a sentence x s = f w s 1 ; w s 2 ; ::: g from s , we first use V t and e V s to initialize a vector representation for each w ti and w si . We also apply a character-based CNN (denoted as CharCNN) (Kim et al., 2016) for each language to compose a word representation from its characters. For each word, we 3 We set k=15,000 in our experiment. concatenate its word representation and character based representation. Then we feed the sequence of vector representations into a weight sharing Bi-LSTM encoder E to obtain sequential features H t = f h t 1 ; h t 2 ; ::: g and H s = f h s 1 ; h s 2 ; ::: g for x t and x s respectively. The parameter set of optimizing both language-dependent CharCNN and the sequence encoder can be denoted as (cid:2) e = f (cid:18) CharCNN t ; (cid:18) CharCNN s ; (cid:18) E g . Based on these sequential features, we use a sequence discriminator to predict the language source of each sentence. Given a sentence x (cid:3) and its sequential features H = f h (cid:3) 1 ; h (cid:3) 2 ; ::: g from E , we first apply a language-independent CNN with max-pooling to get an overall vector representation for x (cid:3) , then feed it into another multi-layer perceptron, ~ D , to predict the probability that x (cid:3) comes from language t . The sequence discriminator is trained by minimizing the following binary cross-entropy loss: L xdis = (cid:0) 1 ~ I t ; s (cid:1) ~ I t ; s  i =0 ( ~ y i (cid:1) log ( ~ D ( x (cid:3) i )) + (1 (cid:0) ~ y i ) (cid:1) log (1 (cid:0) ~ D ( x (cid:3) i )) ) ; ~ y i =~ (cid:14) i (1 (cid:0) 2 (cid:17) ) + (cid:17) ; where ~ (cid:14) i = 1 if the sentence x (cid:3) i is from t and ~ (cid:14) i = 0 otherwise. ~ I t ; s represents the number of sentences sampled from the whole data set of t and s . (cid:17) is another smoothed value for sequence labels. (cid:2) f dis = f (cid:18) CNN ; (cid:18) ~ D g denotes the parameter set for optimizing the sequence discriminator. The sequence encoder E and the sequence discriminator ~ D are two adversarial players and E is optimized by trying to fool ~ D to correctly predict the language source of each sentence. Thus we flip the sequence labels and optimize E by minimizing the following loss: L xe = (cid:0) 1 ~ I t ; s (cid:1) ~ I t ; s  i =0 ( (1 (cid:0) ~ y i ) (cid:1) log ( ~ D ( x (cid:3) i )) + ~ y i (cid:1) log (1 (cid:0) ~ D ( x (cid:3) i )) ) ; ~ y i =~ (cid:14) i (1 (cid:0) 2 (cid:17) ) + (cid:17) 2.4 Name Tagger Training With the language-agnostic sequential features from E , we can directly combine all annotated Algorithm 1 Multi-level Adversarial Training for Improving Target Language Name Tagging Input: Monolingual pre-trained word embeddings V t for target language t , and V s for source language s . Annotated sentence set  t for t and  s for related language s . 1. for iter = 1 to word _ epoch do 2. for a = 1 to word _ dis _ steps do 3. sample a batch of words b t (cid:24) V t , b s (cid:24) V s 4. loss = L wdis ([ b t ; f ( b s )]) 5. update (cid:2) dis to minimize loss 6. sample a batch of words b  t (cid:24) V t , b  s (cid:24) V s 7. loss  = L wf ([ b  t ; f ( b  s )]) 8. update (cid:2) f to minimize loss 9. build a parallel dictionary with V t and f ( V s ) and refine projected word embeddings e V s = f ( V s ) 10. for iter = 1 to seq _ epoch do 11. sample a batch of sentences ~ b t (cid:24)  t , ~ b s (cid:24)  s 12. extract sequential features from ~ b t , ~ b s with E 13. loss = L xdis ([ E (~ b t ) ; E (~ b s )]) 14. update (cid:2) e , (cid:2) g dis to minimize loss 15. for g = 1 to seq _ tagger _ steps do 16. sample a batch of sequences ~ b  t (cid:24)  t , ~ b  s (cid:24)  s 17. loss  = L xe ([ E (~ b  t ) ; E (~ b  s )]) + L crf ([~ b  t ; ~ b  s ]) 18. update (cid:2) e , (cid:2) c to minimize loss  training data from both t and s to train the name tagger for t . To do so, we feed the sequential features from E to another Bi-LSTM encoder E c to refine the context information for each token, and use a CRF output layer to render predictions for each token, which can effectively capture dependencies among name tags (e.g., an inside-organization token cannot follow a beginning-person token). Specifically, given an input sentence x = f w 1 ; w 2 ; :::w n g , we extract language-agnostic sequential features with E , and further obtain a new sequence of contextual features e H = f ~ h 1 ; ~ h 2 ; :::; ~ h n g with E c . Then we a apply a linear layer  to further convert each ~ h i to a score vector y i , in which each dimension denotes the predicted score for a tag (the starting, inside or outside of a name mention with a pre-defined entity type). Then we feed the sequence of score vectors Y = f y 1 ; y 2 ; :::; y n g into the CRF layer. The score of a sequence of tags Z = f z 1 ; z 2 ; :::; z n g is defined as: Score ( x; Y ; Z ) = n  i =1 ( R z i (cid:0) 1 ;z i + Y i;z i ) where R is a transition matrix and R p;q denotes the binary score of transitioning from tag p to tag q . Y i;z represents the unary score of assigning tag z to the i -th word. Given the annotated sequence of tags Z , the CRF loss is: L crf = log  Z  2 ~ Z e Score ( x; Y ; Z  ) (cid:0) Score ( x; Y ; Z ) where ~ Z is the set of all possible tagging paths. The parameter set for optimizing the name tagger can be denoted as (cid:2) c = f (cid:18) E c ; (cid:18)  ; (cid:18) CRF g . We jointly optimize the sequence encoder E , the context encoder E c and the CRF together by minimizing the loss L  = L xe + L crf , and successively minimize L xdis and L  with SGD. The end-to-end training for our neural architecture is described in Algorithm 1. 3 Experiment 3.1 Data and Experimental Setup We evaluate our methods from multiple settings. We first evaluate our architecture on 10 low-resource languages from the DARPA LORELEI project. The annotations are released by the Linguistic Data Consortium (LDC). 4 Each dataset has four predefined name types: person (PER), organization (ORG), location (LOC) and geo-political entity (GPE). For each target low-resource language, we choose a source language if they are from the same language family or use the same script. To show the impact of resource transfer between distinct languages, we also use English as a source language for each target low-resource language. We create the English annotated resource by combining the TAC-KBP 2015 English Entity Discovery and Linking (Ji et al., 2015) data set and the Automatic Content Extraction (ACE2005) data set. 5 To avoid the impact of parameter initialization, we perform 5-fold cross validation. For each experiment, we run twice and get the averaged F-score. Table 1 shows the statistics of each data set. We also evaluate our approach on high-resource languages. We use Dutch (nl) and Spanish (es) data sets from the CoNLL 2002 (Tjong Kim Sang, 2002) shared task as target languages, and use English (en) data from the CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) shared task as 4 The annotations are from: am (LDC2016E87), ti (LDC2017E39), ar (LDC2016E89), fa (LDC2016E93), om (LDC2017E27), so (LDC2016E91), sw (LDC2017E64), yo (LDC2016E105), ug (LDC2016E70), uz (LDC2016E29) 5 The data sets are LDC2015E103 and LDC2006T06 Language # of Sents # of Tokens # of Names Amharic (am) 4,770 71,399 3,891 Tigrinya (ti) 5,023 95,364 6,201 Arabic (ar) 4,781 80,715 4,937 Farsi (fa) 3,855 72,629 3,966 Oromo (om) 2,987 52,876 4,985 Somali (so) 3,453 78,400 5,571 Swahili (sw) 4,155 96,902 6,044 Yoruba (yo) 1,599 46,084 2,016 Uyghur (ug) 3,961 60,999 2,575 Uzbek (uz) 11,135 177,816 10,937 English (en) 17,936 388,120 23,938 Table 1: Data set statistics for each low-resource language. the source language. All the data sets have four pre-defined name types: PER, ORG, LOC and miscellaneous (MISC). Table 2 shows the statistics of these data sets. For fair comparison, we use the same pretrained word embeddings of English, Dutch and Spanish as Lin et al. (2018), while for each low-resource language we train their word embeddings using the documents from their LDC packages with FastText. 6 Table 3 lists the key hyper-parameters we used in our experiments. We compare our methods with three categories of baseline methods: 7 Monolingual Name Tagging Using monolingual annotations only, the current state-of-the-art name tagging model is the Bi-LSTM-CRF network (Huang et al., 2015; Lample et al., 2016; Ma and Hovy, 2016). 8  Multi-task Learning Lin et al. (2018) apply multi-task learning to boost name tagging performance by introducing additional annotations from source languages using a weight sharing context encoder across multiple languages. Language Universal Representations We apply word adversarial transfer only to project the source language into the same semantic space as the target language, then train the name tagger on the annotations of source and target languages. Word-Adv 1 refers to the approach which is directly trained on the combination of the anno-6 https://fasttext.cc/ 7 All the baselines are trained for 100 epochs 8 For each word, we also combine its word embedding with a CharCNN based representation. tations, while Word-Adv 2 refers to the baseline that is first trained on the target language annotations and then further tuned on the related language annotations. We first evaluate our approach on a cross-lingual transfer setting without using any annotated training data from the target language. We conduct experiments on 8 low-resource languages. Among those, some pairs, such as Amharic (am) and Tigrinya (ti), Oromo (om) and Somali (so), or Yoruba (yo) and Swahili (sw), are from the same language family and are closely related, while some are not, such as Arabic (ar) and Farsi (fa). Since our approach requires some unlabeled sentences from the target language to train the sentence-level discriminator, we entirely remove the annotations from the annotated data set of the target language. Table 4 presents the results. Our approach significantly outperforms the previous methods on all languages. Specifically, compared with the Word-Adv 1 baseline, which only performs word-level adversarial transfer, our approach achieves 10% absolute F-score gain on average, which demonstrates the effectiveness of the sentence-level adversarial transfer. In addition, compared with Lin et al. (2018), who only apply a shared context-encoder to transfer the knowledge, our approach not only includes a language-sharing target Cross-lingual Multitask Our (source) Word-Adv 1 Learning Approach am (ti) 15.19 19.72 26.86 ti (am) 16.20 9.06 29.36 ar (fa) 1.53 3.52 13.83 fa (ar) 2.59 0.91 11.14 om (so) 4.66 3.40 14.14 so (om) 4.12 2.98 20.02 sw (yo) 7.20 5.60 18.25 yo (sw) 13.07 6.14 23.73 Table 4: Cross-lingual transfer when the target language has no resources (F-score %). encoder, but also performs multi-level adversarial training to encourage the semantic alignment of words from both languages and a sequence encoder to extract language-agnostic sequential features. Here we use some Arabic (Farsi) examples to further show the effectiveness of each level of adversarial training in our architecture. Without using any annotated training data from Arabic, both our approach and the Word-Adv 1 baseline successfully identify  ( French ) as a GPE from the Arabic (ar) sentence in Figure 3, since with word-level adversarial training, the semantics of  is well aligned with the GPE names in Farsi annotated data, such as  ( France ),  ( Russia ) and  ( Germany ). However, both the Word-Adv 1 and Lin et al. (2018) baselines fail to identify  ( Algerian ) as a GPE since its top ranked similar words in Farsi include  ( negotiations ),  ( Doha ) and  ( agreement ). With sentence-level adversarial training, our approach successfully captures language-agnostic sequential features, such as   ( or ) usually connects two names with the same type , thus our approach successfully identifies  ( Algerian ) as a GPE name. We also investigate the impact of cross-lingual transfer when the target languages have some annotated resources. For each target low-resource language, we explore the use of a related low-resource language vs. using the high-resource En-target Monolingual Cross-lingual Embedding Multitask Our Approach (related) Bi-LSTM-CRF Word-Adv 1 Word-Adv 2 Learning Multi-Adversarial am (ti) 72.23 72.15 72.01 72.35 73.98 ti (am) 74.68 74.43 74.83 74.71 74.93 ar (fa) 48.92 48.37 47.90 47.53 49.76 fa (ar) 64.35 63.93 64.43 63.21 65.09 om (so) 76.37 76.43 76.19 76.18 77.19 so (om) 77.63 77.31 77.13 77.99 78.15 sw (yo) 77.01 77.31 77.85 77.86 76.28 yo (sw) 68.97 68.89 69.62 70.12 70.59 ug (uz) 68.73 68.53 68.29 68.39 69.46 uz (ug) 74.59 74.21 74.74 74.56 75.37 am (en) 72.23 72.43 71.63 72.22 73.35 ti (en) 74.68 74.61 74.69 74.68 74.80 ar (en) 48.92 48.50 47.91 47.40 50.08 fa (en) 64.35 64.04 64.25 63.44 63.92 om (en) 76..27 76.68 76.53 76.2 77.29 so (en) 77.63 76.67 77.88 77.88 78.21 sw (en) 77.01 77.52 76.84 77.89 77.01 yo (en) 68.97 69.21 69.46 70.43 70.88 ug (en) 68.73 68.14 68.79 68.69 69.06 uz (en) 74.59 73.95 74.46 74.48 74.75 Table 5: Cross-lingual transfer when the target language has resources (F-score %). AR:                 1  2  3   . EN: The deputy prosecutor has ruled that the evidence against those with French 3 or 2 Algerian 1 nationality is mostly sufficient. glish as our source language. Table 5 shows the performance on 10 low-resource languages. Comparing cross-lingual embedding based baselines to the monolingual baseline, we observe that for most low-resource languages, directly adding the annotations from the source language to the target language slightly hurts the model. This suggests that when the training data for the target language is not enough, the model will be very sensitive to noise. The multitask learning based baseline (Lin et al., 2018) performs better than the monolingual baseline only when the target and source languages are very close, such as Amharic (am) and Tigrinya (ti), or Swahili (sw) and Yoruba (yo). By introducing annotated training data from English, the performance of all the baselines becomes worse than the monolingual baseline. Since the script and sequence structure of English is very different from these low-resource languages, the addition of English to the limited target language training data yields a considerably noisy corpus. However, by forcing the sequence encoder to extract language-agnostic features, our approach still achieves better performance than the monolingual baseline for most languages. All of these experiments demonstrate that our approach is more effective in leveraging annotations from other languages to improve target language name tagging. We finally investigate the results when both the source and target languages are all high-resource languages. Table 6 presents the performance on Dutch and Spanish while using English as the source language. Our approach significantly outperforms all the other approaches even when the size of the annotated training data for the target language is huge. We notice that our approach achieves larger improvement on Dutch than Spanish. The reason may be that, compared with Spanish, Dutch is much closer to English (Cutler and Pasveer, 2006). Both English and Dutch are from the same West Germanic branch of the Indo-European language family while Spanish is from the Italic branch. We use Amharic as the target language and Tigrinya as the source language to show the impact of the size of their annotations. Specifically, to explore the impact of the size of target language annotations, we use 0, 10%, 50%, or 100% annotated training data from Amharic. Similarly, to show the effect of the size of source language annotations, for each experiment, we also gradually add 0, 20%, 50%, or 100% annotated training data from Tigrinya. For all experiments, we use the same dev and test set of Amharic. As Figure 4 shows, as we gradually add annotations from the source or target language, the performance can always be improved. When the size of target language annotations is small, such as 400 sentences, we can achieve 5%-30% F-score gain by adding about 4,000 sentences from the source language. When the size of target language annotations is over 2,000 sentences, the improvement is about 2% if we add in about 4,000 sentences from source language annotations. Name tagging methods based on sequence labeling have been widely studied in recent years. Huang et al. (2015) and Lample et al. (2016) propose an effective Bi-LSTM-CRF architecture; the Bi-LSTM encodes previous and following contexts, and the CRF is used for tag prediction. Other studies incorporate a character-level CNN (Ma and Hovy, 2016), global contexts (Zhang et al., 2018), or language models (Liu et al., 2018; Peters et al., 2017, 2018; Devlin et al., 2018) to improve name tagging. In addition, several approaches (Zhang et al., 2016a, 2017a; Al-Badrashiny et al., 2017) attempt to incorporate hand-crafted linguistic features into a Bi-LSTM-CRF to improve low-resource name tagging performance. Recent attempts on cross-lingual transfer for name tagging can be divided into two categories: the first projects annotations from a source language to a target language via parallel corpora (Yarowsky et al., 2001; Wang and Manning, 2013; Wang et al., 2013; Zhang et al., 2016b; Fang and Cohn, 2016; Ehrmann et al., 2011; En-ghoff et al., 2018; Ni et al., 2017), a bilingual gazetteer (Feng et al., 2017; Zirikly and Hagiwara, 2015), Wikipedia anchor links (Kim et al., 2012; Nothman et al., 2013; Tsai et al., 2016; Pan et al., 2017), and language universal representations, including Unicode bytes (Gillick et al., 2016) and cross-lingual word embeddings (Fang and Cohn, 2017; Wang et al., 2017; Huang et al., 2018; Xie et al., 2018). The second is based on multitask learning via a weight sharing encoder (Yang et al., 2016, 2017; Lin et al., 2018). Compared to these studies, our approach not only automatically learns cross-lingual word embeddings without requiring any parallel resources, but also carefully extracts language-agnostic sequential features, yielding better performance. Adversarial training has also been extensively studied and applied for cross-lingual and cross-domain transfer. Several studies (Barone, 2016; Zhang et al., 2017c,b; Conneau et al., 2017; Chen and Cardie, 2018) explore adversarial training to automatically induce bilingual and multilingual word representations without using any parallel corpora or bilingual gazetteers. Adversarial training is also applied to extract language-agnostic (Chen et al., 2016; Zou et al., 2018; Wang and Pan, 2018; Kim et al., 2017a; Muis et al., 2018; Cao et al., 2018) and domain-agnostic features (Kim et al., 2017b; Ganin et al., 2016; Tzeng et al., 2017; Chen et al., 2017; Li et al., 2017; Fu et al., 2017; Bousmalis et al., 2016; Shi et al., 2018) for cross-lingual and cross-domain adaptation. Compared with these methods, our approach combines both word-level and sentence-level adversarial training. We design a new neural architecture which integrates multi-level adversarial transfer into a Bi-LSTM-CRF to improve low-resource name tagging. With word-level adversarial training, it can automatically project the source language into a shared semantic space with the target language without requiring any comparable data or bilingual gazetteers. Moreover, considering the different underlying sequential structures among various languages, we further design a sentence-level adversarial transfer to encourage the sequence encoder to extract language-agnostic features. The experiments show that our approach achieves the state-of-the-art on both CoNLL data sets and 10 low-resource languages. In the future, we will further explore selecting the feature-consistent annotations from the source language and add to the target language, and explore unsupervised pretrained cross-lingual language models (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018; Lample and Conneau, 2019) for cross-lingual low resource name tagging. This research is based upon work supported in part by U.S. DARPA LORELEI Program # HR0011-15-C-0115, and the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract # FA8650-17-C-9116, and ARL NS-CTA No. W911NF-09-2-0053. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
result abstain abstain objective result result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain other method method method method abstain abstain method abstain abstain method method method abstain method abstain other abstain abstain abstain method method method abstain abstain method method method method abstain method abstain method method other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain abstain abstain abstain method method method method abstain abstain method method method method method abstain method method abstain abstain abstain method method method abstain abstain abstain method other abstain abstain method abstain method abstain abstain abstain method abstain abstain method abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method other other other other other other method other other other method objective abstain method result abstain other other other other