Cross-domain NER is a challenging yet practical problem. Entity mentions can be highly different across domains. However, the correlations between entity types can be relatively more stable across domains. We investigate a multi-cell compositional LSTM structure for multi-task learning, modeling each entity type using a separate cell state. With the help of entity typed units, cross-domain knowledge transfer can be made in an entity type level. Theoretically, the resulting distinct feature distributions for each entity type make it more powerful for cross-domain transfer. Empirically, experiments on four few-shot and zero-shot datasets show our method significantly outperforms a series of multi-task learning methods and achieves the best results. Named entity recognition (NER) is a fundamental task in information extraction, providing necessary information for relation classification (Mooney and Bunescu, 2006), event detection (Popescu et al., 2011), sentiment classification (Mitchell et al., 2013), etc. NER is challenging because entity mentions are an open set and can be ambiguous in the context of a sentence. Due to relatively high cost in manual labeling, cross-domain NER has received increasing research attention. Recently, multi-task learning methods (Yang et al., 2017; Wang et al., 2018, 2019; Zhou et al., 2019; Jia et al., 2019) have achieved great success for cross-domain NER. Other methods such as fine-tuning (Rodriguez et al., 2018), share-private (Cao et al., 2018; Lin and Lu, 2018) and knowledge distill (Yang et al., 2019) also show effectivenesses for cross-domain NER. There are three main source of challenges in cross-domain NER. First, instances of the same type entities can be different across domains. For example, typical person names can include Trump and Clinton in the political news domain, but James and Trout in the sports domain. Second, different types of entities can exhibit different degrees of dissimilarities across domains. For example, a large number of location names are shared in the political news domain and the sports domain, such as Barcelona and Los Angeles, but the case is very different for organization names across these domains. Third, even types of entities can be different across domains. For example, while disease names are a type of entities in the medical domain, it is not so in the biochemistry domain. We investigate a multi-cell compositional LSTM structure to deal with the above challenges by separately and simultaneously considering the possibilities of all entity types for each word when processing a sentence. As shown in Figure 1, the main idea is to extend a standard LSTM structure by using a separate LSTM cell to model the state for each entity type in a recurrent step. Intuitively, the model differs from the baseline LSTM by simultaneously considering all possible entity types. A compositional cell (C cell) combines the entity typed cells (ET cells) for the next recurrent state transition by calculating a weighted sum of each ET cell, where the weight of each ET cell corresponds to the probability of its corresponding entity type. Different from naive parameter sharing on LSTM (Yang et al., 2017), source domain and target domain in our multi-task learning framework share only the ET cells corresponding to the same entity types and the same C cell, but not for the domain-specific ET cells. In this way, our model learns domain-invariant in the entity level. Intuitively, our model addresses the above challenges by modeling entity type sequences more explicity, which are relatively more robust across domains compared with entity instances. For example, the pattern P ERO PERO LOC  can exist in both the political and sports domains, despite ) ( ~ t c ) ( t c ) ( t h ) ( t h ) ( t c ) ( t h )1 (  t c )1 ( t h )( t o )( t i )( t f ) ( t w ) ( t w ) ( ~ t c ) (  t c ) (  t h )1 (  t c )1 (  t h ) (  t c ) (  t h ) (  t h ) ( tl c ) ( ~ t l c )(  t o )(  t i )(  1 t i  )( tl i )( 1 tl i  ) (  t c that the specific PER instances can be different. In addition, thanks to the merging operation at each step, our method effectively encodes multiple entity type sequences in linear time by having a sausage shaped multi-cell LSTM. Thus it allows us to learn distributional differences between entity type chains across domains. This effectively reduces the confusions of different entities when source domain and target domain have different entity types in few-shot transfer, where the target domain has a few training data. In zero-shot transfer where the target domain has no training data, a target-domain LM transfers source-domain knowledge. This knowledge transfer is also in the entity level thanks to the compositional weights which are supervised by gold-standard entity type knowledge in source-domain training. Theoretically, our method creates distinct feature distributions for each entity type across domains, which can give better transfer learning power compared to representation networks that do not explicitly differentiate entity types (  3.4). Empirically, experiments on four few-shot and zero-shot datasets show that our method gives significantly better results compared to standard BiLSTM baselines with the same num-bers of parameters. In addition, we obtain the best resutls on four cross-domain NER datasets. The code is released at https://github.com/ jiachenwestlake/Multi-Cell_LSTM . Given a sentence x = [ x 1 , . . . , x m ] , the vector representation w t for each word x t is the concatenation of its word embedding and the output of a character level CNN, following Yang et al. (2018). A bi-directional LSTM encoder is used to obtain sequence level features h = [ h 1 , . . . , h m ] . We use the forward LSTM component to explain the details in the following subsections. Finally, a CRF layer outputs the label sequence y = l 1 , . . . , l m . We adopt the standard LSTM (Graves and Schmid-huber, 2005) for the baseline. At each time step t ( t  [1 , ..., m ]) , the baseline calculates a current hidden vector h ( t ) based on a memory cell c ( t ) . In particular, a set of input gate i ( t ) , output gate o ( t ) and forget gate f ( t ) are calculated as follows:   i ( t ) o ( t ) f ( t ) (cid:101) c ( t )   =    tanh   (cid:16) W [ h ( t  1) ; w ( t ) ] + b (cid:17) c ( t ) = i ( t ) (cid:12) (cid:101) c ( t ) + f ( t ) (cid:12) c ( t  1) h ( t ) = o ( t ) (cid:12) tanh( c ( t ) ) , (1) where [ W ; b ] are trainable parameters. As shown in Figure 1 (b), we split cell computation in the baseline LSTM unit into l copies, each corresponding to one entity type. These cells are shown in black. A compositional cell (shown in red) is used to merge the entity typed LSTM cells into one cell state for calculating the final hidden vector. In this process, a weight is assigned to each entity type according to the local context. Entity typed LSTM cells (ET cells). Given w ( t ) and  h ( t  1) , the input gate i ( t ) k and the temporary memory cell state (cid:101) c ( t ) k of the k -th ( k  [1 , . . . , l ]) entity typed cells (ET cells) are computed as: (cid:34) i ( t ) k (cid:101) c ( t ) k (cid:35) = (cid:20)  tanh (cid:21) (cid:16) W k [  h ( t  1) ; w ( t ) ] + b k (cid:17) , (2) where the [ W k ; b k ] represent the trainable parameters specific to the k -th ET cell. Then a copy of the compositional memory cell state  c ( t  1) of the previous time step ( t  1) is used to update the temporary memory cell state. c ( t ) k = i ( t ) k (cid:12) (cid:101) c ( t ) k + ( 1  i ( t ) k ) (cid:12)  c ( t  1) (3) The above operations are repeated for l ET cells with the same  c ( t  1) . We finally acquire a list of ET cell states [ c ( t ) 1 , . . . , c ( t ) l ] . Compositional LSTM cell (C cell). For facilitating integration of ET cells, a input gate  i ( t ) and a temporary cell state (cid:101)  c ( t ) of the compositional cell (C cell) are computed similarly to those of the ET cells, but another output gate  o ( t ) is added, which are computed as follows:    i ( t )  o ( t ) (cid:101)  c ( t )   =    tanh   (cid:16)  W [ h ( t  1) ; w ( t ) ] +  b (cid:17) , (4) where [  W ;  b ] are trainable parameters of the C cell. Merging. We use the temporary cell state of the C cell (cid:101)  c ( t ) to weigh the internal representations of ET cells [ c ( t ) 1 , . . . , c ( t ) l ] for obtaining a compositional representation. To this end, additive attention (Dzmitry et al., 2015) is used, which achieves better results in our development compared with other attention mechanism (Vaswani et al., 2017). The temporary memory cell state of the C cell  c ( t )  is a weighted sum of [ c ( t ) 1 , . . . , c ( t ) l ] :  c ( t )  = l (cid:88) k =1  ( t ) k c ( t ) k s.t. l (cid:88) k =1  ( t ) k = 1 (5) The weight  ( t ) k reflects the similarity between (cid:101)  c ( t ) and the k -th ET cell state c ( t ) k . ( t ) k is computed as: I ( t ) k = v (cid:62) tanh( P (cid:101)  c ( t ) + Qc ( t ) k )  ( t ) k = exp( I ( t ) k ) (cid:80) lj =1 exp( I ( t ) j ) , (6) where [ P ; Q ; v ] are trainable parameters. The memory cell state of the C cell is updated as:  c ( t ) =  i ( t ) (cid:12)  c ( t )  + ( 1   i ( t ) ) (cid:12)  c ( t  1) (7) Finally, we obtain the hidden state  h ( t ) :  h ( t ) =  o ( t ) (cid:12) tanh( c ( t ) ) (8) 2.3 Training Tasks Below we discuss the two auxiliary tasks before introducing the main NER task. The auxiliary tasks are designed in addition to the main NER task in order to better extract entity type knowledge from a set of labeled training data for training ET cells and C cell. Formally, denote a training set as D ent = { ( x n , e n ) } Nn =1 , where each training instance consists of word sequence x = [ x 1 , . . . , x m ] and its corresponding entity types e = [ e 1 , . . . , e m ] . Here each entity type e t is a label such as [PER , O, LOC ,. . . ] without segmentation tags (e.g., B/I/E). Entity type prediction. Given the ET cell states of x t : c ( t ) = [  c ( t ) 1   c ( t ) 1 , . . . ,  c ( t ) l   c ( t ) l ] , we define the aligned entity distribution for x t : p ( e k | x t ) = exp { w (cid:62) k c ( t ) k + b k } (cid:80) lj =1 exp { w (cid:62) j c ( t ) j + b j } , (9) Where [ w k ; b k ] are parameters specific to the k -th entity type e k . Attention scoring. Similar to the entity type prediction task, given the attention scores between the temporary C cell and ET cells in Equation 6: I ( t ) = [(  I ( t ) 1 +  I ( t ) 1 ) / 2 , . . . , (  I ( t ) l +  I ( t ) l ) / 2] , we convert the attention scores to entity aligned distributions for x t using softmax : p ( e k | x t ) = exp( I ( t ) k ) (cid:80) lj =1 exp( I ( t ) j ) (11) Similar to the loss of entity type prediction: L atten =  1 |D ent | N (cid:88) n =1 m (cid:88) t =1 log( p ( e nt | x nt )) (12) While entity type prediction brings supervised information to guide the ET cells, attention scoring introduces supervision to guide the C cell. NER. This is the main task across domains. Standard CRFs (Ma and Hovy, 2016) are used. Given h = [  h 1   h 1 , . . . ,  h m   h m ] , the output probability p ( y | x ) over labels y = l 1 , . . . , l m is: p ( y | x )= exp { (cid:80) t ( w l t CRF  h t + b ( l t  1 ,l t ) CRF ) } (cid:80) y (cid:48) exp { (cid:80) t ( w l (cid:48) t CRF  h t + b ( l (cid:48) t  1 ,l (cid:48) t ) CRF ) } , (13) where y (cid:48) represents an arbitary labal sequence, and w l t CRF is a model parameter specific to l t , and b ( l t  1 ,l t ) CRF is a bias specific to l t  1 and l t . The multi-cell LSTM structure above is domain agnostic, and can therefore be used for in-domain NER too. However, the main goal of the model is to transfer entity sequence knowledge across domains, and therefore the ET cells and C cell play more significant roles in the transfer learning setting. Below we introduce the specific roles each cell is assigned in cross-domain settings. Following the common cross-domain setting, we use source-domain NER dataset S ner and the target-domain NER dataset T ner or raw data T lm . The entity type sets of source and target domains are represented as E d , where d  { S, T } , respectively. As shown in Figure 1 (c), our multi-task learning structure follows Yang et al. (2017), which consists of shared embedding layer and shared BiLSTM layer, as well as domain-specific CRF layers. Our method replaces LSTM with multi-cell LSTM, following we introduce the multi-task parameter sharing mechanism in multi-cell LSTM. ET cells. All ET cells { C k } k E S E T in multi-cell LSTM are a composion of entity-specific cells from both source and target domains. For each domain d  { S, T } , the actually used ET cells are the domain-specific subset { C k } k E d , aiming to conserve domain-specific features. C cell. In order to make the source and target domains share the same feature space in a word level, we use a shared C cell  C across domains. To better leverage target-domain knowledge without target-domain NER labeled data, we conduct the auxiliary dictionary matching and language modeling tasks on target-domain raw data T lm = { ( x n ) } . Auxiliary tasks. To better extract entity knowledge from raw data, we use a pre-collected named entity dictionary D e by Peng et al. (2019) to label T lm and obtain a set of entity words D + ent , which are used to train entity prediction task and attention scoring task jointly. Language modeling. Follwing Jia et al. (2019), we use sampling softmax to compute forward LM probability p f ( x t | x <t ) and backward LM probability p b ( x t | x >t ) , respectively: p f ( x t | x <t )= 1 Z exp (cid:16) w (cid:62) x t  h t  1 + b x t (cid:17) p b ( x t | x >t )= 1 Z exp (cid:16) w (cid:62) x t  h t +1 + b x t (cid:17) , (15) where w x and b x are the target word vector and bias, respectively. Z is the normalization item computed by the target word and negative samples. The LM loss function on T lm is: L Tlm =  1 2 |T lm | N,m (cid:88) n,t =1 (cid:110) log( p f ( x nt | x n<t )) + log( p b ( x nt | x n>t )) (cid:111) (16) 3.3 Training Objective Algorithm 1 is the transfer learning algorithm under both supervised and unsupervised domain adaptation settings. Both sourceand target-domain training instances undertake auxiliary tasks and obtain the loss L a , which is a combination of L ent and L atten weighted by  ent and  atten , respectively (line 6). LSDA = (cid:88) d { S,T } (cid:110)  d L dner + L da (cid:111) +  2 (cid:107)  (cid:107) 2 , (17) NER tasks. is the L 2 regularization parameters and  represents the parameters set. Unsupervised domain adaptation. The training objective for UDA is similar to that of SDA, except for using target-domain LM task (line 13) instead of target-domain NER task: LUDA = L Sner + L Tlm + L Sa + L Ta +  2 (cid:107)  (cid:107) 2 (18) 3.4 Theoretical Discussion Below we show theoretically that our method in  2.2 is stronger than the baseline method in  2.1 for domain adaptation. Following Ben-David et al. (2010), a domain is defined as a pair of input distribution D on X and a labeling function y : XY , where Y is a ( l  1) -simplex 1 . According to this definition, < DS , y S > and < DT , y T > represent source and target domains, respectively. A hypothesis is a function h : X{ 1 , ..., l } , which can be a classification model. Target-domain error is defined as the probability h T disagrees with y T , (cid:15) ( h T ) = (cid:15) ( h T , y T ) = E x D T [ | y T  h T ( x ) | ] . The training target for h is to minimize a convex weighted combination of source and target errors, (cid:15)  ( h ) = (cid:15) T ( h ) + (1   ) (cid:15) S ( h ) , where   [0 , 1) is the domain weight, when  = 0 , it is the setting of UDA. Theorem 1 Let h be a hypothesis in class H , then: (cid:15) T ( h )  (cid:15)  ( h ) + (1   ) (cid:0) 12 d H  H ( DS , DT ) +  (cid:1) , where d H  H ( DS , DT ) = 2 sup h (cid:48) ,h (cid:48)(cid:48) H (cid:12)(cid:12)(cid:12) Pr x D S (cid:2) h (cid:48) ( x ) (cid:54) = h (cid:48)(cid:48) ( x ) (cid:3)  Pr x D T (cid:2) h (cid:48) ( x ) (cid:54) = h (cid:48)(cid:48) ( x ) (cid:3) (cid:12)(cid:12)(cid:12) Here  is a constant that values the shared error of the ideal joint hypothesis. In d H  H ( DS , DT ) , sup denotes the supremum of the right term for  h (cid:48) , h (cid:48)(cid:48)  H . Pr x D S [ h (cid:48) ( x ) (cid:54) = h (cid:48)(cid:48) ( x )] denotes the probability according to the distribution DS that h (cid:48) disagrees with h (cid:48)(cid:48) and Pr x D T [ h (cid:48) ( x ) (cid:54) = h (cid:48)(cid:48) ( x )] is similar. Intuitively, the theorem states the upper bound of (cid:15) T ( h ) based on (cid:15)  ( h ) and the distance between DS and DT in the H  H space, which is measured as the discrepancy between the two classifiers h (cid:48) and h (cid:48)(cid:48) . 1 l is the total number of entity types in the source and target domains, such as { O, PER , LOC , ORG , MISC } . Our discussion also makes sense in the case that source domain and target domain have different entity types. The original theorem, however, concerns only one model h for transfer learning. In our supervised settings, in contrast, their CRF layers are specific to the source and target domains, respectively. Below we use h  to denote our overall model with shared multi-cell LSTM model and domain-specific CRF layers. Further, we use h 1 to denote the target domain subsystem that consists of the shared multi-cell LSTM model and the target-specific CRF layer, and h 2 to denote its source counterpart. Theorem 1 can be extended to our settings as follows: Lemma 1 If (cid:15)  ( h  ) = (cid:15) T ( h 1 ) + (1   ) (cid:15) S ( h 2 ) , then: (cid:15) T ( h 1 )  2 (cid:15)  ( h  ) + (1   ) (cid:0) 32 d H  H ( DS , DT ) +   (cid:1) Proof . inequalities, see Appendix A for details. (cid:3) Considering that the upper bounds of (cid:15) T ( h ) ( (cid:15) T ( h 1 ) ), (cid:15)  ( h ) ( (cid:15)  ( h  ) ) and  (   ) are small when training converges, our goal is to reduce d H  H ( DS , DT ) . In particular, we define a model h is a composition function h = g  f , where f represents the multi-cell LSTM model and g represents the CRF layer,  denotes function composition. We assume h (cid:48) and h (cid:48)(cid:48) share the same multi-cell LSTM model, namely h (cid:48) = g (cid:48)  f and h (cid:48)(cid:48) = g (cid:48)(cid:48)  f , we have d H  H ( DS , DT ) =2 sup g (cid:48) ,g (cid:48)(cid:48)  G (cid:12)(cid:12)(cid:12) Pr x D S (cid:2) g (cid:48)  f ( x ) (cid:54) = g (cid:48)(cid:48)  f ( x ) (cid:3)  Pr x D T (cid:2) g (cid:48)  f ( x ) (cid:54) = g (cid:48)(cid:48)  f ( x ) (cid:3) (cid:12)(cid:12)(cid:12) To obtain the supremum of the right term, we may wish to assume that both g (cid:48) and g (cid:48)(cid:48) can classify correctly in the source domain, then d H  H ( DS , DT )  2 sup g (cid:48) ,g (cid:48)(cid:48)  G (cid:12)(cid:12)(cid:12) Pr x D T (cid:2) g (cid:48)  f ( x ) (cid:54) = g (cid:48)(cid:48)  f ( x ) (cid:3) (cid:12)(cid:12)(cid:12) The optimization objective is as follows: min f F sup g (cid:48) ,g (cid:48)(cid:48)  G (cid:12)(cid:12)(cid:12) Pr x D T (cid:2) g (cid:48)  f ( x ) (cid:54) = g (cid:48)(cid:48)  f ( x ) (cid:3) (cid:12)(cid:12)(cid:12) Aiming to min f F d H  H ( DS , DT ) , we decompose the unified feature space into several entity typed distributions using multi-cell LSTM, resulting in that sourceand target-domain features belonging to the same entity type are clustered together. The proof is mainly based on the cluster assumption (Chapelle and Zien, 2005), which is equivalent to the low density separation assumption, states that the decision boundary should lie on a low-density region. According to the cluster assumption, both g (cid:48) and g (cid:48)(cid:48) tend to cross the low-density regions in the shared Dataset Entity Type Size Train Dev Test CoNLL-2003 PER , LOC #Sentence 15.0K 3.5K 3.7K ORG , MISC #Entity 23.5K 5.9K 5.6K Broad Twitter PER , LOC #Sentence 6.3K 1.0K 2.0K ORG #Entity 8.8K 1.7K 4.4K Twitter PER , LOC #Sentence 4.3K 1.4K 1.5K ORG , MISC #Entity 7.5K 2.5K 2.5K BioNLP13PC CHEM , CC #Sentence 2.5K 0.9K 1.7K GGP , etc. feature space of both source and target domains. This results in Pr x D T [ g (cid:48)  f ( x ) (cid:54) = g (cid:48)(cid:48)  f ( x )]  Pr x D S [ g (cid:48)  f ( x ) (cid:54) = g (cid:48)(cid:48)  f ( x )]  0 , which well meets the above optimization objecive. Datasets. We take six publicly available datasets for experiments, including BioNLP13PC and BioNLP13CG (Nedellec et al., 2013), CoNLL-2003 English dataset (Sang and Meulder, 2003), Broad Twitter dataset (Derczynski et al., 2016), Twitter dataset (Lu et al., 2018) and CBS SciTech News dataset (Jia et al., 2019). Statistics of the datasets are shown in Table 1. In unsupervised domain adaptation experiments, 398,990 unlabeled sentences from CBS SciTech News collected by Jia et al. (2019) are used for target-domain LM training, a named entity dictionary from Web resource collected by Peng et al. (2019) is used for target-domain auxiliary tasks training. The CoNLL-2003, Twitter and CBS News have the same four types of entities, namely PER (per-son), LOC (location), ORG (organization) and MISC (miscellaneous). The Broad Twitter dataset consists of three types: PER , LOC and ORG . BioNLP13CG mainly consists of five types, namely CHEM (simple chemical), CC (cellular component), GGP (gene and gene product), SPE (species) and CELL (cell), BioNLP13PC mainly consists of three types: CHEM , CC and GGP . Hyperparameters. We choose NCRF++ (Yang and Zhang, 2018) for developing the models. The multi-task baselines are based on Jia et al. (2019). Our hyperparameter settings largely follow Yang et al. (2018); word embeddings for all models are initialized with PubMed 200-dimension vectors (Chiu et al., 2016) in BioNLP experiments and 20 40 60 80 100 Iteration 0.55 0.60 0.65 0.70 0.75 0.80 Fs c o r e NEREntity Prediction (right) Attention Scoring (right) 0.86 0.88 0.90 0.92 0.94 0.96 0.98 A cc u r a c y GloVe 100-dimension vectors (Pennington et al., 2014) in other experiments. All word embeddings are fine-tuned during training. Character embeddings are randomly initialized. Figure 2 shows the performances of the main target-domain NER task and the auxiliary entity prediction and attention scoring tasks on the development sets of BioNLP13CG and Twitter when the number of training iterations increases. As can be seen from the figure, all the three tasks have the same trend of improvement without potential conflicts between tasks, which shows that all the three tasks take the feature space of the same form. We conduct supervised domain adaptation on BioNLP dataset, Broad Twitter dataset and Twitter dataset, respectively. In particular, for the BioNLP dataset, BioNLP13CG is used as the target-domain NER dataset and BioNLP13PC as the source-domain dataset. These two datasets have some different entity types. In the Broad Twitter dataset, Broad Twitter is used as the target-domain dataset and the CoNLL-2003 as the source-domain dataset. These two datasets have a different entity type MISC . In the Twitter dataset, Twitter is used as the target-domain dataset and the CoNLL-2003 as the source-domain dataset. These two datasets have the same entity types. The overall results are listed in Table 2. Target-domain only settings. CELLLSTM , all of the multi-task models obtain significantly better results on all of the three datasets. This shows the effectiveness of multi-task learning in few-shot transfer. Cross-domain settings. We make comparisons with the traditional parameter sharing mechanism MULTI-TASK (LSTM ) (Yang et al., 2017) together with two improved methods, MULTI-TASK +P GN (Jia et al., 2019), which adds an parameter generation networks (PGN ) to generate parameters for sourceand target-domain LSTMs and MULTITASK +G RAD (Zhou et al., 2019), which adds a generalized resource-adversarial discriminator (GRAD ) and leverages adversarial training. The results show that our method can significantly outperform these multi-task methods on the same datasets, which shows the effectiveness of our multi-cell structure in cross-domain settings. Comparison with the state-of-the-art models. Results show that our model outperforms cross-domain method of Jia et al. (2019), cross-type method of Wang et al. (2019) and methods using addition features (Crichton et al., 2017; Lu et al., 2018). Recently, LM pre-training based methods such as ELMO /B IOELMO (Peters et al., 2018), BERT (Devlin et al., 2019) and BIOBERT (Lee et al., 2020) achieve state-of-the-art results on NER. However, these methods use additional large-scale language resources, thus it is unfair to make direct comparisons with our method. Thus we leverage the outputs of LM pre-training meth-Methods F 1 # Params # Raw Jia et al. (2019) 73.59 12,916K 18,474K BERT-BASE (Devlin et al., 2019) 74.23 108M 3,700M BILSTM 70.73 211K MULTI-CELLLSTM 70.03 743K BILSTM +L M 71.30 211K 1,931K BILSTM +L M +D ICT 72.49 212K 1,931K MULTI-CELLLSTM +L M 72.81 743K 1,931K MULTI-CELLLSTM +L M (ALL ) 73.56 743K 8,664K MULTI-CELLLSTM +L M +D ICT 75.19  743K 1,931K Table 3: Results on CBS News datasets. ods as contextualized word embeddings. In particular, we use the same batch size as our method and the Adam optimizer with an initial learning rate 3e-5 in BERT fine-tuning baselines. Results show that our method benifits from these LM pretraining output features and outperforms these LM pre-training based methods. We conduct unsupervised domain adaptation on the CBS SciTech News test set, using CoNLL-2003 as the source-domain dataset. The overall results are listed in Table 3. Adding target-domain LM training. Only using the source-domain NER data, BILSTM and MULTICELLLSTM give comparable results, 70.73% F 1 and 70.03% F 1 , respectively. In comparison with the source-domain only models, all of the models 60 40 20 0 20 40 60 60 40 20 0 20 40 60 O(S) O(T) MISC(S) LOC(S) LOC(T) ORG(S) ORG(T) PER(S) PER(T) Figure 3: t-SNE visualization of ET cell states { c k } lk =1 on the CoNLL-2003 test set and Broad Twitter test set, differentiated by signal star and dot, respectively. using LM obtain significantly better results, which shows the effectiveness of using target-domain LM in zero-shot transfer. When using the same amount of target-domain raw data as Jia et al. (2019), The result of MULTI-CELLLSTM +L M (ALL ) is comparable to the state-of-the-art (Jia et al., 2019) (73.56% F 1 v.s. 73.59% F 1 ), which uses both source-domain LM and target-domain LM. This shows the effectiveness of multi-cell structure for zero-shot transfer. Adding a named entity dictionary. With the named entity dictionary collected by Peng et al. (2019), the results show a significant improvement (75.19% F 1 v.s. 72.81% F 1 ). To make fair comparison, we add the entity dictionary information to BILSTM +L M by doing an entity type prediction task together with the target-domain LM. BILSTM +L M +D ICT achieves better result than BILSTM +L M (72.49% F 1 v.s. 71.30% F 1 ), but it still cannot be comparable to our results. This shows that the auxiliary tasks can help learn entity knowledge from raw data, even if the named entity dictionary can not label all entities in a sentence. Visualization. In the proposed multi-cell LSTM, both ET cells and C cell play important roles in constructing a shared feature spaces across domains. We visualize feature spaces of ET cells and C cell in the Broad Twitter experiments. Figure 3 uses t-SNE (Maaten and Hinton, 2008) to visualize the ET cell states { c k } lk =1 . From the figure we can see that different ET cells can generate different feature distributions (gathering in different clusters of different colours), and states Entity group CHEMCCGGPCELLSPE All Is in Source? of the same ET cell gather together across domains. This indicates that our model can learn cross-domain entity typed knowledge with the help of ET cells, which are more robust across domains. Figure 4 visualize the hidden vectors of the target-domain only baseline, the multi-task baseline and the proposed model. From the figure, we can see that both the multi-task baseline and ours can obtain similar feature distributions across domains compared with the target-domain only baseline. In comparison with the multi-task baseline, our model also shows strong matches across domains in an entity type level, which can better narrow the gap between source and target domains as discussed in  3.4. Fine-grained comparison. We make fine-grained comparisons between our model and the multi-task baseline on the BioNLP dataset, aiming to show how our model achieves better results on the entity type level. Following Crichton et al. (2017) and Jia et al. (2019), we study five well studied entity groups (not including all entity types) in BioNLP13CG. As shown in Table 4, both MULTI (Multi-Task baseline) and Ours achieve significant F 1 improvement over the target-domain only baseline LSTM on the biochemistry entity groups that appear in both the target and the source datasets, such as CHEM , CC and GGP , which is consistent with intuition. But for biology entity groups not appearing in the source dataset, such as CELL and SPE , MULTI using traditional parameter sharing hardly improves the performances (+0.14% F 1 for CELL and +0.39% F 1 for SPE v.s. +1.82% F 1 for All). In contrast, Ours achieves relatively strong improvements (+2.10% F 1 for CELL and +2.84% F 1 for SPE ). This benefits from the distinct feature distributions across entity types by the multi-cell LSTM structure, which can effectively prevent the confusions drawn in a unified feature space. Ablation study. We conduct ablation studies on auxiliary tasks and model parameters. The results 40 20 0 20 40 40 20 0 20 40 60 O(S)PER(S)LOC(S)ORG(S)MISC(S) O(T)PER(T)ORG(T)LOC(T) Auxiliary tasks. When we only ablate L ent , the results on all of the three datasets suffer significant decline (-0.44% F 1 on BioNLP dataset, -0.85% F 1 on Broad Twitter dataset and -0.24% F 1 on CBS News dataset, respectively). When we only ablate L atten , the results on all of the three datasets suffer significant decline (over -1.5% F 1 on all of the three datasets). When we both ablate L ent and L atten , our model achieves similar results as the BILSTM-BASED baseline. This indicates that domain transfer of our model depends heavily on both auxiliary tasks. Number of parameters. We use two strategies to make the number of parameters of BILSTMBASED baseline comparable to that of our model: (i) STACKEDBILSTMS , stacking multi-layer BiL-STMs and enlarging the hidden size. (ii) HIDDENEXPANSION , with similar model structure, just enlarging the hidden size. Our model still significantly outperforms these baselines, which shows that the effects of our model do not arise from a larger number of parameters. Case study. Table 6 shows a case study, WHO is an organization and Nipah is a virus. Without using target-domain raw data, BI-LSTM baseline miclassifies Nipah as ORG . Both Ours and Sentence TheWorldHealthOrganization(WHO)describesNipahinfectionasa newlyemergingzoonosisthatcausesseverediseaseinbothanimalsandhumans. BILSTM + LM give the correct results because this entity is mentioned in raw data. Using the multi-cell structure, our method learns the pattern O RG , O, ORG , O from source data without confusions by target-domain specific entities, thus Ours recognizes WHO correctly. We have investigated a multi-cell compositional LSTM structure for cross-domain NER under the multi-task learning strategy. Theoretically, our method benefits from the distinct feature distributions for each entity type across domains. Results on a range of cross-domain datasets show that multi-cell compositional LSTM outperforms BiLSTM under the multi-task learning strategy. We thank the anonymous reviewers for their helpful comments and suggestions. We gratefully acknowledge funding from the National Natural Science Foundation of China (NSFC No.61976180) and the Westlake University and Bright Dream Joint Institute for Intelligent Robotics. Yue Zhang is the corresponding author.
abstain abstain abstain objective abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain method other abstain method abstain abstain abstain abstain method result result other abstain abstain method abstain method abstain other abstain method abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain other other other abstain abstain abstain abstain method abstain method abstain abstain abstain other abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain method method abstain abstain abstain method abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain method method other abstain abstain abstain method other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain result abstain abstain result abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain other other other