Multi-task learning (MTL) has been studied recently for sequence labeling. Typically, auxiliary tasks are selected specifically in order to improve the performance of a target task. Jointly learning multiple tasks in a way that benefit all of them simultaneously can in-crease the utility of MTL. In order to do so, we propose a new LSTM cell which contains both shared parameters that can learn from all tasks, and task-specific parameters that can learn task specific information. We name it a Shared-Cell Long-Short Term Memory (SC-LSTM). Experimental results on three sequence labeling benchmarks (named-entity recognition, text chunking, and part-of-speech tagging) demonstrate the effectiveness of our SC-LSTM cell. As one of the fundamental tasks in NLP, sequence labeling has been studied for years. Before the blooming of neural network methods, handcrafted features were widely used in traditional approaches like CRFs, HMMs, and maximum entropy classifiers (Lafferty et al., 2001; McCallum et al., 2000; McCallum and Li, 2003; Florian et al., 2003). However, applying them to different tasks or domains is hard. Recently, instead of using handcrafted features, end-to-end neural network based systems have been developed for sequence labeling tasks, such as LSTM-CNN (Chiu and Nichols, 2015), LSTM-CRF (Huang et al., 2015; Lample et al., 2016), and LSTM-CNN-CRF (Ma and Hovy, 2016). These models utilize LSTM to encode the global information of a sentence into a word-level representation of its tokens, which avoids manual feature engineering. Moreover, by incorporating a character-level representation of tokens, these models further improve. In many such studies, though, neural network models are trained toward a single task in a supervised way by making use of relatively small annotated training material. Jointly learning multiple tasks can reduce the risk of over-fitting to one task, and many attempts have been made at doing so for sequence labeling tasks (Caruana, 1997; Collobert and Weston, 2008; Collobert et al., 2011). Results so far are not conclusive. Some works have reported negative results overall. For instance in their pioneering work, Collobert et al. (2011) observed that training their model on NER, POS tagging and chunking altogether led to slight decrease in performance compared to a similar model trained on each task separately. Sgaard and Goldberg (2016) study chunking and CCG super tagging, coupled with an additional POS tagging task. They do report gains on both target tasks over single task models, but results varied depending where the additional task was taken care of in their architecture. The authors actually reported a failure to leverage other labelling tasks, and concluded that combined tasks should be sufficiently similar to the target one, for significant gains to be observed. Similarly, Alonso and Plank (2017) achieved significant improvements for only 1 out of 5 tasks considered. Also of interest is the work of (Changpinyo et al., 2018) where the authors investigate the classical shared encoder-based MTL framework (Collobert et al., 2011; Collobert and Weston, 2008) on 11 sequence labeling datasets including POS, NER, and chunking. They report that chunking is beneficial to NER, while POS tagging can be harmful. We present in Section 2 the two major approaches proposed for multi-task learning and discuss their limitations. We describe our approach in Section 3, and present our experimental settings and results in Section 4 and 5 respectively. We further analyze our approach in Section 6, discuss related works in Section 7 and conclude in Section 8. We are given a set of K tasks (in our case named-entity recognition, text chunking and part of speech tagging) that we want to train jointly in an end-to-end fashion. Each task k has an associated training set S k  tp x ki , y ki q i Pr 1 ,n k s u of n k examples, where x ki and y ki are sequences of size m i of tokens and tags respectively. We wish to learn a single function F which maps any token input sequence x i to its task-specific labels, where the mapping defines a probabilistic distribution for each involved task: p p y 1 i , . . . , y Ki q  F p x i q . There are two kinds of neural-based MTL methods. The first one  LSTM-s hereafter  uses an identical representation for all tasks, as proposed in (Collobert et al., 2011). This is illustrated in Figure 1 (left), where 3 layers of LSTMs are being stacked. 1 While different tasks directly interact with all parameters of the model, this increases the risk of optimization conflicts when gold-standard labels from different tasks have no significant correlation. The second class of multi-task architectures is depicted in the middle part of Figure 1, and is named LSTM-d hereafter. In this configuration, each LSTM layer feeds a task-specific classifier and serves as input to the next stacked LSTM layer (Sgaard and Goldberg, 2016). The underlying as-sumption is that tasks may be ordered in such a way that easier tasks are learned first, the target tasks being the latest one considered, thus benefit-ing the hidden state of the lower layers. One drawback, however, is that one must decide which task to consider first, a decision which may impact the overall performance. Furthermore, using the hidden state of lower layers increases the limitation of learning representation for that task. We believe that one reason for the lack of consistent benefits of MTL in the labelling literature is that the proposed models share all or part of parameters for extracting hidden states, which leads to optimization conflicts when different tasks 1 This typically delivers better performance than having just one. In practice also, LSTM layers are replaced by biLSTM ones. require different features. We believe it would be helpful if we make the model have ability to learn a task-specific representation (Ammar et al., 2016; stling and Tiedemann, 2016; Kiperwasser and Ballesteros, 2018) at the same time. This observation led us to design a new LSTM cell which allows at almost no additional computation cost to efficiently train a single RNN-based model, where task-specific labelers clearly outperform their singly-tasked counterparts. Actually, by training our model on NER, chunking and POS tagging, we report state-of-the-art (or highly competitive) results on each task, without using external knowledge (such as gazetteers that has been shown to be important for NER), or hand-picking tasks to combine. Our solution is depicted in the right part of Figure 1 and detailed in the next section. It is actually very similar to the LSTM-s one, except that each LSTM layer passes on task-specific hidden representations that learn the peculiarities of individual tasks. In the last layer, each classifier is fed with a concatenation of a global representation (as in LSTM-s) and the task-specific one. By doing so, we keep the advantages of both aforementioned approaches, where one task can have its task-specific representation as in LSTM-d, while not enforcing any task order, further giving the freedom to the model to learn specificities of each task. For this architecture to work, we need to modify the classical LSTM cell, which is described in the next section. An LSTM cell (Hochreiter and Schmidhuber, 1997) is made up of four functional gates which control the input and output of the memory state c t : a forget gate f t controls what information to remove from the memory state of the last time step, an input gate i t controls the information to add to the current memory, and an output gate o t controls what information to release from the current memory state. This mechanism is formalized in Equation 1 where x t , h t is the input vector and hidden vector at time step t ,  is the sigmoid function,  c is the new candidate state, c t is the memory state, which encodes information of the current input and history information, and  indicates the element-wise product. W f , U f , W i , U i , W o , U o , W c , U c are weight matrices that are being learned. f t   p W f h t  1 ` U f x t q , (1) i t   p W i h t  1 ` U i x t q , o t   p W o h t  1 ` U o x t q ,  c  tanh p W c h t  1 ` U c x t q , c t  i t   c ` f t  c t  1 , h t  o t  tanh p c t q , Figure 2: Structure of an SC-LSTM cell. The overall structure of our cell is depicted in Figure 2. On top of a standard LSTM cell, we add one cell per task with its own parameters. The standard LSTM cell is thus shared among the K task-specific cells, therefore the name we choose for this new cell, which stands for Shared-Cell LSTM. Task-specific cells are each parametrized by an output gate o kt which learns to select the useful information from the shared memory cell c t and outputs q kt . This is formally described in Equation 2, where W k and U k are two extra weight matrices that parametrize the k th task, and q kt has to be understood as a task-specific hidden representation since parameters of k th task-specific cell are only updated by supervision from task k . o kt   p W k q kt  1 ` U k x kt q (2) q kt  o kt  tanh p c t q In order to make use of both shared and task-specific information (Kim et al., 2016; Peng et al., 2017; Hershcovich et al., 2018), for the k th task, we concatenate the output of the shared cell h t and of the task-specific one q kt to generate the final latent representation, as noted in Equation 3, where  is the concatenation operation. In practice, we stack SC-LSTM layers. The top-most layer uses s kt as a representation of the current input, while cells in lower layers pass the current shared hidden state h t to the upper SC-LSTM cell. The use of s in the topmost layer only is arbitrary and should be investigated. The training material available may be gathered from different datasets S k which means that input sequences differ from one task to another. Therefore in practice, we build K dataloaders, and the training is achieved in a stochastic manner by looping over the tasks at each epoch, as detailed in Algorithm 1. The loss function (cid:15) we minimize is a linear combination of task-specific loss functions, where the weighting coefficients (  k in Equation 4) are hyper-parameters. We seek to minimize cross-entropy of the predicted and true distributions, therefore task-specific loss functions are defined according to Equation 5. where  y ki,j is the prediction of the k th softmax classifier parametrized by a projection matrix W k and a bias vector b k : where s ki , j stands for j th token of the i th sequence for task k . In an SC-LSTM cell with k tasks, we will add k matrices and k bias vectors, compared with a vanilla LSTM cell, which increases the capacity of the resulting model. This extra calculation is conducted in parallel with the original LSTM computations. We test several baseline systems and our SC-LSTM model on three well-established sequence labeling benchmarks: CoNLL2003 (Tjong Kim Sang and De Meulder, 2003) for named-entity recognition, CoNLL2000 (Tjong Kim Sang and Buchholz, 2000) for chunking, and the more recent Universal Dependency dataset (Nivre et al., 2016) for part-of-speech tagging, 2 and on which recent MTL investigations have been conducted (Alonso and Plank, 2017; Changpinyo et al., 2018). recent works (Peters et al., 2017; Liu et al., 2017), sections 15-18 of the Wall Street Journal are used for training, and we randomly sampled 1000 sentences in the training set as the development set. Section 20 is used for tests. Table 1 presents the main characteristics of the training, development and test sets we used. We used bidirectional LSTM or SC-LSTM as our encoders for the vector representation of words. Bidirectional LSTM can capture global information of the whole sentence, thanks to the encoding of a sequence in a recurrent way. The vector representation of words consists of three parts: word embedding, character-level representation and contextual representation:  previous works have proven that pre-trained word embeddings like Word2vec (Mikolov et al., 2013), SENNA (Collobert et al., 2011), or Glove (Pennington et al., 2014) have a positive impact on sequence labeling tasks. We used Glove embeddings 3 of dimension 100, that are fine-tuned during training;  character-level information has been proven useful for three sequence labeling tasks (San-tos and Zadrozny, 2014; Ma and Hovy, 2016; Lample et al., 2016) and Some works further show its effectiveness (Reimers and Gurevych, 2017; Yang et al., 2018). In order to encode character sequences, we used a CNN. The character embedding look-up table is initialized by randomly sampling from 3 The embedding file glove6B are available https:// nlp.stanford.edu/projects/glove/ the uniform distribution in the range [-0.1, 0.1];  the third part of the input vector is contextual embedding. Most of the recent works found that contextualized features such as ELMo or BERT (Peters et al., 2018; Devlin et al., 2018) can greatly boost performance. We incorporate ELMo into our input vector by using the ELMo implementation of Gardner et al. (2018). Conditional random field (CRF) classifiers can consider the dependency of output tags and has been proven useful for tasks like NER or chunking. We consider CRF layers in our models. To compare the effectiveness of three MTL models, we first test our SC-LSTM and other vanilla LSTM based models without CNN based character-level information extractor and contextual embeddings. In this case, the input will be a concatenation of word embedding and capitalization features 4 . To compare with the state-of-art models, we further implemented three more variants: SC-LSTM-CNN-CRF which makes use of CNN-based character level features with a CRF layer on top, very similar to (Ma and Hovy, 2016), and SC-LSTM-LM-CNN, a variant which considers contextualized word embeddings as in (Peters et al., 2018). For SC-LSTM-LM-CNN, we used mainly the configuration advocated in (Peters et al., 2018) except for the hidden size of SC-LSTM, which we report in Table 2. We further weighted the NER task in the objective function of Equation 4 to 3 (  NER ), the weights of the other tasks where set to 1. 5 We trained this model SC-LSTM-LM-CNN using the Adam optimizer (Kingma and Ba, 2014) with default setting. Such a model spends typically less than 30 epochs to converge. We choose the mini-batch size of 10 and used the gradient clipping threshold 5. All models are implemented using the Pytorch library (Paszke et al., 2017). Several tagging schemas have been proposed for conducting chunking and NER tasks. We used the most common BIO one in this work. 4 Eight features encode capitalization patterns, such as AllUpper, InitialUpper, etc. 5 We found the loss of the NER task to be around 1/3 of the loss of the chunking and POS tasks. In this section, we start by comparing MTL approaches based on LSTM and SC-LSTM cells. We then report the performance of variants of our approach we implemented and compare it to state-of-the-art models. We compare bidirectional LSTM (STL), SC-LSTM, and baseline MTL models. The results are shown in Table 3. Training Bi-LSTM models on each task separately (STL) was first conducted as a point of comparison (line 1). For LSTM-s and LSTM-d, we regard one task as the main task and the others as auxiliary tasks; a setting consistent with Sgaard and Goldberg (2016). Because LSTM-s and LSTM-d always fail to achieve stable and competitive results on the three tasks we considered at the same time, we report the best performance we could obtain for each task (line 2 and 3) specifically. On the contrary, our SC-LSTM model is trained once jointly on all tasks, and only one model is being tested in the end, which is much easier and more realistic of a real deployment (line 4). The results show that our SC-LSTM model improves the performance of the three tasks simultaneously compared with LSTM (STL), and outperforms the other two MTL methods. By joint learning three tasks, both LSTM-s and LSTM-d can boost the chunking task significantly, but both fail to improve NER and POS tasks. This is consistent with observations made in (Collobert et al., 2011; Sgaard and Goldberg, 2016). We also observe that our SC-LSTM model also benefits the chunking task the most. We will analyze this further in Section 6. To further demonstrate the effectiveness of our SC-LSTM model, we compared different variants with state-of-the-art approaches, that we classify into three broad categories: Single sequence labeling where models are trained without the supervision of other tasks. Specifically, we compare our results to the LSTM-CRF model of Lample et al. (2016) and the LSTM-CNN-CRF of Ma and Hovy (2016), since those are state-of-the-art singly tasked sequence labelers. Multi-tasked sequence labelers where models leverage the supervision of other tasks. We compare our model with the representative approaches of Luo et al. (2015); Sgaard and Goldberg (2016); Collobert and Weston (2008); Collobert et al. (2011). Models with language model. Recently, several studies in using contextualized word embeddings achieved great success in a number of tasks. Some recent studies (Peters et al., 2017; Rei, 2017; Peters et al., 2018; Devlin et al., 2018) are particularly considered. Results for the CoNLL 2003 dataset are reported in Table 4. We observe that our SC-LSTM-LM-CNN model outperforms all approaches but Devlin et al. (2018) and Akbik et al. (2018). The latter work is using the development set as training NER F1-score Collobert et al. (2011) 89.59 Chiu and Nichols (2015)  91.62 Huang et al. (2015) 88.83 Luo et al. (2015) 91.20 Ma and Hovy (2016) 91.21 Lample et al. (2016) 90.94 Shen et al. (2017) 90.89 Yang et al. (2017) 91.20 Rei (2017) 86.26 Liu et al. (2017) 91.71 Peters et al. (2017) 91.93 Zhang et al. (2018) 91.2 Liu et al. (2018) 91.95 Peters et al. (2018) 92.22 Clark et al. (2018) 92.60 Akbik et al. (2018)  93.09 Devlin et al. (2018) 92.80 SC-LSTM 89.96 SC-LSTM-CNN-CRF 91.37 SC-LSTM-LM-CNN 92.60 Table 4: F1-score on the CoNLL03 NER dataset. material, which avoids a direct comparison. The former model (BERT) is achieving great success by leveraging a huge amount of unannotated data as well as a lot of computation resources we could not afford in this study. We are however pleased that our model is leveraging contextual embeddings with 0.38 absolute F1 improvement over the results of Peters et al. (2018). We leave as future work to investigate whether our MTL model can leverage BERT embeddings. We compared a number of models on the CoNLL2000 chunking dataset. A few of them (Ma and Hovy, 2016; Lample et al., 2016; Peters et al., 2018) where not tested on this benchmark, and we reimplemented them. We also trained the companion toolkits of those models, but (as detailed in the next section) got slightly lower results for some reasons. Table 5 reports the performance of the many approaches we tested. We observe that our SC-LSTM-LM-CNN architecture achieves a new state-of-the-art F1 score, with over 1 absolute point over the competitive approach of Peters et al. (2017), and an improvement of 0.4% over the current state-of-the-art method of Clark et al. (2018). We conducted experiments on the Universal Dependency POS English dataset and present the results in Table 6. The only study we found that reports results on the UD v1.3 benchmark we used here is (Bjerva et al., 2016), and we report the results they published. For Liu et al. (2017), Peters et al. (2018) we used the available companion toolkits 6 that we trained ourself with the default settings. We re-implemented the other approaches. Again, we observe that SC-LSTM-LM-CNN outperforms all other approaches we tested. The absolute improvement in F1 score over the current state-of-the-art of Peters et al. (2018) is 0.21%. In order to further validate our implementations, we also ran the toolkits of Ma and Hovy (2016) and Lample et al. (2016) and obtained slightly lower results 7 . We conducted a number of investigations in order to understand better why our multi-task learning model is effective. We report in Figure 3 the convergence of different MTL models on the development set. To obtain those curves, we collected the F1-score on the NER and chunking tasks as well as the accuracy of the POS task, and averaged them after each epoch. We clearly see that the SC-LSTM model converges faster than other ones. It achieves higher performance after the first epoch, and after about ten epochs, it shows a smooth performance curve, while LSTM-s and LSTM-d models still fluctuate. This indicates that our model can learn the hidden representation of multiple tasks in a faster and smoother way than the other two methods. Besides, we observe in Figure 3c and 3d that combinations of tasks involving chunking typically show a smooth training curve, on the contrary to Figure 3b where NER and POS tasks are combined. The fact that the training regimen fluctuates in the latter case for both LSTM-s and LSTM-d suggests that conflicts with those two tasks happen during optimisation, which we do not observe for our model. Also, Figure 3a illustrates that combining the three tasks altogether leads to comparably better performance of our model over LSTM-s and LSTM-d. We analyzed which task is benefited or harmed by others under the three MTL settings we considered and present results in Figure 4. We find that it leads to better performance for all MTL models by jointly learning chunking with NER or POS(see in Figure 4b). This is in particular the case of our SC-LSTM model which records the largest gain, especially when all tasks are being trained on. Figure 4c shows results obtained on POS. Only our SC-LSTM model achieves a meaningful improvement. We however observe that the NER task tends to hurt the performance of POS, since in most cases, the performance of POS+NER is lower than the one obtained with POS+chunking. Clearly, the combination of different tasks has a different effect on the final performance of each task. The chunking task seems compatible with NER and POS tasks, and it boosts the other two tasks in all three MTL settings, which is consistent with the results of Changpinyo et al. (2018). Directly jointly training on POS and NER datasets tends to reduce the performance in LSTM-s and LSTM-d, which is also consistent with the conclusion in (Changpinyo et al., 2018). In conclusion, all of the results show that our SC-LSTM model is effective at capturing the mutual benefits of all combined tasks. Since it performs consistently better in various settings, we believe our model to be more robust. There are many works that use extra knowledge to improve the performance of sequence labeling tasks. Many works have focussed on jointly learning two tasks, often with one being considered as the main task, the other being the auxiliary one (S-gaard and Goldberg, 2016; Bjerva et al., 2016; Alonso and Plank, 2017). For instance, chunking, combinatory categorical grammar supertag-ging, NER, super senses (SemCor), or multiword expression + supersense will be taken as the main task, while POS is the auxiliary task in (Sgaard and Goldberg, 2016). Exceptions to this line of work include (Collobert et al., 2011) that evaluates four tasks: POS, chunking, NER and semantic role labeling; (Kiperwasser and Ballesteros, 2018) that considers a machine translation task with POS and dependency parsing. And Niehues and Cho (2017) considers machine translation with POS and NER tasks; Zhang and Weiss (2016) show that jointly learning a POS tagger and a dependency parser is effective. Miwa and Bansal (2016) jointly trained models for entity detection and relation extraction in the field of relation extraction. Other works are also trying to leverage language models to empower the performance of sequence labeling tasks. Notably, Liu et al. (2017) propose a model which uses a neural language model to learn character-level knowledge, and conducts sequence labeling to guide the language model towards specific tasks. Others (Peters et al., 2017, 2018; Devlin et al., 2018) use neural language models pre-trained on a large unlabeled corpus to learn context-sensitive representations of words, and leverage this representation into the sequence labeling model. More related to the present work are studies that analyze the effectiveness of different combinations of sequence labeling tasks in a multi-task learning. In particular, Changpinyo et al. (2018) conduct an investigation on 11 sequence labeling tasks, while Alonso and Plank (2017) evaluate 5 tasks but report signifiant gains for only one task. In this paper, we propose a simple yet powerful LSTM cell that leverage both shared and task-Figure task-Figure 4: Results of different task groups on each test set: (a) NER, (b) chunking, and (c) POS. Horizontal lines show results for single-task models(NER: 89.39, Chunking: 94.44 and POS: 95.46). Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic tagging with deep residual networks. arXiv preprint arXiv:1609.07053 . Rich Caruana. 1997. Multitask learning. Machine learning , 28(1):4175. Soravit Changpinyo, Hexiang Hu, and Fei Sha. 2018. Multi-task learning for sequence tagging: An empirical study. arXiv preprint arXiv:1808.04151 . Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308 . Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 1914 1925. Association for Computational Linguistics. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning , pages 160167. ACM. Ronan Collobert, Jason Weston, Lon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research , 12(Aug):24932537. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . specific parameters in a multi-task setting designed for sequence labelling. We conduct extensive experiments to compare both single-task learning and multi-task learning models. We analyzed the influence of grouping different tasks under various multi-task settings. Experiments demonstrate the effectiveness of our model for sequence labeling tasks. We report new state-of-the-art results on both POS and chunking tasks, and close to state-of-the-art performance on NER, without exploiting external ressources, neither diving into dedicated feature engineering. Despite those positive outcomes, several issues with multi-task learning for sequence labeling remain open. In particular, we only considered 3 tasks here, and therefore plan to test our approach on more tasks, perhaps understanding better why some tasks are less useful to others. Also, there are several ways we could have used our SC-LSTM cell into our models which we would like to investigate further. In particular, in this work, we only used the specific-task hidden states in the last layer of the model, which can obviously be revisited.
abstain abstain abstain objective abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method method abstain abstain method method other abstain abstain other method method method abstain abstain method method abstain abstain abstain method method method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result other abstain result result abstain abstain abstain result abstain abstain result result abstain abstain result abstain result method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method other other other other other other other other other other other other objective abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method method objective objective abstain method objective method