Fine-tuning neural networks is widely used to transfer valuable knowledge from high-resource to low-resource domains. In a standard fine-tuning scheme, source and target problems are trained using the same architecture. Although capable of adapting to new domains, pre-trained units struggle with learning uncommon target-specific patterns. In this paper, we propose to augment the target-network with normalised, weighted and randomly initialised units that beget a better adaptation while maintaining the valuable source knowledge. Our experiments on POS tagging of social media texts (Tweets domain) demonstrate that our method achieves state-of-the-art performances on 3 commonly used datasets. POS tagging is a sequence labelling problem, that consists on assigning to each sentence' word, its disambiguated POS tag ( e.g. , Pronoun, Noun) in the phrasal context in which the word is used. Such information is useful for higher-level applications, such as machine-translation (Niehues and Cho, 2017) or cross-lingual information retrieval (Semmar et al., 2006, 2008). One of the best approaches for POS tagging of social media text (Meftah et al., 2018a), is transfer-learning, which relies on a neural-network learned on a source-dataset with suffi-cient annotated data, then further adapted to the problem of interest ( target-dataset ). While this approach is known to be very effective (Zen-naki et al., 2019), because it takes benefit from pre-trained neurons, it has one main drawback by design. Indeed, it has been shown in computer-vision (Zhou et al., 2018a) that, when fine-tuning on scenes a model pre-trained on objects, it is the neuron firing on the white dog object that became highly sensitive to the white waterfall scene. Simply said, pre-trained neurons are biased by what they have learned in the source-dataset. This is Figure 1: Given a word representation x i , a BiLSTM (  ) models the sequence, and a FC layer (  ) performs classification. also the case on NLP (see experiments). Consequently, pre-trained units struggle with learning patterns specific to the target-dataset ( e.g. , wanna or gonna in the Tweets domain). This last is non-desirable, since it has been shown recently (Zhou et al., 2018b) that such specific units are important for performance. To overcome this drawback, one can propose to take benefit from randomly initialised units, that are by design nonbiased. However, it is common to face small target-datasets that contain too few data to learn such neurons from scratch. Hence, in such setting, it is hard to learn random units that fire on specific patterns and generalise well. In this article, we propose a hybrid method that takes benefit from both worlds, without their drawbacks. It consists in augmenting the source-network (set of pre-trained units) with randomly initialised units and jointly learn them. We call our method PretRand ( Pret rained and Rand om units) and illustrate it in Fig. 1. The main difficulty is forcing the network to consider random units, because they have different behaviours than pre-trained ones. Indeed, while these last strongly fire discriminatively on many words, these first do not fire on any word at the initial stage of fine-tuning. Therefore, random units do not significantly contribute to the computation of gradients and are thus slowly updated. To overcome this problem, we proposed to independently normalise pre-trained and random layers. This last balances their range of activations and thus forces the network to consider them, both. Last but not least, we do not know which of pre-trained and random units are the best for every class-predictor, thus we propose to learn weighting vectors on top of each branch. Evaluation was carried on 3 POS tagging Tweets datasets in a transfer-learning setting. Our method outperforms SOTA methods and significantly surpasses fairly comparable baselines. 2.1 Base Model Given an input sentence S = [ w 1 , . . . , w n ] of n successive tokens w i , the goal of a POS tagger is to predict the POS-tag c i  C of every w i , with C  RC being the tag-set. Hence, for our base model, we used a common sequence labelling model which first, computes for each token w i , a word-level embedding (denoted  w ) and character-level embedding using biLSTM encoder (  c ), and concatenates them to get a final representation x i . Second, it feeds the later representation into a biLSTM features extractor (denoted  ) that outputs a hidden representation, that is itself fed into a fully-connected (FC) layer (denoted  ) for classification. Formally, given w i , the logits are obtained using:  y w i =      ( w i ) , with  being the concatenation of the output of  c and  w for w i . In a standard fine-tuning scheme (Meftah et al., 2018b),  and  are pre-trained on the source-task and  is randomly initialised. Then, the three modules are further jointly trained on the target-task by minimising a Softmax Cross-Entropy (SCE) loss using the SGD algorithm. As mentioned in the introduction, pre-trained neurons are biased by design, thus limited. This motivated our proposal to augment the pre-trained branch with additional random units (as illustrated in Fig. 1). To do so, theoretically one can add the new units in any layer of the base model. However in practice, we have to make a trade-off between performances and the number of parameters (model complexity). Thus, given that deep layers are more task-specific than shallow ones (Peters et al., 2018; Mou et al., 2016), and that word embeddings (shallow layers) contain the majority of parameters, we choose to expand only the top layers. With this choice, we desirably increase the complexity of the model only by 1 . 02  compared to the base one. In terms of the layers expanded, we specifically add k units to  resulting in an extra biLSTM layer:  r ( r for rand); and C units in  resulting in an extra FC layer:  r . Hence, for every w i , the additional random branch predicts class-probabilities following:  y rw i =  r   r ( x i ) (with x i =  ( w i ) ). Note that, having two FC layers obviously outputs two predictions per class (one from the pre-trained FC  y pw i and one from the random  y rw i ), that thus need to be merged. Hence, to get the final predictions, we simply apply an element-wise sum between the output of both branches:  y w i =  y pw i   y rw i . As in the classical scheme, SCE is minimised but here, both branches are trained jointly. Nevertheless, while at the initial stage of fine-tuning, the pre-trained units are strongly firing on many words, the random ones are firing very weakly. As stated in some computer-vision works (Liu et al., 2015; Tamaazousti et al., 2018), the later setting causes an absorption of the weights, outputs and thus gradients of the random units by the pre-trained ones, which thus makes them useless at the end. We encountered the same problem with textual data on the POS-tagging problem. Indeed, as illustrated in the left plot of Fig.2, at the end of training, the distribution of the random units' weights is still absorbed (closer to zero) by that of the pre-trained ones. To prompt the two classifiers to work cooperatively, we normalise (using an (cid:96) p -norm) both of them independently before merging them. Formally, we apply N p ( x ) = x || x || p on  y pw i and  y rw i . The normalisation is desirably solving the weights absorption problem since at the end of the training, the distributions of the pre-trained and random weights become very similar (right of Fig. 2). Furthermore, we have observed that despite the normalisation, the performances of the pre-trained classifiers were still much better than the randomly initialised ones. Thus, to make them more competitive, we propose to start with optimising only the randomly initialised units while freezing the pre-trained ones, then, launch the joint training. This is called random++ in the following. 2.4 Learnable Weighting Vectors Back to the extra predictor (FC layer of random branch), it is important to note that both branches are equally important for making a decision for every class, i.e., no weight is applied on the dimensions of  y pw i and  y rw i . However, this latter is sub-optimal since we, a priori, do not know which kind of units (random or pre-trained) is better for making a decision. Consequently, we propose to weight the contribution of the predictions for each class. For this end, instead of simply performing an element-wise sum between the random and pre-trained predictions, we first weight each of them with learnable weighting vectors, then compute a Hadamard product with their associated normalised predictions; the learnable vectors u  RC and v  RC , respectively corresponding to the pre-trained and random branch, are initialised with 1-values and are learned by SGD. Formally, the final predictions are computed following:  y w i = u (cid:12) N p ( y pw i )  v (cid:12) N p ( y rw i ) . In the word-level embeddings, tokens are lower-cased while the character-level component still retains access to the capitalisation information. We set the character embedding dimension at 50, the dimension of hidden states of the character-level biLSTM at 100 and used 300-dimensional word-level embeddings. The latter were pre-loaded from publicly available Glove pre-trained vectors on 42 billions words from a web crawling and contain-Corpus TPoS Ark TweeBank Train 10,652 26,594 24,753 Dev 2,242 n/a 11,742 Test 2,291 7,707 19,112 Table 1: Number of tokens in every used dataset. ing 1.9M words (Pennington et al., 2014). Note that, these embeddings are also updated during fine-tuning. For biLSTM (token-level feature ex-tractor), we set the number of units of the pre-trained branch to 200 and experimented our approach with k added random-units, with k  { 50 , 100 , 150 , 200 } . For the normalisation, we used (cid:96) 2 -norm. Finally, in all experiments, training was performed using SGD with momentum and mini-batches of 8 sentences. Evidently, all the hy-perparameters have been cross-validated. For the source-dataset, we used the Wall Street Journal ( WSJ ) part of Penn-Tree-Bank (PTB), a large English dataset containing 1.2M+ tokens from the newswire domain annotated with the PTB tag-set. Regarding the target-datasets, we used three Tweets datasets: TPoS (Ritter et al., 2011), annotated with 40 tags ; ARK (Owoputi et al., 2013) containing 25 coarse tags; and the re-cent TweeBank (Liu et al., 2018) containing 17 tags (PTB universal tag-set). The number of tokens in the datasets are given in Table 1. To assess the POS tagging performances of our PretRand model, we compared it to 5 baselines: Random-200 and Random-400 : randomly initialised neural model with 200 and 400 biLSTM's units; Fine-tuning : pre-trained neural model, fine-tuned with the standard scheme; Ensemble (2 rand) : averaging the predictions of two base models randomly initialised and learned independently (with different random initialisation) on Tweets datasets; and Ensemble (1 pret + 1 rand) : same as the previous but with one pre-trained on WSJ and the other randomly initialised. We also compared it to the 3 best SOTA methods: Derczynski et al. (2013) (GATE) is a model based on HMMs with a set of normalisation rules, external dictionaries and lexical features. They experiment it on TPoS, with WSJ and 32K tokens from the NPS IRC corpus. They also used 1.5M additional training tokens annotated by vote-constrained bootstrapping (GATE-bootstrap). Owoputi et al. (2013) proposed a model based on first-order Maximum Entropy Method #params TPoS ArK TweeBank Dev Test Test Dev Test Avg GATE (Derczynski et al., 2013) n/a 89.37 88.69 n/a n/a n/a n/a GATE-bootstrap (Derczynski et al., 2013) n/a n/a 90.54 n/a n/a n/a n/a ARK (Owoputi et al., 2013) n/a n/a 90.40 93.2 n/a 94.6 n/a TPANN (Gui et al., 2017) n/a 91.08 90.92 92.8 n/a n/a n/a Random-200 1  88.32 87.76 90.67 91.20 91.56 89.90 Random-400 1 . Markov Model (MEMM) with greedy decoding and using brown clustering and careful hand-engineered features. Recently, Gui et al. (2017) proposed TPANN that uses adversarial training to leverage huge amounts of unlabelled Tweets. From the results given in Table 2, one can first see that our approach outperforms the SOTA and baseline methods on all the datasets. More interestingly, PretRand significantly outperforms the popular fine-tuning baseline by +1.4% absolute point on average and is better on all classes (see per-class improvement on Fig. 4). PretRand also outperforms the challenging Ensemble Model by a large margin (+2.2%), while using much less parameters. This clearly highlights the difference of Figure 4: Sorted class-accuracy improvement (%) on TweeBank of PretRand compared to fine-tuning. our method with ensemble methods and the importance of having a shared word representation as well as our normalisation and weighting learnable vectors during training. A key asset of PretRand, is that it uses only 0.02% more parameters compared to the fine-tuning baseline. An interesting experiment is to evaluate the gain of performance of PretRand compared to fine-tuning, according different target-datasets' sizes. From the results in Fig. 3, PretRand has desirably a bigger gain with bigger target-task datasets, which clearly means that the more target training-data, the more interesting our method will be. To assess the contribution of different components of PretRand, we performed an ablation study. Specifically, we successively ablated the main components of PretRand, namely, the learnable vectors (learnVect), the longer training for random units (random++) and the normalisation ( (cid:96) 2 -norm). From the results in Table 3, we can observe that the performances are only marginally better than standard fine-tuning when ablating the three components from PretRand. More importantly, adding each of them successively, makes the performances significantly better, which highlights the importance of every component. Here our goal is to highlight that as in (Zhou et al., 2018a), pre-trained units can be biased the standard fine-tuning scheme. To do so, we follow (Tamaazousti et al., 2017) and analyse the units of  (biLSTM layer) before and after fine-tuning. Specifically, we compute the Pearson's correlation between all the units of the layer before and after fine-tuning. Here, a unit is represented by the random variable being the concatenation of its output activations from all the validation samples of the TweeBank dataset. From the resulting correlation matrix illustrated in Fig. 5, one can clearly observe the white diagonal, highlighting the fact that, every unit after fine-tuning is more correlated with itself before fine-tuning than with any other unit. This clearly confirms our initial motivation that pre-trained units are highly biased to what they have learned in the source-dataset, making them limited to learn patterns specific to the target-dataset. Additionally, we visualise in Fig. 6 a concrete example of a biased neuron when transferring from newswire to Tweets domain. Specifically, we show the top-10 words activating unit-169 of  (from the standard fine-tuning baseline), before fine-tuning (at this stage, the model is trained on the source-dataset WSJ) and during fine-tuning on the TweeBank dataset. We can observe that this unit is highly sensitive to proper nouns ( e.g. , George and Washington ) before fine-tuning, and to words with capitalised first-letter whether the word is a proper noun or not ( e.g. , Man and Father ) during fine-tuning on TweeBank dataset. Indeed, we found that most of tokens with upper-cased first letter are mistakenly predicted as proper nouns ( PROPN ) in the standard fine-tuning scheme. In fact, in standard English, inside sentences, only proper nouns start with upper-cased letter thus fine-tuning the pre-trained model fails to slough this pattern which is not always respected in Tweets. Unique units emerge in random branch Finally, we highlight the ability of randomly initialised units to learn patterns specific to the target-dataset and not learned by the pre-trained ones because of their bias problem. To do so, we visualise unique units  i.e. , random units having a correlation lower than 0.4 with pre-trained ones  emerging in the random branch. While only one shown in Fig. 7, many unique units have been learned by the random branch of our PretRand model: 37.5% of the 200 random units have correlation lower than 0.4 with the pre-trained ones. Regarding unit-99, it is highly discriminative to tokens na , ta and n't . Indeed, in TweeBank, words like gonna (going to) are tokenized into two tokens: gon and na , with the later annotated as a particle and the former as a verb . Importantly, not even one unit from the standard fine-tuning scheme has been found firing on the same important and target-dataset specific pattern. In this paper, we introduced a method to improve fine-tuning using 3 main ideas: adding random units and jointly learn them with pre-trained ones; normalising the activations of both to balance their different behaviours; applying learnable weights on both predictors to let the network learn which of random or pre-trained one is better for every class. We have demonstrated its effectiveness on domain adaptation from newswire domain to three commonly used Tweets-datasets for POS tagging.
abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain objective abstain objective abstain result abstain method abstain abstain abstain abstain abstain method abstain method method method abstain method abstain abstain method abstain abstain abstain method abstain method method abstain method method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain result objective