While deep learning is a powerful tool for natural language processing (NLP) problems, successful solutions to these problems rely heavily on large amounts of annotated samples. However, manually annotating data is expensive and time-consuming. Active Learning (AL) strategies reduce the need for huge volumes of labeled data by iteratively selecting a small number of examples for manual annotation based on their estimated utility in training the given model. In this paper, we argue that since AL strategies choose examples independently, they may potentially select similar examples, all of which may not contribute significantly to the learning process. Our proposed approach, Active 2 Learning (A 2 L), actively adapts to the deep learning model being trained to eliminate such redundant examples chosen by an AL strategy. We show that A 2 L is widely applicable by using it in conjunction with several different AL strategies and NLP tasks. We empirically demonstrate that the proposed approach is further able to reduce the data requirements of state-of-the-art AL strategies by  3  25 % on an absolute scale on multiple NLP tasks while achieving the same performance with virtually no additional computation overhead. Active Learning (AL) (Freund et al., 1997; McCallum and Nigam, 1998) reduces the need for large quantities of labeled data by intelligently selecting unlabeled examples for expert annotation in an iterative process. Many Natural Language Processing (NLP) tasks like sequence tagging (NER, POS) and Neural Machine Translation (NMT) are very data-intensive and require a meticulous, time-consuming, and costly annotation process. On the other hand, unlabeled data is practically unlimited. Due to this, many researchers have explored applications of active learning for NLP (Thompson et al., 1999; Figueroa et al., 2012). A general AL method proceeds as follows: (i) The partially trained model for a given task is used to (possibly incorrectly) annotate the unlabeled examples. (ii) An active learning strategy selects a subset of the newly labeled examples via a criterion that quantifies the perceived utility of examples in training the model. (iii) The experts verify/improve the annotations for the selected examples. (iv) These examples are added to the training set, and the process repeats. AL strategies differ in the criterion used in step (ii). We claim that all AL strategies select redundant examples in step (ii) . If one example satisfies the selection criterion, then many other similar examples will also satisfy it (see the next paragraph for de-tails). As the examples are selected independently, AL strategies redundantly choose all of these examples even though, in practice, it is enough to label only a few of them (ideally just one) for training the model. This leads to higher annotation costs, wastage of resources, and reduces the effectiveness of AL strategies. This paper addresses this problem by proposing a new approach called A 2 L (read as active-squared learning) that further reduces the redundancies of existing AL strategies. Any approach for eliminating redundant examples must have the following qualities: (i) The redundancy should be evaluated in the context of the trained model. (ii) The approach should apply to a wide variety of commonly used models in NLP. (iii) It should be compatible with several existing AL strategies. The first point merits more explanation. As a model is trained, depending on the downstream task, it learns to focus on certain properties of the input. Examples that share these properties (for instance, the sentence structure) are similar from the model's perspective. If the model is confused about one such example, it will likely be confused about all of them. We refer to a similarity measure that is computed in the context of a model as a model-aware similarity (Section 3.1). twin-(Bromley et al., 1994; Mueller and Thyagarajan, 2016) based method for computing model-aware similarity to eliminate redundant examples chosen by an AL strategy. This Siamese network actively adapts itself to the underlying model as the training progresses. We then use clustering based on similarity scores to eliminate redundant examples. (ii) We develop a second, computationally more effi-cient approach that approximates the first one with a minimal drop in performance by avoiding the clustering step. Both of these approaches have the desirable properties mentioned above. (iii) We experiment with several AL strategies and NLP tasks to empirically demonstrate that our approaches are widely applicable and significantly reduce the data requirements of existing AL strategies while achieving the same performance. To the best of our knowledge, we are the first to identify the importance of model-aware similarity and exploit it to address the problem of redundancy in AL. 2 Related Work Active learning has a long and successful history in the field of machine learning (Dasgupta et al., 2009; Awasthi et al., 2017). However, as the learning models have become more complex, especially with the advent of deep learning, the known theoretical results for active learning are no longer applicable (Shen et al., 2018). This has prompted a diverse range of heuristics to adapt the active learning framework to deep learning models (Shen et al., 2018). Many AL strategies have been proposed (Sha and Saul, 2007; Haffari et al., 2009; Bloodgood and Callison-Burch, 2010; Blundell et al., 2015; Gal and Ghahramani, 2016a), however, since they choose the examples independently, the problem of redundancy (Section 1) applies to all. We experiment with various NLP tasks like named entity recognition (NER) (Nadeau and Sekine, 2007), part-of-speech tagging (POS) (Mar-cus et al., 1993), neural machine translation (NMT) (Hutchins, 2004; Nepveu et al., 2004; Bahdanau et al., 2014; Cho et al., 2014; Sutskever et al., 2014; Ortiz-Martnez, 2016) and so on (Landes et al., 1998; Tjong Kim Sang and Buchholz, 2000). The tasks chosen by us form the backbone of many practical problems and are known to be computationally expensive during both training and inference. Many deep learning models have recently advanced the state-of-art for these tasks (Bahdanau et al., 2014; Lample et al., 2016; Siddhant and Lipton, 2018). Our proposed approach is compatible with any NLP model, provided it supports the usage of an AL strategy. Existing approaches have used model-independent similarity scores to promote diversity in the chosen examples. For instance, in Chen et al. (2015), the authors use cosine similarity to pre-calculate pairwise similarity between examples. We instead argue in favor of model-aware similarity scores and learn an expressive notion of similarity using neural networks. We compare our approach with a modified version of this baseline using cosine similarity on Infersent embeddings (Conneau et al., 2017). We use M to denote the model being trained for a given task. M has a module called encoder for encoding the input sentences. For instance, the encoder in M may be modeled by an LSTM (Hochre-iter and Schmidhuber, 1997). A measure of similarity between examples is required to discover redundancy. The simplest solution is to compute the cosine similarity between input sentences (Chen et al., 2015; Shen et al., 2018) using, for instance, the InferSent encodings (Con-neau et al., 2017). However, sentences that have a low cosine similarity may still be similar in the context of the downstream task. Model M has no incentive to distinguish among such examples. A good strategy is to label a diverse set of sentences from the perspective of the model. For example, it is unnecessary to label sentences that use different verb forms but are otherwise similar if the task is agnostic to the tense of the sentence. A straightforward extension of cosine similarity to the encodings generated by model M achieves this. However, a simplistic approach like this would likely be incapable of discovering complex similarity patterns in the data. Next, we describe two approaches that use more expressive model-aware similarity measures. In this approach, we use a Siamese twin's network (Bromley et al., 1994) to compute the pairwise similarity between encodings obtained from model M . A Siamese twin's network consists of an encoder (called the Siamese encoder) that feeds on the output of model M 's encoder. The outputs of the Algorithm 1: Active 2 Learning Data: D 1 : task dataset; D 2 : auxiliary similarity dataset Input: D  2% of dataset D 1 ; D  D 1  D ; // unlabeled data Output: Labeled data Initialization: D ; M  TRAIN ( D ); MA 2 L  TRAIN ( M ( D 2 ) ); for i  1 to l do S  AL (D) ; // top 2% confused samples if // Model-Aware Siamese then for each pair (s m , s n ) in S do S [ m, n ]  MA 2 L (s m , s n ) ; R  CLUSTER ( S ); else // Integrated Clustering R  MA 2 L ( S ) ; R  ANNOTATE ( R ); D  D  R ; M  RETRAIN ( D ) Siamese encoder are used for computing the similarity between each pair of examples a and b as: sim( a, b ) = exp ( || o a  o b || 2 ) , (1) where o a and o b are the outputs of the Siamese encoder for sentences a and b respectively. Let N denote the number of examples chosen by an AL strategy. We use the Siamese network to compute the entries of an N  N similarity matrix S where the entry S ab = sim( a, b ) . We then use the spectral clustering algorithm (Ng et al., 2002) on the similarity matrix S to group similar examples. A fixed number of examples from each cluster are added to the training dataset after annotation by experts. We train the Siamese encoder to predict the similarity between sentences from the SICK (Sentences Involving Compositional Knowledge) dataset (Marelli et al., 2014) using mean squared error. This dataset contains pairs of sentences with manually annotated similarity scores. The sentences are encoded using the encoder in M and then passed on to the Siamese encoder for computing similarities. The encoder in M is kept fixed while training the Siamese encoder. The trained Siamese encoder is then used for computing similarity between sentences selected by an AL strategy for the given NLP task as described above. As M is trained over time, the distribution of its encoder output changes, and hence we periodically retrain the Siamese network to sustain its model-awareness. The number of clusters and the number of examples drawn from each cluster are user-specified hyper-parameters. The similarity computation can be done efficiently by computing the output of the Siamese encoder for all N examples before evaluating equation 1, instead of running the Siamese encoder O ( N 2 ) times. The clustering algorithm runs in O ( N 3 ) time. For an AL strategy to be useful, it should select a small number of examples to benefit from interactive and intelligent labeling. We expect N to be small for most practical problems, in which case the computational complexity added by our approach would only be a small fraction of the overall computational complexity of training the model with active learning (see Figure 1). While the approach described in Section 3.2 works well for small to moderate values of N , it suffers from a computational bottleneck when N is large. We integrate the clustering step into the similarity computation step to remedy this (see Figure 1) and call the resultant approach as Integrated Clustering Model ( Int Model ). Here, the output of model M 's encoder is fed to a clustering neural network C that has K output units with the softmax activation function. These units correspond to the K clusters, and each example is directly assigned to one of the clusters based on the softmax output. To train the network C , we choose a pair of similar examples (say a and b ) and randomly select a negative example (say c ). We experimented with both SICK and Quora Pairs dataset 3 . All examples are encoded via the encoder of model M and then passed to network C . The unit with the highest probability value for a is treated as the ground-truth class for b . Minimizing the objective given below maximizes the probability of b belonging to its ground truth class while minimizing the probability of c belonging to the same class: L ( a, b, c ) =   1 log p bi a   2 log(1  p ci a ) +  3 K (cid:88) k =1 p bk log p bk . (2) Here  1 ,  2 , and  3 are user-specified hyperparameters, p xj is the softmax output of the j th unit for example x , j = 1 , 2 , . . . , K , x = a, b, c , and i a = arg max j { 1 , 2 ,...K } p aj . The third term encourages the utilization of all the K units across examples in the dataset. As before, a trained network C is used for clustering examples chosen by an AL strategy, and we select a fixed number of examples from each cluster for manual annotation. It is important to note that: (i) These methods are not AL strategies. Rather, they can be used in conjunction with any existing AL strategy. Moreover, given a suitable Siamese encoder or clustering network C , they apply to any model M . (ii) Our methods compute model-aware similarity since the input to the Siamese or the clustering network is encoded using the model M . The proposed networks also adapt to the underlying model as the training progresses. Algorithm 1 describes our general approach called Active 2 Learning. We establish the effectiveness of our approaches by demonstrating that they: (i) work well across a variety of NLP tasks and models, (ii) are compatible with several popular AL strategies, and (iii) further reduce the data requirements of existing AL strategies, while achieving the same performance. In particular, we experiment 1 with two broad categories of NLP tasks: (a) Sequence Tagging (b) Neural Machine Translation. Table 1 lists these tasks and information about the corresponding datasets (in-cluding the two auxiliary datasets for training the Siamese network (Section 3.2)) used in our experiments. We begin by describing the AL strategies for the two kinds of NLP tasks. Margin-based strategy: Let s ( y ) = P  ( Y = y | X = x ) be the score assigned by a model M with parameters  to output y for a given example x . Margin is defined as the difference in scores obtained by the best scoring output y and the second best scoring output y (cid:48) , i.e.: M margin = max y s ( y )  max y (cid:48) (cid:54) = y max s ( y (cid:48) ) , (3) where, y max = arg max y s ( y ) . The strategy selects examples for which M margin   1 , where  1 is a hyper-parameter. We use Viterbi's algorithm (Ryan and Nudd, 1993) to compute the scores s ( y ) . 1 Codes for the experiments are available at the following github link: https://github.com/parag1604/A2L. Entropy-based strategy: All the NLP tasks that we consider require the model M to produce an output for each token in the sentence. Let x be an input sentence that contains n ( x ) tokens and define  s j = max o O P  ( y j = o | X = x ) to be the probability of the most likely output for the j th token in x . Here O is set of all possible outputs and y j is the output corresponding to the j th token in x . We define the normalized entropy score as: M entropy =  1 n ( x ) n ( x ) (cid:88) j =1  s j ( y ) log  s j ( y ) . A length normalization n ( x ) is added to avoid bias due to the example length as it may be undesirable to annotate longer length examples (Claveau and Kijak, 2017). The strategy selects examples with M entropy   2 , where  2 is a hyper-parameter. Bayesian Active Learning by Disagreement (BALD): Due to stochasticity, models that use dropout (Srivastava et al., 2014) produce a different output each time they are executed. BALD (Houlsby et al., 2011) exploits this variability in the predicted output to compute model uncertainty. Let y ( t ) denote the best scoring output for x in the t th forward pass, and let N be the number of forward passes with a fixed dropout rate, then: M bald = 1  count(mode( y (1) , . . . , y ( N ) )) N . Here the mode( . ) operation finds the output which is repeated most often among y (1) , . . . , y ( N ) , and the count( . ) operation counts the number of times this output was encountered. This strategy selects examples with M bald   3 (hyper-parameter). Least Confidence (LC) This strategy estimates the uncertainty of a trained model on a source sentence x by calculating the conditional probability of the prediction  y conditioned on the source sen-tence(Lewis and Catlett, 1994). Coverage Sampling (CS) A translation model is said to cover the source sentence if it translates all of its tokens. Coverage is estimated by mapping a particular source token to its appropriate target token, without which the model may suffer from under-translation or over-translation issues (Tu et al., 2016). Peris and Casacuberta (2018) proposed to use translation coverage as a measure of uncertainty by: MCS = (cid:80) n ( x ) j=1 log(min( (cid:80) n ( y ) i=1  i , j , 1)) n ( x ) (7) Here  i , j denotes the attention probability calculated by the model for the j th source word in predicting the i th target word. It can be noted that the coverage score will be 0 for samples for which the model almost fully covers the source sentences. Attention Distraction Sampling (ADS) Peris and Casacuberta (2018) claimed that in translating an uncertain sample, the model's attention mechanism would be distracted (dispersed throughout the sentence). Such samples yield attention probability distribution with light tails (e.g., uniform distribu-tion), which can be obtained by taking the Kurtosis of the attention weights for each target token y i . where 1 n ( x ) is the mean of the distribution of the attention weights (for a target word) over the source words. The kurtosis value will be lower for distributions with light tails, so the average of the negative Figure 1: Comparison of time taken for one data selection step in NMT task by the Model Aware (MA) Siamese and Integrated Clustering (Int) Model across different ALS. kurtosis values for all words in the target sentence is used as the distraction score. For sequence tagging, we use two kinds of architectures: CNN-BiLSTM-CRF model (CNN for character-level encoding and BiLSTM for word-level encoding) and a BiLSTM-BiLSTM-CRF model (BiLSTM for both character-level and word-level encoding) (Lample et al. (2016); Siddhant and Lipton (2018)). For the translation task, we use LSTM based encoder-decoder architecture with Bahdanau attention (Bahdanau et al., 2014). These models were chosen for their performance and ease of implementation. The Siamese network used for model-aware similarity computation (Section 3.2) consists of two bidirectional LSTM (BiLSTM) encoders. We pass each sentence in the pair from the SICK dataset to model M and feed the resulting encodings to the Siamese BiLSTM encoder. The output is a concatenation of terminal hidden states of the forward and backward LSTMs, which is used to compute the similarity score using (1). As noted before, we keep model M fixed while training the Siamese encoders and use the trained Siamese encoders for computing similarity between examples chosen by an AL strategy. We maintain the model-awareness by retraining the Siamese after every 10 iterations. The architecture of the clustering model C (Sec-tion 3.3) is similar to that of the Siamese encoder. Additionally, it has a linear layer with a softmax activation function that maps the concatenation of terminal hidden states of the forward and backward LSTMs to K units, where K is the number of clusters. To assign an input example to a cluster, we first pass it through the encoder in M and feed the resulting encodings to the clustering model C . The example is assigned to the cluster with the highest softmax output. This network is also retrained after every 10 iterations to retain model-awareness. The initial data splits used for training the model M were set at 2% of randomly sampled data for Sequence Tagging ( 20% for NMT). These are in accordance with the splitting techniques used in the existing literature on AL (Siddhant and Lipton, 2018; Liu et al., 2018). The model is then used to provide input to train the Siamese/Clustering network using the SICK/Quora Pairs. At each iteration, we gradually add another 2% of data for sequence tagging ( 5% for NMT) by retrieving low confidence examples using an AL strategy, followed by clustering to extract the most representative examples. We average the results over five independent runs with randomly chosen initial splits. [Hyperparameters details in Appendix A]. We claim that A 2 L mitigates the redundancies in the existing AL strategies by working in conjunction with them. We validate our claims by comparing our approaches with three baselines that highlight the importance of various components. Cosine: Clustering is done based on cosine similarity between last output encodings (correspond-ing to sentence length) from encoder in M . Although this similarity computation is model-aware, it is simplistic and shows the benefit of using a more expressive similarity measure. None: In this baseline, we use the AL strategy without applying Active 2 learning to remove redundant examples. This validates our claim about redundancy in examples chosen by AL strategies. Random: No active learning is used, and random examples are selected at each time. We perform ablation studies to demonstrate the utility of model-awareness using these baselines: Infersent: Clustering is done based on cosine similarity between sentence embeddings (Chen et al., 2015) obtained from a pre-trained InferSent model (Conneau et al., 2017). This similarity computation is not model-aware and shows the utility of model-aware similarity computation. Iso Siamese: To show that the Siamese network alone is not sufficient and model-awareness is needed, in this baseline, we train the Siamese network by directly using GloVe embeddings of the words as input rather than using output from the 5 We process the dataset to use only those sentences which are present in at least 5 other pairs. We retrieve 16000 sets, each with a source sentence and 5 other samples (comprising both positive and negative labels). An additional 1000 sets were generated for evaluation. model M 's encoder. This similarity, which is not model-aware, is then used for clustering. Figure 2 compares the performance of our methods with baselines. It shows the test-set metric on the y -axis against the percentage of training data used on the x -axis for all tasks. See Figures 7 and 8 in the Appendix for additional results. 1. As shown in Figure 2, our approach consistently outperforms all baselines on all tasks. Note that one should observe how fast the performance increases with the addition of training data (and not just the final performance) as we are trying to evaluate the effect of adding new examples. Our ablation studies in Figure 3 show the utility of using model-aware similarity. 2. In sequence tagging, we match the performance obtained by training on the full dataset using only a smaller fraction of the data ( 3  25 % less data as compared to state-of-art AL strategies) (Table 2). On a large dataset in NMT task (Europarl), A 2 L takes  4300 sentences fewer than the Least Confidence AL strategy to reach a Bleu score of 12 . 3. While comparing different AL strategies is not our motive, Figure 2 also demonstrates that one can achieve performance comparable to a complex AL strategy like BALD, using simple AL strategies like margin and entropy, by using the proposed A 2 L framework. 4. Additionally, from Figure 1, it can be observed that for one step of data selection: (i) The proposed MA Siamese model adds minimal overhead to the overall AL pipeline since it takes less than 5 additional seconds (  112 of the time taken for ALS); (ii) By approximating the clustering step, Integrated Clustering (Int) Model further reduces the overhead down to 2 seconds. However, owing to this approximation, MA Siamese is observed to perform slightly better than the Int Model (Fig 3). A comparison of training time for various stages of the A 2 L pipeline is provided in Figure 4. We wish to state that our approach should be evaluated not in terms of the gain in the F1 score but Figure 5: [Best viewed in color] Qualitative case study to convey the notion of redundancy and the model aware similarity. in terms of the reduction in data required to achieve the same (3-25 % on multiple datasets). More importantly, this improvement comes at a negligible computation overhead cost. The reported improvements are not relative with respect to any baseline but represent an absolute value and are very signifi-cant in the context of similar performance improvements reported in the literature. In Figure 5, we provide a qualitative case study that demonstrates the problem of redundancy. This paper shows that one can further reduce the data requirements of Active Learning strategies by proposing a new method, A 2 L, which uses a model-aware-similarity computation. We empirically demonstrated that our proposed approaches consistently perform well across many tasks and AL strategies. We compared the performance of our approach with strong baselines to ensure that the role of each component is properly understood. This work was funded by British Telecom India Research Center project on Advanced Chatbot.
abstain abstain abstain objective objective result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method objective abstain objective abstain abstain abstain abstain method method abstain objective abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain other abstain method method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain objective method other