Because large, human-annotated datasets suffer from labeling errors, it is crucial to be able to train deep neural networks in the presence of label noise. While training image classification models with label noise have received much attention, training text classification models have not. In this paper, we propose an approach to training deep networks that is robust to label noise. This approach introduces a non-linear processing layer ( noise model ) that models the statistics of the label noise into a convolutional neural network (CNN) architecture. The noise model and the CNN weights are learned jointly from noisy training data, which prevents the model from overfitting to erroneous labels. Through extensive experiments on several text classification datasets, we show that this approach enables the CNN to learn better sentence representations and is robust even to extreme label noise. We find that proper initialization and regularization of this noise model is critical. Further, by contrast to results focusing on large batch sizes for mitigating label noise for image classification, we find that altering the batch size does not have much effect on classification performance. Deep Neural Networks (DNNs) have led to sig-nificant advances in the fields of computer vision (He et al., 2016), speech processing (Graves et al., 2013) and natural language processing (Kim, 2014; Young et al., 2018; Devlin et al., 2018). To be effective, supervised DNNs rely on large amounts of carefully labeled training data. However, it is not always realistic to assume that example labels are clean. Humans make mistakes and, depending on the complexity of the task, there may be disagreement even among expert labelers. Further, samples drawn from the class conditional densities with overlapping supports gives rise to the label noise in training datasets. To support noisy labels in data, we need new training methods that can be used to train DNNs directly from the corrupted labels to significantly reduce human labeling efforts. Zhu and Wu (2004) perform an extensive study on the effect of label noise on classification performance of a classifier and find that noise in input features is less important than noise in training labels. In this work, we add a noise model layer on top of our target model to account for label noise in the training set, following (Jindal et al., 2016; Sukhbaatar et al., 2014). We provide extensive experiments on several text classification datasets with artificially injected label noise. We study the effect of two different types of label noise; Uniform label flipping (Uni) , where a clean label is swapped with another label sampled uniformly at random; and Random label flipping (Rand) where a clean label is swapped with another label from the given number of labels sampled randomly over a unit simplex. We also study the effect of different initialization, regularization, and batch sizes when training with noisy labels. We observe that proper initialization and regularization helps the noise model learn to be robust to even extreme amounts of noise. Finally, we use low-dimensional projections of the features of the training examples to understand the effectiveness of the noise model. The rest of the paper is organized as follows. Section 2 discusses the various approaches in literature to handle label noise. In Section 3, we describe the problem statement along with the proposed approach. We describe the experimental setup and datasets in Section 4. We empirically evaluate the performance of the proposed approach along with the discussion in Section 5 and finally conclude our work in Section 6. Learning from label noise is a widely studied problem in the classical machine learning setting. Earlier works (Brodley and Friedl, 1999; Rebbapragada and Brodley, 2007; Manwani and Sastry, 2013) consider learning from noisy labels for a wide range of classifiers including SVMs (Natarajan et al., 2013) and fisher discriminants (Lawrence, 2001). Traditional approaches handle label noise by detecting and eliminating the corrupted labels. More details about these approaches can be found in (Frenay and Verleysen, 2014). Recently, DNNs have made huge gains in performance over traditional methods on large datasets with very clean labels. However large real-world datasets often contain label errors. A number of works have attempted to address this problem of learning from corrupted labels for DNNs. These approaches can be divided into two categories; attempts to mitigate the effect of label noise using auxiliary clean data, and attempts to learn directly from the noisy labels. Presence of auxiliary clean data: This line of research exploits a small, clean dataset to correct the corrupted labels. For instance, Li et al. (2017) learn a teacher network with clean data to re-weight a noisy label with a soft label in the loss function. Similarly, Veit et al. (2017) use the clean data as a label correction network. One can use this auxiliary source of information to do inference over latent clean labels (Vahdat, 2017). Further, Yao et al. (2018) models the auxiliary trustworthiness of noisy image labels to alleviate the effect of label noise. Though these methods show very promising results, the absence of clean data in some situations might hinder the applicability of these methods. Learning directly from noisy labels: This research directly learns from the noisy labels by designing a robust loss function, or by modeling the latent labels. For instance, Reed et al. (2014), apply bootstrapping to the loss function to have consistent label prediction for similar images. Similarly, Joulin et al. (2016) alleviate the label noise effect by adequately weighting the loss function using the sample number. Jiang et al. (2017) propose a sequential meta-learning model that takes in a sequence of loss values and outputs the weights for the labels. Ghosh et al. (2017) further explores the conditions on loss functions such that the loss function is noise tolerant. A number of approaches learn the transition from latent labels to the noisy labels. For example, Mnih and Hinton (2012) propose a noise adaptation framework for symmetric label noise. Based on this work, several other works (Sukhbaatar et al., 2014; Jindal et al., 2016; Patrini et al., 2017; Han et al., 2018) account for the label noise by learning a noisy layer on top of a DNN where the learned transition matrix represents the label flip probabilities. Similarly, Xiao et al. (2015) propose a probabilistic image conditioned noise model. Azadi et al. (2015) proposed an image regularization technique to detect and discard the noisy labeled images. Other approaches include building two parallel classifiers (Misra et al., 2016) where one classifier deals with image recognition and the other classifier models humans reporting bias. All of these approaches have targeted image classification. In this work, we propose a framework for learning from noisy labels for text classification using a DNN architecture. Similar to (Sukhbaatar et al., 2014; Jindal et al., 2016; Patrini et al., 2017), we append a non-linear processing layer on top of this architecture to model the label noise. This layer helps the base architecture to learn better representations, even in the presence of label noise. We empirically show that, for better classification performance, the knowledge of noise transition matrix is not needed. Instead, the process forces the DNN to learn better sentence representations. In a supervised text classification setting where x i  R d is a d -dimensional word embedding of the i th word in a sentence of length l (padded wherever necessary), we represent the sample as a temporal embedding matrix X  R d  l which belongs to one of the K classes. Let the noise-free training set be denoted by D = { ( X 1 , y 1 ) , ( X 2 , y 2 ) ,    , ( X n , y n ) } , where y i  { 1 , . . . , K } represents the category of i th sample, n is the total number of training samples, and there is an unknown joint distribution p ( X , y ) on the sample/label pairs. This temporal representation of a sample X is fed as input to a classifier on the training set D with sample categories y . However, as mentioned in Section 2, we cannot access the true noise-free samples labels and instead, observe noisy labels corrupted by an unknown noise distribution. Let this noisy training set be denoted by D (cid:48) = { ( X 1 , y (cid:48) 1 ) , ( X 2 , y (cid:48) 2 ) ,    , ( X n , y (cid:48) n ) } , where y (cid:48) i represents the corrupted label for the sentence X i . In this work, we suppose the label noise is class-conditional , where the noisy label y (cid:48) i depends only on the true label y i , but not on the input X i or any other labels y j or y (cid:48) j . Under this model, the label noise is characterized by the conditional distribution p ( y (cid:48) = i | y = j ) =  ij , which we describe via the K  K column-stochastic matrix  ij , parameterized by a matrix Q = {  ij } . In our experiments, we artificially inject label noise into the training and validation sets. We fix the noise distribution  ij and, for a training sample, we generate a noisy label by drawing i.i.d from this noise distribution  ij . However, we do not alter the test labels. Though the proposed approach works for any noise distribution, for this study, we focus on two different types of label flip distributions. We use a noise model parameterized by the overall probability of a label error, denoted by 0  p  1 . For a noise level p , we set the noise distribution matrix  = (1  p ) I + p K IIK , (1) and we call it Uniform label flip noise model . Here, I represents the identity matrix and II denotes the all-ones matrix. Similarly, we describe the random label flip noise model as  = (1  p ) I + p  , (2) where I is the identity matrix, and  is a matrix with zeros along the diagonal and remaining entries of each column are drawn uniformly and independently from the K  1 -dimensional unit simplex. The label error probability for each class is p , while the probability distribution within the erroneous classes is drawn uniformly at random. Our objective is to train a classifier on the noisy labeled sample categories on the training set D (cid:48) such that it jointly makes accurate predictions of the true label y and learns the noise transition matrix simultaneously, given X . For the noisy dataset D (cid:48) , it is straightforward to train a classifier that predicts the noisy labels using conditional distribution for the noisy labeled input sentence X : p ( y (cid:48) =  y (cid:48) | X ) = (cid:88) i (cid:16) p ( y (cid:48) =  y (cid:48) | y =  y i ) p ( y =  y i | X ) (cid:17) . (3) One can learn the classifier associated with p ( y (cid:48) =  y (cid:48) | x ) via standard training on the noisy set D (cid:48) . To predict the clean labels by learning the conditional distribution p ( y =  y i | x ) requires more effort, as we cannot extract the clean classifier from the noisy classifier when the label noise distribution is unknown. We refer to the DNN model without the final layer as the base model or network without noise model (WoNM) . This model, along with the non-linear layer, is trained via back-propagation on the noisy training dataset. The non-linear processing layer in the noise model transforms the base model outputs to match the noisy labels during the forward pass better and presents the denoised labels to the base model during the backward pass. The noise layer is parameterized by a square matrix   RK  K ). At test time, we remove this learned noise model and use the output of the base model as final predictions. We refer to the base model parameters as  . The base model outputs a probability distribution over the number of K categories denoted as p ( y =  y i | X ; )  i  { 1 , 2 ,    , K } . During the forward pass the noise model transforms this output to obtain the noisy labels as p (cid:0) y (cid:48) | X ;  ,  (cid:1) =  (  p ( y | X ; )) , (4) where  (  ) represents the usual softmax operator. Note that both the equations (3) and (4) compute the probability distribution over noisy labels  our noise model does not learn a noise transition matrix. However, we assert that the knowledge of exact noise statistics is neither necessary nor suffi-cient for the better prediction results. We learn the base model parameters  and the noise model parameters  by maximizing the log likelihood (4) over all of the training samples, minimizing the cross-entropy loss: L ( , ; D (cid:48) ) =  1 n n (cid:88) i =1 log (cid:2) p (cid:0) y (cid:48) | X i ;  ,  (cid:1)(cid:3) =  1 n n (cid:88) i =1 log [  (  p ( y | X i ; ))] y i (5) Similar to (Sukhbaatar et al., 2014), we initialize the noise model weights to the identity matrix. Since DNNs have high capacity, we may encounter the situation when the base model itself absorbs all the label noise and, thus, the noise model does not learn anything at all. In order to avoid this situation, and to prevent overfitting, we apply l 2 regularization to the noise model. However, we want the noise model to overfit the label noise. In the experiment section, we observe that with proper regularization and weight initialization the noise model absorbs most of the label noise. Finally, we train the entire network according to the following loss function: L =  1 n n (cid:88) i =1 log [  (  p ( y | X i ; ))] y i + 1 2  ||  || 22 . Here,  is a tuning parameter and we validate the value of  by repeating the experiment multiple times with multiple  values over different datasets and choose the one with better classification performance on the validation set for the respective datasets. A value of  = 0 . 01 works best. In this section, we empirically evaluate the performance of the proposed approach for text classification and compare our results with the other methods. In all the experiments, we use a publicly-available deep learning library Baseline  a fast model development tool for NLP tasks (Pres-sel et al., 2018). For all the different datasets, we choose a commonly-used, high-performance model from (Kim, 2014) as a base model. To examine the robustness of the proposed approach, we intentionally flip the class labels with 0% to 70% label noise, in other words: T e x t D a t a Dataset K L N T Type SST-2 2 19 76961 1821 Balanced TREC 6 10 5000 500 Not Balanced AG-News 4 38 110K 10K Balanced DBpedia 14 29 504K 70K Balanced Table 1: Summary of text classification datasets; K: denotes the number of classes, L: represents the average length of sentence, N: denotes the number of training samples, T: represents the number of test samples, Type: describes whether the dataset is balanced. p  { 0 . 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 } , and observe the effect of different types of label flipping, such as uniform ( Uni ) and random ( Rand ) label flipping, along with instance-dependent label noise. For all the experiments, we use early stopping based on validation set accuracy where the class labels in validation are also corrupted. We indicate the performance of a standard deep network Without Noise model (WoNM) on the noisy label dataset. We also plot the results for the stacked Noise Model Without Regularization (NMWoRegu) and stacked Noise Model With Regularization (NMwRegu) . Unless otherwise stated, in all the deep networks with the stacked noise model, we initialize the noise layer parameters as an identity matrix. We further analyze the effect of the noise layer initialization on the overall performance. We define TDwRegu as the stacked noise model with regularization, initialized with true injected noise distribution and RandwRegu as the stacked noise model with regularization, initialized randomly. We run all experiments five times and report the mean accuracy. Here, we describe all the text classification datasets used to evaluate the performance of the proposed approach. The base model architecture is the same for all datasets. For each set, we tune the number of filter windows and filter lengths using the development set. Along with the description, we also provide the hyper-parameters we selected for each. Table 1 summarizes the basic statistic of the datasets. 1. SST-2 1 (Socher et al., 2011): Stanford Sentiment Treebank dataset for predicting the sentiment of movie reviews. The classification task involves detecting positive or negative reviews. Using the base model with 1 http://nlp.stanford.edu/sentiment/ SST2 BatchSize 50 100 LabelFlips Random Random Noise% CleanLabels 10 20 30 40 45 47 50 0 10 20 30 40 45 47 50 WoNM 87 . clean labels we obtain classification accuracy of 87 . 27% . For this dataset, the base model network architecture consists of an input and embedding layer + [3 , 4 , 5] feature windows with 100 feature maps each and dropout rate 0 . 5 with batch size 50. 2. TREC 2 (Voorhees and Tice, 1999): A question classification dataset consisting of fact based questions divided into broad semantic categories. We use a six-class version of TREC dataset. For this dataset, the base model network architecture consists of an input and embedding layer + [3] one feature windows with 100 feature maps and dropout rate 0 . 5 with batch size 10. 3. Ag-News 3 (Zhang et al., 2015): A large-scale, four-class topic classification dataset. It contains approx 110K training samples. For this dataset, the base model network architecture consists of Input layer + Embedding layer + [3 , 4 , 5] feature windows with 200 feature maps and dropout rate 0 . 5 with batch size 100. 4. DBpedia 3 (Zhang et al., 2015): A large scale 14-class topic classification dataset containing 36 K training samples per category. For this dataset, the base model network architecture consists of Input layer + Embedding layer + [1 , 2 , 3 , 4 , 5 , 7] feature windows with 400 feature maps each and dropout rate 0 . 5 with batch size 1024. 3 http://www.di.unipi.it/gulli/AG_ corpus_of_news_articles.html For all the datasets, we use Rectified Linear Units (ReLU) and fix the base model architecture. We use early stopping on dev. sets for all the datasets. We run all the experiments 5 times and report the average classification accuracy in Table 2. We train all the networks end-to-end via stochastic gradient descent over shuf-fled mini-batches with the Adadelta update rule (Zeiler, 2012) except for the DBpedia, where we use SGD. In order to improve base model performance, we initialize the word embedding layer with the publicly available word2vec word vectors (Mikolov et al., 2013) for all the datasets except for DBpedia, where we use GloVe embeddings (Pennington et al., 2014). We evaluate the performance of our model in Table 2 for each datasets in the presence of uniform and random label noise and compare the performance with the base model ( WoNM ) as our baseline. For all datasets, the proposed approach is significantly better than the baseline for both random and uniform label noise. For all datasets, we observe a gain of approximately 30% w.r.t the baseline in the presence of extreme label noise. We do observe a drop in classification accuracy as we increase the percentage of label noise but even at the extreme label noise our method outperformed the baseline method. Interestingly, if we assume an oracle to determine prior knowledge of true noise distribution ( TDwRegu01 ), it does not necessarily improve classification performance, especially for multi-class classification problems. For binary classification, using the SST-2 dataset, we did observe that the noise model initialized with the true noise distribution works better than all the other models. In addition to this, we also observe a slight performance gain for the proposed approach over the baseline with clean labels  perhaps due to label noise inherent in the datasets. The NMwRegu01 performs better in all cases for both types of label noise. We plot the weight matrix learned by all the noise models in all the noise regimes. For brevity, we only plot the weight matrix for AG-News datasets with 30% label noise in Fig.1. We find that l 2 regularization diffuses the diagonal weight elements and learns more smoothed off-diagonal elements which resemble the corresponding input label noise distribution in Fig. 1d. This also means that, without regularization, the noise model has less ability to diffuse the diagonal elements which leads to poor classification performance. Therefore, we use a regularizer ( l 2 ) to diffuse the diagonal entries. In some cases, especially for low label noise, we find the l 2 regularization with a small penalty works better than a large penalty since, for low label noise, learning a less diffuse noise is beneficial. The proposed approach scales to a large number of label categories, as evident from the experiments on DBpedia dataset in the last row of Table .2. We observe the effect of different gain values on the overall performance of the proposed network in Fig. 2 where on x-axis we vary the scaling factor  a function of number of classes in the dataset. We plot the classification performance for the DBpedia dataset with 50% random noise. For each noise model in Fig. 2a, we find that setting the gain to K works best and any other gain results in poor performance. In Fig. 2b we plot the Frobenius norm of the learned noise model weights with respect to the different gain values. We find that, using the high gain initialization, the model learns a high noise model norm, resulting in poor classification performance. This finding provides support to the claim in (Liao et al., 2018) that higher layer norm leads to highest test errors. 5.3 Effect of Batch Size We also observe the effect of different batch sizes on performance as described in (Rolnick et al., 2017). For all datasets, we do observe small performance gains for highly non-uniform noisy labels, for instance 70% , in Fig. 3 row 2. However, for uniform label flips, we do not observe perfor-TRB TRPr Data(N % ) WoNM Noisy True NMwRegu01 Noisy True SST2 (40%) 70.24 70.95 79.24 82.32 73.90 83.25 AG (70%) 59.70 52.44 79.18 90.33 86.27 89.4 AG (60%) 83.25 68.8 88.28 90.45 87.77 90.78 TREC (40%) 66.80 63.4 79.0 73.40 69.6 83.2 TREC (20%) 83.6 80.0 86.0 87.40 83.6 90.0 Table 3: SVM Classification mance gains with increasing batch size. We further investigate the performance of the proposed approach on instance-dependent label noise by flipping each class labels with different noise percentages as shown in Fig. 4a. For brevity, we present results on AG-News dataset in Fig. 4. On this type of label noise, the performance of the proposed approach is far better than the baseline with a performance improvement of  6% . The noise model learned by the proposed approach is shown in Fig. 4b and we show the normalized weight matrix in Fig. 4c. We observe that the learned noise model is able to capture the input label noise statistics and is highly correlated to the input noise distribution with Pearson Correlation Coefficient 0 . 988 . In order to further understand the noise model, we first train the base model and the proposed model on noisy labels. Afterward, we collect We get two different sets of feature representations, one corresponding to the base model ( TRB ), and the other corresponding to the proposed model ( TRPr ). Given these learned feature representations  the artificially injected noisy labels and the true labels of the training data  we learn two different SVMs for each model, with and without noise. For the base model, for both SVMs, we use TRB representation as inputs and train the first SVM with the true labels as targets and the second SVM with the unreliable labels as targets. Similarly, we train two SVMs for the proposed model. After training, we evaluate the performance of all the learned SVMs on clean test data in Table 3, where the 1st column represents the corresponding model performance, Noisy and True column represents the SVM performance when trained on noisy and clean labels, respectively. We run these experiments for different datasets with different label noise. The SVM, trained on TRB and noisy labels, is very close to the base model performance (3). This suggests that the base model is just fitting the noisy labels. On the other hand, when we train an SVM on the TRPr representations with true labels as targets, the SVM achieves the proposed model performance. This means that the proposed approach helps the base model to learn better feature representations even with the noisy targets, which suggest that this noise model is learning a label de-noising operator. We analyze the representation of training samples in feature domain by plotting the t-SNE embeddings (Van Der Maaten, 2014) of the TRB and TRPr. For brevity, we plot the t-SNE visualizations for TREC dataset with 50% label noise in Fig. 5 . For each network, we show two different t-SNE plots. For example in Fig. 5a we plot two rows of t-SNE embeddings for the proposed model. In the first row of Fig. 5a, each training sample is represented by its corresponding true label, while in the second row (the noisy label plot) each training sample is represented by its corresponding noisy label. We observe that, as the learning process progresses, the noise model helps the base model to cluster the training samples in the feature domain. With each iteration, we can see the formation of clusters in Row 1. However, in Row 2, when the noisy labels are superimposed, the clusters are not well separated. This means that the noise model denoises the labels and presents the true labels to the base network to learn. In Fig. 5b, we plot two rows of t-SNE embeddings of the TRB representations. It seems that the network directly learns the noisy labels. This provides further evidence to support (Zhang et al., 2016)'s finding that the deep network memorizes data without knowing of true labels. In Row 2 of Fig. 5b, we can observe that the network learns noisy features representations which can be well clustered according to given noisy labels. In this work, we propose a framework to enable a DNN to learn better sentence representations in the presence of label noise for text classification tasks. To model the label noise, we append a nonlinear noise model on top of the standard DNN architecture. With proper initialization and regularization, the noise model is able to absorb most of the label noise and helps the base model to learn better sentence representations. We thank the anonymous reviewers for their detailed and insightful comments. We would also like to thank Patrick Haffner, Sagnik Ray Choud-hury, Yanjie Zhao and Amy Hemmeter for their valuable discussions with us during the course of this research.
abstain abstain objective abstain abstain result result objective abstain abstain abstain abstain abstain objective abstain method method method method result method abstain abstain objective objective other other other other other other other other other other other other other other other other other other other other other abstain other other other other objective abstain other abstain other method other abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain method abstain abstain method method abstain abstain method method method other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain method method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain method abstain abstain abstain result result result result abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain result result result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain abstain result objective abstain abstain other other