Early stopping, which is widely used to prevent overfitting, is generally based on a separate validation set. However, in low resource settings, validation-based stopping can be risky because a small validation set may not be sufficiently representative, and the reduction in the number of samples by validation split may result in insufficient samples for training. In this study, we propose an early stopping method that uses unlabeled samples. The proposed method is based on confidence and class distribution similarities. To further improve the performance, we present a calibration method to better estimate the class distribution of the unlabeled samples. The proposed method is advantageous because it does not require a separate validation set and provides a better stopping point by using a large unlabeled set. Extensive experiments are conducted on five text classification datasets and several stop-methods are compared. Our results show that the proposed model even performs better than using an additional validation set as well as the existing stop-methods, in both balanced and imbalanced data settings. Our code is available at https://github. com/DMCB-GIST/BUS-stop . Early stopping, a form of regularization, is a widely used technique to prevent a model from over-fitting (Yao et al., 2007; Zhang et al., 2017). It is generally based on a separate validation set (Goodfel-low et al., 2016). While monitoring the validation performance during training, the training process stops when the validation error starts to increase. Validation-based early stopping is advantageous because it is easy to implement and can be interpreted directly (Prechelt, 1998). In a scenario where sufficient labeled data are available, the use of a validation set is generally preferred (Goodfellow et al., 2016). However, when  Corresponding author only a few labeled data exist, a tradeoff problem is encountered (Kann et al., 2019; Choi and Lee, 2021). For example, although the usage of a relatively large validation set enables more reliable estimation, the number of samples for training becomes insufficient. Conversely, if small fractions of the samples are assigned to the validation set, the stopping point becomes ambiguous because the small validation set is not representative enough. Early stopping is more important in a low resource setting because the prediction accuracy fluc-tuates highly during training. Such high fluctuations render it challenging when to stop the model. One way to mitigate these fluctuations is to use sufficient training data. In this context, training all the available samples would be more effective, and for this purpose, an appropriate stopping point should be determined without validation split. However, this has not been extensively studied. Duvenaud et al. (2016) and Mahsereci et al. (2017) proposed gradient-based stop-methods and applied statistical inference on the training samples. Lee and Chung (2021) suggested the usage of local intrinsic dimensionality (LID) for early stopping. In addition, some studies treat the stopping epoch as a hyperparameter: the stopping epoch is obtained by grid-search or averaging in cross validation (Choi and Lee, 2021). These methods allow the training of all the labeled samples. However, they do not consider the task-related performance metrics (e.g., accuracy) during training, and the LID and gradient-based stop criteria have not been commonly used in natural language processing (NLP). Furthermore, gradient-based stop-criteria depend on the training samples, the size of which may still be small to be representative. In this study, we propose an early stop ping method b ased on u nlabeled s amples (BUS-stop). We are motivated by the following two considerations: (i) The probabilities of the predicted class label (i.e., the prediction confidences) can serve as 708 Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers , pages 708 718 May 22-27, 2022 c (cid:13) 2022 Association for Computational Linguistics an indicator for over-fitting or under-fitting. (ii) In a better model, the output class distribution is more likely to be closer to the class distribution of the true labels. To incorporate these two assumptions, two stop criteria are proposed, and combined in the BUS-stop method. Our method monitors the prediction results of unlabeled samples during training and utilizes them for determining the stop-criteria. The first proposed stop-criterion is based on confidence similarity (conf-sim). The model stops when the prediction confidences for the unlabeled samples are most similar to the reference confidences, which are precalculated on the labeled set with cross-validation. Conf-sim is observed to reflect the long-term trend of the loss curve, and thereby assist in preventing over-training. The second stop criterion is based on the class distribution similarity (class-sim). This criterion stops the model when the predicted class distribution on the unlabeled set is most similar to the pre-estimated distribution. To this end, we present a novel estimation method for the true class distribution, which calibrates the predicted distribution by extrapolation such that it is closer to the true distribution. Class-sim is observed to reflect the short-term trend of the accuracy. Our method requires several retraining steps to obtain the reference confidences for conf-sim and the estimated class distribution for class-sim. The BUS-stop method that combines class-sim and conf-sim includes the advantages of both, and thereby performs with better accuracy and loss compared to each (class-sim and conf-sim). The following characteristics of our method contribute to performance improvement. Our method does not require a separate validation set; hence, all the labeled samples can be trained. Training can stop at a more generalized model, using a large unlabeled set. The proposed stop-criteria, conf-sim and class-sim, consider two performance metrics, namely, the loss and accuracy. Our contributions are summarized as follows:  We propose BUS-stop, an early stopping method, based on unlabeled samples. BUS-stop can stop the training at a more generalized model, and the performance is better even than using an additional validation set. Furthermore, we present a calibration method to better estimate the class distribution. This method calibrates the output class distribution to render it closer to the true distribution, improving the class-sim performance. Extensive experiments are conducted on five popular text classification datasets in English. Comparison with several stop-methods demonstrates that the proposed method outperforms these existing stop-methods in both balanced and imbalanced data settings. Prechelt (1998) experimented on 14 different validation-based stop criteria. Prechelt (1998) focused on an issue that the validation error during training may represent many local minima prior to a global optimum. Existing non-validation stop-criteria are generally based on statistical inference. Duvenaud et al. (2016) interpreted stochastic gradient descent in terms of the variational inference and proposed an estimation method for the marginal likelihood of the posterior, which was applied as an early stopping criterion. However, this method requires considerable computation for the Hessian, which is not practical in large models. Mahsereci et al. (2017) also proposed a gradient-related stopping method referred to as evidence-based stopping (EB). The EB-criterion is based on the fast-to-compute local statistics of the computed gradients. The criterion represents whether the gradients of the training samples lie within the expected range. Intrinsic dimensionality (ID), which refers to the minimum number of parameters required to represent a dataset, has been used for analyzing the training or redundancy of neural networks (Amsaleg et al., 2015). LID is a version of ID that estimates the subspace dimensions of the local regions. Lee and Chung (2021) found that LID works well as a stopping-criterion in several few-shot image classification datasets. Moreover, LID can be applied to unlabeled samples. Another method involves the pre-estimation of the the number of training epochs by training the model multiple times, such as cross validation (Choi and Lee, 2021); the model can stop at the pre-estimated (PE) stop-epoch when training all the labeled samples. However, these methods have not been commonly studied for NLP tasks and do not consider the performance metrics during training. Furthermore, comparisons among the non-validation stop-methods have not been reported. In this study, we compare our method with the EB, LID, PE, and validation-based stopping methods on five text classification datasets. The method proposed by 709 Algorithm 1 Preliminary stage for BUS-stop Input: Labeled set D l , Unlabeled set D u Output: Sorted output probabilities (cid:126)P l , Calibrated class distribution (cid:126)C u Let Count [1    n l ] = 0 Let P l [1    n l ] = 0 for t  { 1 ,    , T } do Initialize a model, M Split D l into D train and D val at a ratio of r Train the M with ( D train , D val ) M  load the M that was the best on D val for x i  D val do p i  M ( x i ) P l [ i ] = P l [ i ] + p i Count [ i ] = Count [ i ] + 1 end for  C u  M ( D u )  C val , Acc val  M ( D val ) (cid:126) C t u = Calibration (  C u ,  C val , Acc val ) end for for x i  D l do P l [ i ] = P l [ i ] /Count [ i ] end for (cid:126)P l  sort P l in ascending (or descending) order (cid:126)C u = (cid:80) Tt =1 (cid:126)C tu /T return (cid:126)P l , (cid:126)C u Duvenaud et al. (2016) was not compared because it involves considerable computational cost. In this section, we describe the proposed method in detail. The main notations used are as follows: D l = { ( x i , y i ) } n l i =1 and D u = { ( x i ) } n u i =1 denote the labeled and unlabeled sets, respectively. x i and y i are the i -th sample and its true label, respectively, and n l and n u are the numbers of labeled and unlabeled samples, respectively. p ij denotes the prediction probability of the j -th class on the i th sample. Let C be the true class distribution of the samples. The output probability (i.e., confidence) p i associated with the predicted label on sample x i and the predicted (i.e., output) class distribution  C of the samples are defined as follows: p i =max j ( p ij )  C [ j ] = n data (cid:88) i =1 p ij /n data where  j { 1 ,    , n c } ; n c is the number of classes. The pseudocode for the preliminary stage is summarized in Alg. 1. In the preliminary stage, the prediction confidences (cid:126)P l for the labeled samples in D l and the estimated class distribution (cid:126)C u of the unlabeled set D u are calculated. Using D l , the model is reinitialized-and-retrained T -times using a resampling method such as cross-validation. In low-resource settings, such retraining enables more reliable predictions by averaging the results. Each sample in P l is evaluated when the validation loss is the lowest. Each sample should be validated at least once; the prediction confidences are averaged for each sample. P l (and P u in Alg.2 as well) is sorted in order of size for confidence comparison between two different sample sets, D l and D u , in the main stage; we denoted it as (cid:126)P l ( (cid:126)P u for P u ). When retraining T -times, the output class distributions of the unlabeled set D u are obtained and calibrated (this calibration is defined in Section 3.3). Then, the T calibrated class distributions are averaged, resulting in (cid:126)C u . After this stage, (cid:126)P l and (cid:126)C u are used to calculate the similarities for the two stop criteria, conf-sim and class-sim, respectively. After the preliminary stage, we train all the labeled samples and refer to this stage as the main stage. The combined BUS-stop method applied in the main stage is summarized in Alg. 2. The unlabeled set is predicted at every epoch during training. Conf-sim The first proposed stop criterion conf-sim S conf represents the similarity of the prediction confidences (cid:126)P u for the unlabeled samples with the reference confidences (cid:126)P l . To calculate the similarity between (cid:126)P u and (cid:126)P l , their dimensions must be the same. We sample (cid:126)P u at regular intervals n u n l such that it is the same size as (cid:126) P l and denoted it as ... P u . We use the Euclidean distance to calculate the similarity, resulting in S conf . Then, the first stop criterion is when S conf has the lowest value, i.e., ... P u is most similar to (cid:126)P l . There is a natural concern that ... P u is likely to produce higher (thus dissimilar) confidences than (cid:126)P l because ... P u is obtained by training all the labeled samples, unlike (cid:126)P l . However, the fact that the confidence for each sample in (cid:126)P l is obtained when the validation error is the lowest can alleviate this concern. Thereby, S conf can be a rough criterion for avoiding underand over-fitting, and can reflect the trend of the loss, based on comparison with the reference confidences. Class-sim The second proposed stop criterion is class-sim, S class . The predicted class distribution  C u on the unlabeled set is compared with the estimated class distribution (cid:126)C u from the preliminary stage. The assumption is that a well-trained model can also predict the class distribution more accurately. Therefore, estimation of the true class distribution is crucial. A calibration method that facilitates better estimation of the class distribution is presented in Section 3.3. We use the cosine similarity to calculate the similarity between  C u and (cid:126)C u , and obtain S class . The second stop criterion is when S class has the highest value, i.e.,  C u is most similar to (cid:126)C u . Thereby, S class can reflect the short-term trend of the accuracy because it is more likely that the outputs of a higher accuracy model are closer to the true class distribution. BUS-stop Finally, we combine the two stop-criteria, conf-sim and class-sim, to form the BUS-stop method, as depicted in Alg. 2. A simple   = (,)  = (0.5,0.5)   = 1.0   = 0.8   = 0.5   = (0.65,0.35)       By Equation (1) and (2),   :     :      0.5+ 5 3 0.650.5 = 0.75    0.5+ 5 3 0.350.5 = 0.25   Figure 1: Calibration example in binary classification. product of the two stop criteria can be an ineffective stop criterion because the sizes of S conf and S class are relative. Our combined stop-criterion is to save the model with the highest S class among of the epochs from the lowest S conf to the subsequent ( n que  1) -th epoch. This technique enables fine-stopping by considering both S conf and S class , which reflect the long-term and short-term performances, respectively. It is to be noted that early stopping methods should be operated as an ongoing process, and not as a type of post-hoc method. To this end, we use a fixed-size queue Queue , and its size n que as a hyperparameter, as shown in Alg. 2. 3.3 Calibration of Class Distribution In this section, we describe the calibration of the predicted class distribution. The calibration method aims to better estimate the true class distribution of the unlabeled set, thereby improving the performance of class-sim, particularly for imbalanced classification. Trained neural networks often involve sampling biases. For example, in binary classification, the prediction results of a model trained with a class ratio a : b tend to follow the distribution of a : b . Thus, when the class distributions are different in the test and training sets, the model performance can deteriorate. Let us suppose the following somewhat ideal and naive situations. Let C u be the true class distribution of the unlabeled set. If the model is perfectly trained with an accuracy of 1.0, the output class distribution will be equal to C u . On the other hand, if the model fails to learn any inference knowledge from training, the model will output the predictions only by its sampling bias; i.e., when the accuracy is the same as the random expectation (de-noted as Acc min , e.g., 0.5 in binary classification), the output class distribution will be equal to the sampling bias B . Thus, the model accuracy can reflect whether the output class distribution is closer to the sampling bias or the true distribution. In the preliminary stage, we obtained the models' proxy accuracy and output class distribution as Acc val 711 Data Class Train Test Len SST-2 2 6.9K 1.8K 19 IMDB 2 25K 25K 231 Elec 2 25K 25K 107 AG-news 4 120K 7.6K 38 DBpedia 14 560K 70K 49 Table 1: Statistics for datasets. and  C u , respectively. Assuming that there is an approximate linear relationship, we can define a proportional expression as follows: (1  Acc min ) : ( Acc val  Acc min )  ( C u  B ) : (  C u  B ) (1) We rearrange the above expression in terms of C u : C u  B + (1  Acc min ) ( Acc val  Acc min ) (  C u  B ) (2) Then, we denote the approximation of C u as (cid:126)C u . Considering the class distribution as a vector, Eq. (2) is a type of extrapolation. B can be defined as the class distribution of D train or the predicted distribution in the validation set,  C val , of the preliminary stage. In addition, the Acc can be replaced with F1-score. Fig. 1 illustrates an example of our calibration method. We conducted extensive experiments using five text classification datasets. The statistics are summarized in Table 1. These datasets have been extensively used in NLP research, and are publicly available. The SST-2 (Socher et al., 2013), IMDB (Maas et al., 2011), and Elec (McAuley and Leskovec, 2013) datasets are used for sentiment analysis. SST-2 and IMDB include movie reviews, and Elec includes reviews on Amazon electronics. AG-news (Zhang et al., 2015) and DBpedia (Zhang et al., 2015) are topic classification tasks for Wikipedia and news articles, respectively. For each dataset, we sampled K labeled samples per class from the training set. K was set to 50 for low-resource settings; we also experimented by varying K  { 50 , 100 , 200 , 400 , 800 , 1600 } . We used the test samples as the unlabeled set for each dataset, which is referred to as transductive setting in few-shot classification (Liu et al., 2019). In this section, we describe the various stop-criteria for comparison with our method. EB The EB (Mahsereci et al., 2017) is a criterion based on gradients of training samples. The EB-criterion stops when the following condition is met: 1  |S| DD (cid:88) k =1 [(  LS ,k ) 2   k ] > 0 (3) where S represents a sample set, D is the number of parameters,  L is the gradients of loss, and subscript k indicates the k -th weight of the total parameters. is the variance estimator, which is calculated as follows:   k = 1 ( |S|  1) (cid:88) x S (  l k ( x )   LS ,k ) 2 (4) where  l ( x ) is the loss gradient on sample x . Note that LS = 1 |S| (cid:80) x S l ( x ) . For further details, refer Mahsereci et al. (2017). LID Lee and Chung (2021) approximated LID as follows: LID =  (cid:88) x  D u (cid:20) 1 m m (cid:88) i =1 ln d i ( (cid:126)z ( x )) d m ( (cid:126)z ( x )) (cid:21)  1 (5) where (cid:126)z ( x ) is the representation vector of sample x , and d i is the Euclidean distance of (cid:126)z ( x ) and its i -th nearest neighbor. m is a hyperparameter, which denotes the number of nearest neighbors. The lowest LID is the stop criterion. Val-stop split ( x ) and Val-stop add ( x ) Val-stop denotes validation-based stopping. Val-stop split ( x ) indicates that x validation samples per class are taken from the labeled set. Therefore, K  x samples are trained and x samples are validated for each class. Val-stop add ( x ) indicates that x additional samples per class are used for validation; i.e., Val-stop add ( x ) uses a total of K + x labeled samples per class. Val-stop add ( x ) has an unfair advantage because it uses additional labeled samples. PE-stop-epoch The stopping epoch is considered a hyperparameter, which is p ree stimated with cross-validation, as described in Section 2. We use four-fold cross-validation. Conf-sim and class-sim can also be used as a single stop-criterion, as mentioned before. We compare the single criteria with the combined BUS-stop criterion. Conf-sim stops when S conf is the lowest, and class-sim stops when S class is the highest. BERT-base (Devlin et al., 2019) was adopted as our text encoder. The Adam optimizer (Kingma and Ba, 2015) was applied for categorical cross-entropy loss (i.e.,  (cid:80) y i log p i ), and its learning rate was set to 3 e 5 . The dropout (Srivastava et al., 2014) was set to 0 . 2 , and the batch size was 16 . All the stop-criteria were evaluated simultaneously for each run to reduce the variance of the estimation. We averaged 10 results in all the experiments. In EB, 64 random training samples were used for S in Eq. (3). In LID, the final vector of the [CLS] token in the BERT model was assigned to (cid:126)z ( x ) in Eq. (5), and the best m was selected from { 5 , 10 , 20 , 50 , 100 } . In BUS-stop, n que in Alg. 2 was set to five. Note that K is the number of training samples per class. When K was set to 50 , T and r in the preliminary stage (see Alg. 1) were set to 5 and 1 : 1 , respectively. When K was set above 50 , T and r were set to 4 and 3 : 1 , respectively. In our calibration method, we used  C val as B and macro F1-score as the Acc val . Table 2 shows the results when K = 50 for training. It is noted that the original test sets have a balanced class distribution. We also report the loss measure as well as accuracy because loss can imply over-training. As shown in Table 2, our BUS-stop method exhibits the best performance on an average, and the accuracy is better even than Val-stop add (25) , which uses a larger numbers of labeled samples. Note that Val-stop add (25) uses a total of 75 labeled samples per class. The performance of Val-stop split (25) indicates that splitting data for validation can result in poor performance in low-resource settings. LID underperforms compared to the PE-stop-epoch that does not require unlabeled samples. Conf-sim shows the second-best loss on an average. Class-sim underperforms as a stop criterion by itself. However, the BUS-stop method, which combines these two methods, shows better performance than each one on an average. Figure 2 displays the results of conf-sim and class-sim over the epochs. More examples are presented in Appendix A. In Fig. 2, the conf-sim curve is similar to the long-term trend of the loss; however, it does 713 Dataset SST-2 IMDB Elec Average Method Acc F1 Loss Acc F1 Loss Acc F1 Loss Acc F1 Loss Val-stop split (25) 0.788 0.719 0.499 0.732 0.674 0.589 0.783 0.724 0.507 0.768 0.706 0.532 EB 0.846 (cid:39) 0.786 (cid:39) 0.504 0.810 0.749 0.568 0.839 0.789 0.541 0.832 0.775 0.537 LID 0.750 0.698 0.632 0.712 0.668 0.678 0.780 0.728 0.574 0.747 0.698 0.628 PE-stop-epoch 0.843 0.779 0.527 0.821 0.763 0.589 0.843 0.789 0.521 0.836 0.777 0.545 Conf-sim (ours) 0.816 0.754 0.427 0.813 0.750 0.432 (cid:39) 0.835 0.775 0.398 0.821 0.760 0.419 Class-sim (ours) 0.862 (cid:39) 0.797 (cid:39) 0.489 0.844 (cid:39) 0.779 (cid:39) 0.510 0.873 (cid:39) 0.807 (cid:39) 0.409 0.860 0.794 0.469 BUS-stop (ours) 0.860 0.792 0.379 0.849 0.787 0.406 0.876 0.815 0.343 0.861 0.798 0.376 Val-stop add (25) 0.823 0.767 0.412 0.820 0.767 0.457 0.837 0.784 0.407 0.827 0.773 0.426 Table 4: Performance comparison in an imbalanced setting of binary classification tasks. not accurately reflect the short-term fluctuation of the performance from epochs 716. On the other hand, class-sim is observed to be well responsive to the short-term fluctuation of the accuracy, but does not reflect the long-term trend. BUS-stop, which is a combination of these two methods, takes advantage of the shortas well as long-term methods, and thereby facilitates fine stopping. The EB-criterion shows the statistically similar accuracy to the BUS-stop method in most datasets. In the EB-criterion and PE-stop-epoch, the average loss is not good enough compared to the high accuracy. The accuracy and loss show somewhat conflicting results. That was due to over-confidence on the mis-classified samples, caused by over-training. Note that Loss =  (cid:80) y i log p i . Overconfidence on the wrong label makes p i close to zero on its true label y i . Thus, excessively low p i can increase the loss drastically. Table 3 lists the over-confidence error (OE); the equation for OE is presented in Thulasi-dasan et al. (2019). This confidence error can be detrimental in various applications, as described by Guo et al. (2017). We experimented with an imbalanced setting in binary classification tasks. For testing, we sampled 1,000 instances in the SST-2 test set, and 10,000 instances each in the IMDB and Elec test sets, with a class distribution of 2 : 8 (negative:positive). The macro F1-score is also reported. Table 4 shows the results when K was set to 50 for training. In most cases, BUS-stop exhibits the best performance with respect to the accuracy as well as loss. In addition, it is noted that BUS-stop outperforms the other methods with a greater margin in an imbalanced setting than in a balanced one (Table 2). It is observed that ratios marked with  (cid:39) ' are fewer in the imbalanced setting. Class-sim shows the best or second-best accuracy among the datasets. It is observed that the output class distribution can be an important indicator for a better model. Table 5 shows the results in various imbalanced settings of the SST-2 (both the training and test sets are imbalanced). The number of training samples was fixed to 100 for the different class-distribution settings. In general, when the class distributions of the training and test sets are similar, the results shows better performance for all the three methods, EB, BUS-stop, and Val-stop add (25) . In most cases, BUS-stop consistently outperforms Val-stop add (25) and EB, and the margin is greater when the class distributions are more different between the training and test sets. This result indicates that BUS-stop is robust to imbalanced classification. Impact of the training size Figure 3 indicates the accuracy curve with respect to the training size, using the IMDB dataset. The x values of Val-stop add ( x ) and Val-stop split ( x ) were set to 25, 25, 50, 100, 200, and 400, according to the increase 714 Figure 3: Accuracy by different training sizes in IMDB . in K . It can be observed that the performance of BUS-stop is good in the sufficient-data regime as well. However, the performances of the three stop-criteria converge almost similarly with the increase in the training size. The impact of splitting the samples for validation does not deteriorate the performance when K is greater than 400. Rather, Val-stop split ( x ) performs slightly better when K is 1600. This result suggests that when sufficient labeled data are available, validation-based stopping can be a better choice. Calibration performance In the BUS-stop method, accurate estimation of the class distribution plays a crucial role. The cosine similarity between the class distribution of the test set and the estimated distributions by various estimators are shown in Table 6, where the uncalibrated output distributions (  C u ) and the estimated distributions by the calibration methods, based on the Acc-score (Cali Acc ) and macro F1-score (Cali F 1 ), were compared. When the class distributions are similar between the test and training sets, the performance of  C u is slightly better than those of the other es-Figure 4: BUS-stop accuracy for different class distribution estimators in the 16 imbalanced settings depicted in Table 6. Method Time complexity Measured time SST-2 ( n u = 1 . 8 k) DBpedia( n u = 70 k) EB g ( n l ) +  0.32 m 0.49 m LID g ( n l ) + p ( n u ) 0.12 m 5.02 m PE-stop-epoch ( T + 1)  g ( n l ) 0.43 m 1.14 m BUS-stop ( T + 1)  g ( n l ) + p ( n u ) 0.47 m 5.97 m Val-stop add (25) g ( n l ) 0.07 m 0.19 m Table 7: Running time comparison for different stop-criteria. timators. However, the estimation by calibration based on the F1-score (Cali F 1 ) is better on an average, and particularly when the class distributions of the test and training sets are different. Figure 4 indicates the BUS-stop accuracies when each model stops based on the estimated class distribution in Table 6 (the same color corresponds to one cell in Table 6). For example, the yellow colors correspond to the settings in which the class distribution is 2:8 and 8:2 in the training and test sets, respectively. As shown in Fig. 4, the better the class distribution is estimated, the higher is the accuracy of BUS-stop. Such high correlation indicates the importance of the class distribution estimator. This result is consistent with our assumption that the output class distribution of better models will be closer to the true distribution. Running time The running times are not directly comparable owing to the different hyperpa-rameter settings for each method. For example, the BUS-stop and PE-stop-epoch require a separate preliminary stage that consumes additional time. We add up both the times taken in the preliminary stage and main stage. We denote the average running time per epoch as g ( n l ) for training the labeled samples and p ( n u ) for predicting the unlabeled samples. The time complexity and the measured time are shown in Table 7. Note that T is 715 Method Val-stop split (25) Val-stop add (25) BUS-stop Selection local global local global local Balanced classification SST-2 0.775 0.785 0.819 0.840 0.831 IMDB 0.746 0.786 0.824 0.838 0.828 Elec 0.781 0.805 0.842 0.852 0.848 AG-news 0.846 0.857 0.867 0.871 0.865 Imbalanced classification SST-2 0.788 0.807 0.823 0.832 0.860 IMDB 0.732 0.757 0.820 0.834 0.849 Elec 0.783 0.820 0.837 0.853 0.876 Table 8: Accuracy by global selection in Val-stop. the number of retrainings in the preliminary stage, which was set to five. The experimental settings are the same as in Section 5.1. The time measurement was conducted on a PC with an Intel Core i7 CPU, 64-GB RAM and an NVIDIA Titan X Pascal GPU. As shown in the expression of time complexity, the running time depends on the numbers of labeled and unlabeled samples, n l and n u , respectively. In DBpedia, which has a large number of unlabeled samples, n u , the LID and BUS-stop methods take the two longest running times. On the other hand, in SST-2, the PE-stop-epoch and BUS-stop methods show the two longest running times, because the n u is relatively small such that the g ( n l ) is more dominant than the p ( n u ) . The BUS-stop requires a longer running time than other methods due to the T -times retraining and the continual prediction on the unlabeled set. To reduce the time, we can adjust the T value or sample a smaller amount of data from the unlabeled set. Limitations The proposed BUS-stop method was designed for classification tasks, and thereby can be applied when the model can output confidences. Regression tasks as well can be addressed by converting into classification problems. The continuous values normalized between 0-1 can be represented as confidences in a binary classification. However, it may be difficult to apply to other more complex tasks (e.g., text summarization). This study is limited to classification tasks. Another limitation is that the BUS-stop, which is a nonvalidation stop-method, cannot make direct comparisons between two models with different runs. Early stopping can be seen as selecting the best resulting model over the epochs. In a similar way, it is also possible to select the best model among multiple runs. We refer to the former as local selection and the latter as global selection. In validation-based stopping, the global selection is simply to select the model with the lowest validation loss over multiple runs. However, the non-validation methods have no clear criterion for this purpose. We repeated training five runs for each and selected the best model among the runs based on validation loss. Other experimental settings are the same as in Section 5. As shown in Table 8, the global selection in validation-based stopping improves performance across the datasets in both balanced and imbalanced settings. However, in the imbalanced setting, the BUS-stop still results in better performance. Note that Val-stop add (25) uses additional labeled samples. We also report that the global selections that are based on the S conf , S class , and LID did not show significant performance improvement in our experiment. The development of non-validation global selection methods is left for future work. Validation-based early stopping can be detrimental in low-resource settings because the reduction in the number of samples by validation split may result in insufficient samples for training. In this study, we proposed an early stopping method called BUS-stop, based on unlabeled samples. Moreover, we proposed a calibration method to better estimate the true class distribution, which was used in the BUS-stop method to improve the performance. We conducted experiments on five popular text classification datasets. The results indicated that BUS-stop outperformed the existing stop-criteria in both balanced and imbalanced settings. In particular, BUS-stop showed robustness to imbalanced classification. The proposed BUS-stop method enables the training of all the available samples and presents a better stopping point using large unlabeled samples. In future, we plan to better exploit the unlabeled samples in self-training schemes. We can also combine BUS-stop and self-training methods. BUS-stop can be used to improve the performance of the initial model, which plays an important role in the final self-training performance. Additionally, we consider applying the BUS-stop to domain adaptation tasks in the future. This research was supported by the Bio-Synergy Research Project (NRF-2016M3A9C4939665) of the Ministry of Science and ICT through the National Research Foundation of Korea (NRF) and the NRF grant funded by the Korean government (Ministry of Science and ICT) (NRF-716 2018M3C7A1054932), and partly supported by Institute for Information and communications Technology Promotion (IITP) grant funded by the Korea government (MSIP) [No. 2019-0-01842, Artificial Intelligence Graduate School Program (GIST)].
abstain abstain objective abstain result abstain abstain objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain abstain result abstain abstain abstain abstain abstain objective abstain result abstain objective method abstain abstain objective objective method objective objective objective other other other other other other other other other other other other other other other method other method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain method method abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain result abstain abstain objective objective method abstain abstain abstain abstain method abstain method other other