javier.turek@intel.com huth@cs.utexas.edu Abstract Language model fine-tuning is essential for modern natural language processing, but is computationally expensive and time-consuming. Further, the effectiveness of finetuning is limited by the inclusion of training examples that negatively affect performance. Here we present a general fine-tuning method that we call information gain filtration for improving the overall training efficiency and final performance of language model fine-tuning. We define the information gain of an example as the improvement on a validation metric after training on that example. A secondary learner is then trained to approximate this quantity. During fine-tuning, this learner selects informative examples and skips uninformative ones. We show that our method has consistent improvement across datasets, finetuning tasks, and language model architectures. For example, we achieve a median perplexity of 54.0 on a books dataset compared to 57.3 for standard fine-tuning. We present statistical evidence that offers insight into the improvements of our method over standard finetuning. The generality of our method leads us to propose a new paradigm for language model fine-tuning  we encourage researchers to release pretrained secondary learners on common corpora to promote efficient and effective fine-tuning, thereby improving the performance and reducing the overall energy footprint of language model fine-tuning. Language modeling is the task of generating language from context. This is often framed as an autoregressive task, where a model predicts the conditional probability of the next word based on the sequence of previously observed or generated tokens. Language modeling has seen a recent surge in relevance thanks to its success as a pretraining objective for self-supervised representation learning. The most prominent language models today are Transformer-based models (Vaswani et al., 2017) such as BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019). Language models are most commonly trained with backpropagation using traditional NLP loss functions such as cross entropy. These loss functions are designed so that the models are rewarded for assigning high probability to text that appears commonly in the training corpus. The energy and computational costs of training a state-of-the-art language model from scratch are very high, to the point of impracticality for most researchers. One recent estimate suggests that training a single state-of-the-art model with architecture search takes more energy than five cars will use in their entire lifetimes (Strubell et al., 2019). In practice, this cost is sidestepped by pretraining, where a language model is trained once and then released publicly. This language model can then be updated for use in other tasks through fine-tuning. For example, a generic language model can be fine-tuned to generate text that matches the style and syntax of any new corpus (Howard and Ruder, 2018). While better than training from scratch, the cost of fine-tuning such large networks is still relatively high. Fine-tuning to convergence for a single task can easily take in excess of a day on multiple energy-intensive GPUs (Strubell et al., 2019). Recent work analyzing the fine-tuning process has shown that it has high variability between runs and is particularly sensitive to data ordering (Dodge et al., 2020). Those authors propose to overcome this variability by training models using many random seeds and then only keeping the best, effectively trading computational efficiency for model performance. While this improves performance, the reasons for the high variability between random seeds have yet to be explored. We hypothesize that much of this variability can be explained by the random selection of highly informative training examples, which most effectively capture low-level distributional statistics of the target corpus. If this is the case, then it should be possible to quickly screen for informative training examples, ensuring high performance at reduced cost. In this paper, we suggest replacing the retrospective approach of testing many random seeds (Dodge et al., 2020) with a prospective approach to improving the effectiveness of language model fine-tuning. Our approach uses a secondary learner to estimate the usefulness of each training example, and then selects only informative examples for training. We show that this technique works well and is applicable in a variety of finetuning settings. We examine why it works well, and present evidence that supports our hypothesis about informative examples explaining data ordering effects. In addition to performance gains, this method may mitigate the energy impact of deep neural network language modeling, as we require fewer backpropagation steps than other techniques that trade computational power for fine-tuning performance. Several methods have recently been proposed to improve language model fine-tuning performance. Lee et al. (2020) proposed a technique based on neural network dropout (Srivastava et al., 2014) for regularizing finetuned language models that involved stochastically mixing the parameters of multiple language models for the same domain, and further demonstrated the usefulness of pretrained weight decay over conventional weight decay for improving language model fine-tuning performance. Phang et al. (2018) showed that adding supplementary training to pretrained language models using supervised tasks yielded state of the art results for BERT (Devlin et al., 2019). Moore and Lewis (2010) proposed a related technique for increasing the amount of language model training data from out-of-domain data sources that relies on filtering out high cross-entropy contexts as measured by an in-domain language model. Tenney et al. (2019) and Liu et al. (2019) have both suggested that language model finetuning is better in Transformer-based models when starting when using features from intermediate layers as opposed to later layers. The instability of language model fine-tuning has previously been investigated by others. Mos-bach et al. (2021) suggested that this instability is caused by a combination of insufficiently general training sets and optimization challenges. Zhang et al. (2021) investigated how similar factors, such as non-standard optimization techniques and overreliance on a standard number of training iterations hurts the performance of fine-tuned language models. Dodge et al. (2020), whose work we replicate and build on here, showed that language model finetuning is sufficiently stochastic so that even random seed searches are a suitable technique for improving their overall performance. A language model L is a function with parameters  , which, when given an ordered sequence of tokens X = { x 1 , . . . , x n } as input, outputs a probability distribution over the next token y : Given a test set T of (sequence, next token) pairs, T = { ( X 1 , y 1 ) , . . . , ( X n , y n ) } , the perplexity ( T ;  ) of the language model L ( X ;  ) over the set T is defined as: ( T ;  ) = 2  (cid:80) ( Xi,yi ) T  p ( y i )  log 2 L ( X i ;  ) , where  p ( y i ) denotes the one-hot probability distribution that assigns all of its probability mass to the token y i . Autoregressive language models such as GPT-2 (Radford et al., 2019) are trained to minimize perplexity using backpropagation on very large training corpora. In practice, pre-trained language models are often fine-tuned using a new corpus or transferred to a new task (Howard and Ruder, 2018). Formally, let F = { ( X i , y i ) } i be a target set. Fine-tuning on the set F tries to minimize the expected value of the loss function  :   = arg min  E (log 2 ( F ;  )) . The initial parameterization   0 of the language model is defined by its pre-trained parameters   0 =  . The fine-tuning problem in Eq. (1) is then solved by applying stochastic gradient descent (SGD) on samples from F . Namely, for a given batch B of samples from F , the language model parameters are updated by   k    k  1    ( B ;   k  1 ) , where  is the step size. We refer to methods that randomly sample contexts to update pretrained model parameters as standard finetuning . While random sampling methods are useful (Bottou, 1991), the stochasticity of context sampling suggests an avenue for additional improvement. Such methods make no assumption on the informativeness of the examples in F , instead relying on randomness to find useful training samples. It is worth asking ourselves: can we efficiently measure the informativeness of an example? And if so, can we exploit that measurement for additional fine-tuning improvement? Next, we characterize the informativeness of an example ( X, y )  F , given a pre-trained language model L ( X ;  ) and a target dataset F . We define an example ( X, y ) as informative if our estimate of the improvement that it will grant to the model exceeds a chosen threshold. Namely, if we expect that a given example will reduce model perplexity by more than a preset amount, then we will denote it as informative. We define the information gain (IG) of a example ( X, y ) over an objective set O as the difference in perplexity measured on the objective set O before and after training on the example ( X, y ) , IGO ( X, y ) = ( O ;  (cid:48) ( X, y ))  ( O ;  ) , (2) where  is the initial parameterization of the language model and  (cid:48) ( X, y ) is the parameterization after backpropagating the loss associated with training example ( X, y ) . The objective set O = { ( X 1 , y 1 ) , . . . , ( X n , y n ) } is a held-out subset of training data that informs our decision about which contexts are informative. In practice, the objective set could be a subset of the fine-tuning set F . For brevity, we denote IGO ( X, y ) as simply IG ( X ) since there exists an implicit direct bijection between all X 's and y 's and the objective set is implied. Since information gain evaluates the informativeness of an example, we next propose a method that exploits it for fine-tuning. Let us assume that the method encounters a new example ( X, y ) . Then, the method has a choice between two actions:  BACKPROP : update the language model parameters  by backpropagating the loss ( { ( X, y ) } ;  ) , taking the gradient descent step, and updating parameters from  to  (cid:48) . SKIP : leave the language model parameters unchanged. With this idea in mind we define the function 1 q ( X, action ) and assign a value to each of the actions above: q ( X, BACKPROP ) = IG ( X ) (3) q ( X, SKIP ) = TSKIP , (4) where TSKIP is a free threshold parameter for deciding which IG ( X ) values are sufficiently high to warrant backpropagation. Following this definition, we can apply a greedy policy for filtering examples during fine-tuning:  ( X ) = argmax a { BACKPROP , SKIP } q ( X, a ) . By filtering examples in this way, we aim to reduce the effect of variability in data order observed in previous work (Dodge et al., 2020), and improve the generalizability of our training set (Mosbach et al., 2021). By doing this, we expect to improve the performance of our language model. We call this technique Information Gain Filtration or simply IGF . Thus far, we have described a general method to segregate informative from non-informative examples, deferring the issue of computational cost. Computing IG ( X ) in Equation (2) entails a backpropagation step, making direct application of q ( X, action ) at least as expensive as standard fine-tuning. To address this issue, we aim to approximate the information gain IG ( X ) using a separate model that we will call the secondary learner and denote with  Q ( X ) . To train this secondary learner, we first construct a training dataset D by measuring IG ( X ) for a random subset of examples drawn from the fine-tuning set F . The objective set O used to compute IG ( X ) is selected as a different subset of F . Each entry in D consists of a pair of 1 Due to its intuitive similarity with notions in reinforcement learning (Mnih et al., 2013) of using a network to approximate the expected value of a given action, we abbreviate this normalized informativeness metric as a  Q -value (Watkins and Dayan, 1992) the input text X and its associated IG ( X ) value, i.e., D = { ( X 1 , IG ( X 1 )) , . . . , ( X n , IG ( X n )) } . We then train the secondary learner  Q to approximate a normalized IG ( X ) given X . We normalize IG ( X ) so that TSKIP can be interpreted as a standardized threshold on the selectivity of the filtration. Finally, the resulting secondary learner  Q is used to filter examples during fine-tuning. Algorithm 1 summarizes IGF with a secondary learner for language model fine-tuning. Algorithm 1 Information Gain Filtration Input: Fine-tuning ( F ) and objective ( O ) dataset of contexts, ( X , O ) := { ( X 1 , y 1 ) , ..., ( X n , y n ) } , parameterization of initial pretrained LM,  , and initial secondary learner model  Q . Output:  , new parameterization for the LM 1: Initialize D , B = {} . 2: for i = 0 . . . s do 3: Sample context ( X i , y i ) from X . 4: Append ( X i , IGO ( X i , y i )) to D . 5: Normalize IGO ( X, y ) values in D to N (0 , 1) . 6: Train secondary learner  Q , using dataset D . 7: for i = 0 . . . number of batches  1 do 8: while |B| < batch size do 9: Sample context C = ( X, y ) from X . 10: if  Q ( C )  TSKIP then 11: Add C to batch B . 12: Backpropagate over batch B , updating  13: Reset batch B = {} . 14: Return  . The secondary learner training set D is constructed using the initial pretrained model parameters  0 . This means that the effectiveness of the learner at distinguishing high quality from low quality examples should degrade as the parameters diverge from their initial values. To ameliorate this problem, Equation (4) can be modified by changing TSKIP during the fine-tuning process. Since  Q is most accurate at the first step, we scheduled TSKIP to switch from highly selective (a high value) to highly permissive (a low value). This allows the model to take advantage of the accurate predictions for IG ( X ) early in the fine-tuning process without overfitting once those predictions become less accurate later on. Here we first provide an empirical analysis suggesting that IGF outperforms standard fine-tuning across different choices of datasets, fine-tuning tasks, and neural architectures. We follow this analysis with an examination of why IGF works, and an exploration into the statistical properties of standard fine-tuning and of IGF. We tested these results on a standard Books dataset (Zhu et al., 2015), a mixed dataset which is composed of training examples from two corpora (the Books corpus and a corpus of scraped Reddit comments (Huth et al., 2016)), and the WikiText-103 dataset (Merity et al., 2017). The Books corpus allows us to fairly compare standard fine-tuning against IGF, whereas the Mixed corpus allows us to analyze the effectiveness of the method at separating informative contexts from uninformative ones. In practice, our secondary learner,  Q , represents the input text X by embedding it with 768-dimensional byte-pair embeddings (Gage, 1994). We then pass the input representations through a convolution with kernel width 3, followed by max-pooling operation over the time axis and a 2-layer feedforward network. This architecture was re-fined through coordinate descent, and evaluated on a separate held-out set of measured IG ( X ) values. The choice of architecture does not strongly affect method performance (see Appendix A, Figure 11). Additionally, a neural network is not necessary for the learner, as simpler learning methods are sufficient (see Figure 5). We first compare IGF directly to standard finetuning, which we define as basic batched stochastic gradient descent with Adam (Kingma and Ba, 2015) using random samples from the target corpus. For initial tests, we chose the pretrained GPT-2 Small Transformer model, a commonly used unidirectional language model with roughly 124 million parameters. We used the publicly available GPT-2 Small implementation of the transformers package (Wolf et al., 2020). We performed 50 runs each of standard fine-tuning on (1) training examples sampled from the Mixed corpus, and (2) from the easier Books corpus. We then performed 50 runs of IGF using two thresholding schedules, one with a fixed TSKIP and one with shifting TSKIP . For both methods, batches of size 16 were used to train the language model with 0 10 20 30 40 50 60 Batches 60 70 80 90 100 110 120 T e s t p e r p l e x i t y Standard Mixed Standard Books IGF with Constant Threshold IGF with Shifting Threshold 53 54 55 56 57 58 59 60 61 62 63 T e s t p e r p l e x i t y a t 60 Figure 1: Comparing IGF to Standard Fine-tuning: IGF with constant ( p < 10  3 , t -test) and shifting ( p < 10  6 , t -test) thresholding significantly outperform standard fine-tuning. a learning rate of 5  10  5 and  1 = 0 . 9 ,  2 = 0 . 999 . The convolutional network that we used for our secondary learner was trained using SGD with Adam with a learning rate of 10  5 and  1 = 0 . 9 ,  2 = 0 . 999 . Both types of IGF runs were performed on the strictly more challenging Mixed corpus only. In all cases model perplexity was tested on a set drawn solely from the Books corpus. Figure 1 plots the averaged fine-tuning curves of these 4 different approaches over 60 batches. We see that IGF significantly improves final test perplexity when compared to standard fine-tuning on both the Mixed corpus and the Books corpus. Standard fine-tuning on Books achieves a median perplexity of 57.3, compared to 56.9 for IGF with a constant threshold and 54.0 for IGF with the shifting threshold schedule. 2 All 50 runs of IGF with a shifting schedule outperformed all 50 standard fine-tuning runs. This means that the overall improvements to data order that IGF achieves through selective sampling of informative contexts are far in excess of what might be reasonably achieved through random sampling of contexts. Next, we show that the improvements offered 2 Demo code and data can be found at https://github.com/HuthLab/IGF . by IGF persist across several choices of dataset, fine-tuning specifications, and model architecture. Figure 2 shows the final converged values for fine-tuning GPT-2 Small on a different dataset from Figure 1 (WikiText-103), a different architecture (GPT2-Medium), a different embedding space with different directionality (BERT) (De-vlin et al., 2019), and a different overall finetuning task (SST-2) (Socher et al., 2013). In every case, IGF exceeds the performance of standard fine-tuning. This suggests that IGF is a resilient method that is broadly applicable to a variety of fine-tuning modalities and domains. It is clear that IGF is successful as a general method for improving fine-tuning performance, however why this is the case remains unexamined. Here, we present an analysis of the statistical properties of fine-tuning that illuminates why IGF is able to improve over standard fine-tuning. A main assumption of IGF is that it is possible to approximate IG ( X ) . If IG ( X ) is not approximable, then the secondary learner could not effectively filter out uninformative contexts and therefore would be useless. In order to support this assumption, we will first show that a given example is worth learning from even if it only possesses the correct low-level features of informative contexts, such as the correct unigram frequency distribution. We performed an experiment in which we fine-tuned a language model on either (1) real example sequences from a corpus, (2) artificial sequences that were constructed by independently sampling each token from the frequency distribution of the corpus, and (3) sequences constructed by uniformly sampling tokens from the set of all possible tokens. We then measured the change in loss on a separate portion of the corpus. Figure 4 shows the results of this experiment. The average reduction in loss for examples constructed using the unigram frequency distribution is significantly better than random and roughly 70% as good as using real examples from the corpus. Thus, a significant fraction of the benefit of training on real contexts can be estimated by merely knowing the unigram frequency distribution from which those contexts were derived, which is easily estimable without knowing the particular parameterization of the language model itself. Therefore, it makes sense that IGF can inexpensively estimate whether a given context generalizes well to the target corpus. The secondary learner only bases its estimates on the update to loss after the first backpropoga-tion step. We might question whether early improvement translates to long-term improvement over the course of fine-tuning. If it did not, then the estimates that the secondary learner produces would eventually disappear as fine-tuning continued. Dodge et al. (2020) observed that the quality of a fine-tuning run could usually be established by looking at the trajectory of the loss curve very early during training. In order to explain why these early estimates are sufficient for sample filtration, we attempted to determine whether training on good contexts early is an important element of the variability in data order between fine-tuning runs. Figure 3 compares test perplexity after training from a randomly sampled first batch against the test perplexity after many randomly sampled batches. Good early batches improve the proba-5 0 Predicted 6 4 2 0 2 A c t u a l r = 0.784 Conv. bility of converging to an ideal final value. The correlation between the test perplexity after a single batch and the test perplexity after 50 batches, which is near convergence for most runs, is statistically significant ( r = 0 . 28 ). While this value appears somewhat low, it is significant and therefore can be exploited for improvements in performance. Taken together, the pair of observations that (1) early data quality is important, and (2) that the quality of a context can be summarized by its low-level statistics serves to motivate our understanding of why IGF is effective. Specifically, if we can carefully ensure that early batches are good, as IGF does, then we will likely end up with a su-perior model after convergence. This raises the question of which contexts are considered informative by the secondary learner. To answer this question, we apply IGF to the Mixed corpus containing both Reddit and Books. We created a dataset of 10,000 ( X, IG ( X )) pairs using an objective set of 160 contexts with 32 tokens each drawn solely from the Books corpus. We used this dataset to train a secondary learner. Next, the secondary learner was fed randomly sampled contexts from the Mixed corpus. Because the objective set contains only examples from one corpus, we expect the secondary learner to assign higher IG ( X ) values to other examples from the same corpus. Figure 6 shows that there is indeed a significant difference in the distributions of  Q values between the two corpora, demonstrating that the Books and Reddit corpora can be separated by the secondary learner. Almost all examples from the Reddit corpus are expected by the secondary learner to produce a reduction in perplexity that is at least one standard deviation below the mean. This indicates that the secondary learner can identify with strong confidence that Books corpus examples are more informative for fine-tuning towards the Books objective than Reddit corpus examples. It is also worthwhile to note that the secondary learner achieves dataset separation despite having access to just 160 labeled examples of 32 tokens in our objective set, a total of just 5120 tokens from the Books corpus, and zero examples from the Reddit corpus. For previous results we used a simple convolutional neural network described in Section 3 as our secondary learner. However, it may not be necessary to use a such a complex model for IG ( X ) . Alternative methods could provide similar performance at less cost. Figure 5 shows predicted vs. actual normalized IG ( X ) values for several learning methods. While the 45,000 parameter convolutional neural network is most effective at approximating IG ( X ) , other learners perform almost well. We encoded the contexts both by using the standard GPT-2 Small word embedding and with a one-hot encoding of the token identities. Standard linear regression performed on both encoding types (30K parameters for word embeddings and 450,000 parameters for one-hot encoding) performs nearly as well at approximating IG ( X ) with a convolutional model. We also tested an even simpler learner with only 25,000 parameters that assigned each token a value by averaging the IG ( X ) values for contexts that contained that token. Values for new contexts are then computed as the average of token values contained in that context. Even this model is a reasonable approximator of IG ( X ) . This underscores that, while IG ( X ) is an extremely complex function to compute exactly , it can nevertheless be effectively approximated through simple unigram information. Figure 7 compares the performance of these secondary learners architectures across different numbers of training examples. Here the convolutional network is the most sample efficient method, as it can effectively learn IG ( X ) with as few as 2,000 training examples. We proposed IGF as a prospective alternative to the random seed search approach suggested by Dodge et al. (2020). Since IGF aims to replaces random search with a directed search, we expect IGF to be significantly more efficient. Of course the methods can also be combined: IGF can be run many times with different data orders, and then the best model selected. In Figure 8 we compare the Dodge et al. (2020) method, where the best model is selected across 1, 5, or 50 runs of standard finetuning, to a similar setup where the best IGF model is selected across 1, 5, or 50 runs. We find that even in a single run, IGF significantly outperforms choosing the best of 50 runs of standard finetuning. Still, IGF performance can be improved even further by choosing the best result across 5 or 50 runs. This suggests that while IGF exploits some of the benefits that could be gained from ideal data ordering, there are still improvements to be made over IGF for further improving data order during language model fine-tuning. In the context of language model fine-tuning, we have shown that a secondary learner can efficiently and effectively distinguish between informative and uninformative training examples. This secondary learner can be used to select useful training examples in a technique we call Information Gain Filtration, leading to better model perfor-SF-1 SF-5 SF-50 IGF-1 IGF-5IGF-50 Method and Number of Runs 54 56 58 T e s t p e r p l e x i t y Figure 8: Prospective IGF Is More Efficient Than Retrospective Random Seed Search : We show boxplots of the best run from differently-sized sets of runs to visualize the expected benefit of using random seed testing (Dodge et al., 2020) and compare it to the benefit of using IGF. mance than standard fine-tuning. We encourage researchers to release pretrained secondary learners for frequently used corpora, in order to enable more effective finetuning and save energy. This would cut down the largest computational cost of applying IGF while retaining the performance improvements across the field. We have included several examples of open-sourced secondary learners in the supplementary material to promote this paradigm. This work also raises several questions. Since our focus was on developing a lightweight technique, the most complex secondary learner we tested was a small convolutional network. Data efficiency during training could potentially be further improved by using a more complex model. The question of how far one could reasonably take a function approximator network for estimating information gain remains unexplored. Finally, we do not fully understand why improving performance on early training batches results better performance at convergence. Is this exclusively a property of language models, or do other networks and tasks exhibit this phenomenon? Answering this question could lead to better optimization methods across many different fields. We would like to thank all of the people whose contributions, opinions and suggestions helped with the development of this project, especially Kaj Bostrom, Greg Durrett, Shailee Jain, and Vy Vo. This project was funded by a generous gift from Intel, and by the Burroughs-Wellcome Career Award at the Scientific Interface.
abstain abstain abstain abstain result method abstain abstain result result method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result method result abstain method other other other other other other other other abstain other other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain method method method method other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain result result abstain method abstain abstain abstain objective abstain abstain result abstain abstain other other