State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. To gain a better understanding of how these models learn, we study their generalisation and memorisation capabilities in noisy and low-resource scenarios. We find that the training of these models is almost unaffected by label noise and that it is possible to reach near-optimal results even on extremely noisy datasets. However, our experiments also show that they mainly learn from high-frequency patterns and largely fail when tested on low-resource tasks such as few-shot learning and rare entity recognition. To mitigate such limitations, we propose an extension based on prototypical networks that improves performance in low-resource named entity recognition tasks. With recent advances in pre-trained language models (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; He et al., 2020), the field of natural language processing has seen improvements in a wide range of tasks and applications. Having acquired general-purpose knowledge from large amounts of unlabelled data, such methods have been shown to learn effectively with limited labelled data for downstream tasks (Howard and Ruder, 2018) and to generalise well to out-of-distribution examples (Hendrycks et al., 2020). Previous work has extensively studied what such models learn, e.g. the types of relational or linguistic knowledge (Tenney et al., 2019; Jawahar et al., 2019; Rogers et al., 2020). However, the process of how these models learn from downstream data and the qualitative nature of their learning dynamics remain unclear. Better understanding of the learning processes in these widely-used models is  Work done prior to joining Google. needed in order to know in which scenarios they will fail and how to improve them towards more robust language representations. The fine-tuning process in pre-trained language models such as BERT (Devlin et al., 2019) aims to strike a balance between generalisation and memorisation. For many applications it is important for the model to generaliseto learn the common patterns in the task while discarding irrelevant noise and outliers. However, rejecting everything that occurs infrequently is not a reliable learning strategy and in many low-resource scenarios memorisation can be crucial to performing well on a task (Tu et al., 2020). By constructing experiments that allow for full control over these parameters, we are able to study the learning dynamics of models in conditions of high label noise or low label frequency. To our knowledge, this is the first qualitative study of the learning behaviour of pre-trained transformer-based language models in conditions of extreme label scarcity and label noise. We find that models such as BERT are particularly good at learning general-purpose patterns as generalisation and memorisation become separated into distinct phases during their fine-tuning. We also observe that the main learning phase is followed by a distinct performance plateau for several epochs before the model starts to memorise the noise. This makes the models more robust with regard to the number of training epochs and allows for noisy examples in the data to be identified based only on their training loss. However, we find that these excellent generalisation properties come at the cost of poor performance in few-shot scenarios with extreme class imbalances. Our experiments show that BERT is not able to learn from individual examples and may never predict a particular label until the number of training instances passes a critical threshold. For example, on the CoNLL03 (Sang and De Meulder, 2003) dataset it requires 25 instances of a class to 7564 learn to predict it at all and 100 examples to predict it with some accuracy. To address this limitation, we propose a method based on prototypical networks (Snell et al., 2017) that augments BERT with a layer that classifies test examples by finding their closest class centroid. The method considerably outperforms BERT in challenging training conditions with label imbalances, such as the WNUT17 (Derczynski et al., 2017) rare entities dataset. Our contributions are the following: 1) We identify a second phase of learning where BERT does not overfit to noisy datasets. 2) We present experimental evidence that BERT is particularly robust to label noise and can reach near-optimal performance even with extremely strong label noise. 3) We study forgetting in BERT and verify that it is dramatically less forgetful than some alternative methods. 4) We empirically observe that BERT completely fails to recognise minority classes when the number of examples is limited and we propose a new model, ProtoBERT, which outperforms BERT on few-shot versions of CoNLL03 and JNLPBA , as well as on the WNUT17 dataset. Several studies have been conducted on neural models' ability to memorise and recall facts seen during their training. Petroni et al. (2019) showed that pre-trained language models are surprisingly effective at recalling facts while Carlini et al. (2019) demonstrated that LSTM language models are able to consistently memorise single out-of-distribution (OOD) examples during the very first phase of training and that it is possible to retrieve such examples at test time. Liu et al. (2020) found that regular-ising early phases of training is crucial to prevent the studied CNN residual models from memorising noisy examples later on. They also propose a regularisation procedure useful in this setting. Similarly, Li et al. (2020) analyse how early stopping and gradient descent affect model robustness to label noise. Toneva et al. (2019), on the other hand, study forgetting in visual models. They find that models consistently forget a significant portion of the training data and that this fraction of forgettable examples is mainly dependent on intrinsic properties of the training data rather than the specific model. In contrast, we show that a pretrained BERT forgets examples at a dramatically lower rate compared to a BiLSTM and a non-pretrained variant. Memorisation is closely related to generalisation: neural networks have been observed to learn simple patterns before noise (Arpit et al., 2017) and generalise despite being able to completely memorise random examples (Zhang et al., 2017). Zhang et al. (2021) also show that our current understanding of statistical learning theory cannot explain the superhuman generalisation performance of large neural models across many areas of study. Hendrycks et al. (2020) show that pre-trained models generalise better on out-of-distribution data and are better able to detect such data compared to non-pretrained methods but that they still do not cleanly separate inand out-of-distribution examples. Kumar et al. (2020) find that pre-trained methods such as BERT are sensitive to spelling noise and typos. In contrast to noise in the input, we focus on the models' learning dynamics in the presence of label noise and find that pre-trained methods are remarkably resilient to such cases. We investigate the performance of pre-trained language models in specific adverse conditions. In order to evaluate generalisation abilities, we first create datasets with varying levels of label noise by randomly permuting some of the labels in the training data. This procedure allows us to pinpoint noisy examples and evaluate the performance on clean and noisy datapoints separately. Then, in order to investigate memorisation we train the models on datasets that contain only a small number of examples for a particular class. This allows us to evaluate how well the models are able to learn from individual datapoints as opposed to high-frequency patterns. We make the code for the experiments available online. 1 Datasets We focus on the task of named entity recognition (NER) and employ the CoNLL03 (Sang and De Meulder, 2003), the JNLPBA (Col-lier and Kim, 2004), and the WNUT17 (Derczynski et al., 2017) datasets. NER is commonly used for evaluating pre-trained language models on structured prediction and its natural class imbalance is well suited for our probing experiments. CoNLL03 and JNLPBA are standard datasets for NER and Bio-NER respectively. The WNUT17 dataset is motivated by the observation that state-of-the-art methods tend to memorise entities during training 1 https://github.com/Michael-Tanzer/ BERT-mem-lowres 7565 (Augenstein et al., 2017). The dataset focuses on identifying unusual or rare entities at test time that cannot be simply memorised by the model. We evaluate based on entity-level F 1 unless stated otherwise. Language models We use BERT-base (Devlin et al., 2019) as the main language model for our experiments, as BERT is widely used in practice and other variations of pre-trained language models build on a similar architecture. The model is augmented with a classification feed-forward layer and fine-tuned using the cross-entropy loss with a learning rate of 10  4 . AdamW (Loshchilov and Hutter, 2019) is used during training with weight decay of 0.01 and a linear warm-up rate of 10%. The test results are recorded using the model that produced the highest validation metrics. We compare BERT's behaviour with that of other pre-trained transformers such as RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2020) fine-tuned with the same optimiser and hyper-parameters as above. In order to also compare against non-transformer models, we report performance for a bi-LSTM-CRF (Lample et al., 2016) model with combined character-level and word-level representations. The model is comprised of 10 layers, with 300-dimensional word representations and 50-dimensional character representations, for a total of approximately 30 million trainable parameters. In our experiments, the model is trained with the Adam optimiser (Kingma and Ba, 2014) and a learning rate of 10  4 for 100 epochs using a CRF loss (Lafferty et al., 2001). We first investigate how BERT learns general patterns from datasets that contain label noise. Figure 1 shows how the model performance on the CoNLL03 training and validation sets changes when faced with varying levels of noise, from 0% to 50%. Based on the progression of performance scores, we can divide BERT's learning process into roughly three distinct phases: 1. Fitting : The model uses the training data to learn how to generalise, effectively learning simple patterns that can explain as much of the training data as possible (Arpit et al., 2017). Both the training and validation performance rapidly increase as the model learns these patterns. 2. Settling : The increase in performance plateaus and neither the validation nor the training performance change considerably. The duration of this phase seems to be inversely proportional to the amount of noise present in the dataset. 3. Memorisation : The model rapidly starts to memorise the noisy examples, quickly improving the performance on training data while degrading the validation performance, effectively over-fitting to the noise in the dataset. A second phase of learning We find BERT to exhibit a distinct second settling phase during which it does not over-fit. A resilience to label noise has been observed in other neural networks trained with gradient descent (Li et al., 2020). However, we find this phase to be much more prolonged in BERT compared to models pre-trained on other 7566 modalities such as a pre-trained ResNet fine-tuned on CIFAR10 , which immediately starts memorising noisy examples (see Appendix A for a compar-ison). These results indicate that the precise point of early stopping is not as important when it comes to fine-tuning pre-trained language models. Similar optimal performance is retained for a substantial period, therefore training for a fixed number of epochs can be sufficient. We illustrate BERT's behaviour by evaluating the token-level classification accuracy of noisy examples in Figure 2. During the second phase, BERT completely ignores the noisy tokens and correctly misclassifies them, performing worse than a random classifier. The step-like improvements during the third stage show that the model is unable to learn any patterns from the noise and improves by repeatedly optimising on the same examples, gradually memorising them. Robustness to noise We also observe in Figure 1 that BERT is extremely robust to noise and over-fitting in general. In the absence of noise, the model does not over-fit and maintains its development set performance, regardless of the length of training. Even with a large proportion of noise, model performance comparable to training on the clean dataset can be achieved by stopping the training process somewhere in the second phase. 2 We also hypothesise that due to the robustness to noise shown in the second phase of training, a noise detector can be constructed based only on BERT's training losses, without requiring any other information. We find that a simple detector that clusters the losses using k-means reliably achieves over 90% noise-detection F 1 score in all our experiments, further showing how the model is able to actively detect and reject single noisy examples (see Appendix E for details about the noise detection process). Impact of pre-training The above properties can mostly be attributed to BERT's pre-training processafter large-scale optimisation as a language model, the network is primed for learning general patterns and better able to ignore individual noisy examples. We find that a randomly initialised model with the same architecture does not only achieve lower overall performance but crucially does not exhibit's BERT's distinct second phase of 2 Adding 30% noise to the CoNLL03 dataset causes only a 0.9% decrease of validation performance in the second phase. Other pre-trained transformers We also analyse the behaviour of other pre-trained transformers for comparison. Specifically, studying RoBERTa and DeBERTa, we find the same training pattern that was observed in BERTall models show a clear division into the three phases described above. These models are also all very robust to label noise during the settling phase of training. Notably, RoBERTa is even more resilient to label noise compared to the other two analysed models, despite DeBERTa outperforming it on public benchmarks (He et al., 2020). Training and validation performance visualisations, such as those in Figure 1, can be found for both models in Appendix I. 5 Forgetting of learned information Evaluating only the final model does not always provide the full picture regarding datapoint memorisation, as individual datapoints can be learned and forgotten multiple times during the training process. Following Toneva et al. (2019), we record a forgetting event for an example at epoch t if the model was able to classify it correctly at epoch t  1 , but not at epoch t . Similarly, we identify a learning event for an example at epoch t if the model was not able to classify it correctly at epoch t  1 , but it is able to do so at epoch t . A first learning event thus happens at the first epoch when a model is able to classify an example correctly. We furthermore refer to examples with zero and more than zero forgetting events as unforgettable and forgettable examples, respectively, while the set of learned examples includes all examples with one or more learning events. In Table 1, we show the number of forgettable, unforgettable, and learned examples on the training data of the CoNLL03 and JNLPBA datasets for BERT, a non-pre-trained BERT, and a bi-LSTM model. We also show the ratio between forgettable and learned examples, which indicates how easily a model forgets learned information. We can observe that BERT forgets less than other models and that pre-training is crucial for retaining important information. We show the most forgettable examples in Appendix D, which tend to be atypical examples of the corresponding class. Toneva et al. (2019) found that the number of forgetting events remains comparable across different architectures for the vision modality, given 7567 Dataset Model Forgettable N f Unforgettable N u Learned N l N f /N l (%) CoNNL03 bi-LSTM 71.06% 29.94% 90.90% 78.17% non-pre-trained BERT 9.89% 90.11% 99.87% 9.90% pre-trained BERT 2.97% 97.03% 99.80% 2.98% JNLPBA bi-LSTM 97.16% 5.14% 98.33% 98.81% non-pre-trained BERT 25.50% 74.50% 98.24% 25.96% pre-trained BERT 16.62% 83.38% 98.18% 16.93% Table 1: Number of forgettable, unforgettable, and learned examples during BERT training on the CoNLL03 dataset and JNLPBA dataset. a particular dataset. 3 However, our experiments show that the same does not necessarily hold for pre-trained language models. Specifically, there is a large discrepancy in the ratio between forgettable and learned examples for BERT (  3%) and a bi-LSTM model (  80%) on the CoNLL03 dataset. We additionally analyse the distribution of first learning events throughout BERT's training on CoNLL03 with label noise between 0% and 50% (Figure 3) and notice how BERT learns the majority of learned examples during the first epochs of training. As the training progresses, we see that BERT stops learning new examples entirely, regardless of the level of noise for the third and fourth epochs. Finally, in the last epochs BERT mostly memorises the noise in the data. 4 3 They report proportions of forgettable examples for MNIST, PermutedMNIST, CIFAR10, and CIFAR100 as 8.3%, 24.7%, 68.7%, and 92.38% respectively. 4 We conducted additional experiments on other datasets (see Appendix F for results on the JNLPBA dataset). In all cases we observe the same distribution of first learning events throughout training. In the previous sections, we have observed that BERT learns examples and generalises very early in training. We will now examine if the same behaviour applies in low-resource scenarios where a minority class is only observed very few times. To this end, we remove from the CoNLL03 training set all sentences containing tokens with the minority labels MISC and LOC except for a predetermined number of such sentences. We repeat the process for the JNLPBA dataset with the DNA and Protein labels. We conduct similar experiments to the previous sections by studying how different numbers of sentences containing the target class affect BERT's ability to learn and generalise. We report in Figure 4 the training and validation classification F 1 score for the CoNLL03 datasets from which all but few (5 to 95) sentences containing the LOC label were removed. Note that the reported performance in this experiment refers to the LOC class only. In Figure 5 we also report the distribution of first learning 7568 1 2 3 4 5 6 7 8 9 10 Epoch 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 R a t i o o f t r a i n i n g e x a m p l e s BERT phase 1 phase 2 phase 3 Figure 5: First learning events distribution during the training on the CoNLL03 dataset with varying number of sentences containing the LOC class. events for the LOC class in the same setting. Two phenomena can be observed: 1) reducing the number of sentences greatly reduces the model's ability to generalise (validation performance decreases yet training performance remains comparable); and 2) when fewer sentences are available, they tend to be learned in earlier epochs for the first time. Corresponding experiments on the MISC label can be found in Appendix J. We also show the average entity-level F 1 score on tokens belonging to the minority label and the model performance for the full NER task (i.e. considering all classes) for the CoNLL03 and JNLPBA datasets in Figures 6 and 7 respectively. For the CoNLL03 dataset, we observe that BERT needs at least 25 examples of a minority label in order to be able to start learning it. Performance rapidly improves from there and plateaus at around 100 examples. For the JNLPBA dataset, the minimum number of examples increases to almost 50 and the plateau occurs for a higher number of examples. On the challenging WNUT17 dataset, BERT achieves only 44% entity-level F 1 . This low performance is attributable to the absence of entity overlap between training set and test set, which increases the inter-class variability of the examples. In order to address BERT's limitations in few-shot learning, we propose a new model, ProtoBERT that combines BERT's pre-trained knowledge with the few-shot capabilities of prototypical networks (Snell et al., 2017) for sequence labelling problems. The method builds an embedding space where the inputs are clustered on a per-class basis, allowing us to classify a token by finding its closest centroid and assigning it the corresponding class. The model can be seen in Figure 8. We first define a support set S , which we use as context for the classification and designate with S k all elements of S that have label k . We refer to the set of points that we want to classify as the query set Q , with l ( Q i ) indicating the label of the i th element in Q . We will also refer to f as the function computed by BERT augmented with a linear layer, which produces an M dimensional output. The model then classifies a given input x as follows: for each class k , we compute the centroid of the class in the learned feature space as the mean of all the elements that belong to class k in the 7569 Trainingdata Query set Supportset Query embeddings Support embeddings Class centroidsSimilarities Output probabilities Similarity Per-classaverage S Softmax Randomsampler BERT Mapping layer Figure 8: Schematic representation of the inference using a BERT model with a prototypical network layer. support set S : c k = 1 | S k | (cid:88) x i  S k f ( x i ) (1) Then, we compute the distance from each input x  Q to each centroid: dist k = d ( f ( x ) , c k ) and collect them in a vector v  R k . Finally, we compute the probability of x belonging to class k as p ( y = k | x ) = exp (  d ( f ( x ) , c k )) (cid:80) k (cid:48) exp (  d ( f ( x ) , c k (cid:48) )) = = softmax (  v ) k The model is trained by optimising the cross-entropy loss between the above probability and the one-hot ground-truth label of x . Crucially, S and Q are not a fixed partition of the training set but change at each training step. Following Snell et al. (2017), we use Euclidean distance as a choice for the function d . In order to take into account the extreme underrepresentation of some classes, we create the support by sampling s 1 elements from each minority class and s 2 elements from each non-minority class. A high ratio s 1 /s 2 gives priority to the minority classes, while a low ratio puts more emphasis on the other classes. We then similarly construct the query set with a fixed ratio n between the minority classes and the non-minority classes. For NER, rather than learning a common representation for the negative class  O , we only want the model to treat it as a fallback when no other similar class can be found. For this reason, we define the vector of distances v as follows: v = ( d O , dist 0 , . . . , dist k ) where d O is a scalar parameter of the network that is trained along with the other parameters. (i.e. class O ) when it is not close enough to any centroid, where d O represents the threshold for which we consider a point close enough. If no example of a certain class is available in the support set during the training, we assign a distance of 400 , making it effectively impossible to mistakenly classify the input as the missing class during that particular batch. Finally, we propose two ways to compute the class of a token at test time. The first method employs all examples from X to calculate the centroids needed at test time, which produces better results but is computationally expensive for larger datasets. The second method approximates the centroid c k using the moving average of the centroids produced at each training step: c ( t ) k   c ( t ) k  (1   ) c ( t  1) k where  is a weighting factor. This method results in little overhead during training and only performs marginally worse than the first method. We first compare ProtoBERT to the standard pretrained BERT model with a classification layer on the CoNLL03 and JNLPBA datasets with a smaller number of sentences belonging to the minority classes. We show the results on the few-shot classes and for the full dataset for CoNLL03 in Figures 9 and 10 respectively. Similarly, we show the results for the few-shot class for JNLPBA in Figure 11. 5 In all cases ProtoBERT consistently surpasses the performance of the baseline when training on few examples of the minority class. It particularly excels in the extreme few-shot setting, e.g. outperforming BERT by 40 F 1 points with 15 sentences containing the LOC class. As the number of available examples of the minority class increases, 5 A comparison on the full classification task can be found in Appendix H. 7570 0 25 50 75 100 125 150 175 200 Number of training sentences containing the target label 0.0 0.2 0.4 0.6 0.8 T e s t F 1 s c o r e MISC ProtoBERT MISC BERT + Class. BERT starts to match ProtoBERT's performance and outperforms it on the full dataset in some cases. While the main strength of ProtoBERT is on few-shot learning, we evaluate it also on the full CoNLL03 , JNLPBA and WNUT17 datasets (with-out removing any sentences) in Table 2. In this setting, the proposed architecture achieves results mostly similar to the baseline while considerably outperforming it on the WNUT17 dataset of rare entities. The results in this section show that ProtoBERT, while designed for few-shot learning, performs at least on par with its base model in all tasks. This allows the proposed model to be applied to a much wider range of tasks and datasets without negatively affecting the performance if no label imbalance is 0 25 50 75 100 125 150 175 200 Number of training sentences containing the target label 0.0 0.1 0.2 0.3 0.4 0.5 T e s t F 1 s c o r e DNA ProtoBERT DNA BERT + Class. present, while bringing a substantial improvement in few-shot scenarios. We conduct an ablation study to verify the effect of our improved centroid computation method. From the results in Table 2 we can affirm that, while a difference in performance does exist, it is quite modest (0.10.4%). On the other hand, this method reduces the training time and therefore energy consumption (Strubell et al., 2019) to one third of the original method on CoNLL03 and we expect the reduction to be even greater for larger datasets. In this study, we investigated the learning process during fine-tuning of pre-trained language models, focusing on generalisation and memorisation. By formulating experiments that allow for full control over the label distribution in the training data, we study the learning dynamics of the models in conditions of high label noise and low label frequency. The experiments show that BERT is capable of reaching near-optimal performance even when a large proportion of the training set labels has been corrupted. We find that this ability is due to the model's tendency to separate the training into three distinct phases: fitting, settling, and memorisation, which allows the model to ignore noisy examples in the earlier epochs. The pretrained models experience a prolonged settling phase when fine-tuned, during which their performance remains optimal, indicating that the precise area of early stopping is less crucial. Furthermore, we show that the number of avail-7571 Model CoNLL03 JNLPBA WNUT17 State of the art 93.50 77.59 50.03 BERT + classification layer (baseline) 89.35 75.36 44.09 ProtoBERT 89.87 73.91 48.62 ProtoBERT + running centroids 89.46 73.54 48.56 Table 2: Comparison between the baseline model, the current state-of-the-art 6 and the proposed architecture on the CoNLL03 , JNLPBA and WNUT17 datasets evaluated using entity-level F 1 score. able examples greatly affects the learning process, influencing both when the examples are memorised and the quality of the generalisation. We show that BERT fails to learn from examples in extreme few-shot settings, completely ignoring the minority class at test time. To overcome this limitation, we augment BERT with a prototypical network. This approach partially solves the model's limitations by enabling it to perform well in extremely low-resource scenarios and also achieves comparable performance in higher-resource settings. Michael is funded by the UKRI CDT in AI for Healthcare 7 (Grant No. P/S023283/1).
abstain method result result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective objective result abstain result result abstain abstain abstain objective method result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method abstain method abstain method method abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain objective method abstain result abstain other abstain result objective abstain other