In real-world scenarios, a text classification task often begins with a cold start , when labeled data is scarce. In such cases, the common practice of fine-tuning pre-trained models, such as BERT, for a target classification task, is prone to produce poor performance. We suggest a method to boost the performance of such models by adding an intermediate unsupervised classification task, between the pretraining and fine-tuning phases. As such an intermediate task, we perform clustering and train the pre-trained model on predicting the cluster labels. We test this hypothesis on various data sets, and show that this additional classification phase can significantly improve performance, mainly for topical classification tasks, when the number of labeled instances available for fine-tuning is only a couple of dozen to a few hundred. The standard paradigm for text classification relies on supervised learning, where it is well known that the size and quality of the labeled data strongly im-pact the performance (Raffel et al., 2019). Hence, developing a text classifier in practice typically requires making the most of a relatively small set of annotated examples. The emergence of transformer-based pre-trained language models such as BERT (Devlin et al., 2018) has reshaped the NLP landscape, leading to significant advances in the performance of most NLP tasks, text classification included (e.g., Nogueira and Cho, 2019; Ein-Dor et al., 2020). These models typically rely on pretraining with massive and heterogeneous corpora on a general Masked Language Modeling ( MLM ) task, i.e., predicting a word that is masked in the original text. Later on, the obtained model is fine-tuned to the actual task of interest, termed here the target task ,  These authors contributed equally to this work. using the labeled data available for this task. Thus, pretrained models serve as general sentence encoders which can be adapted to a variety of target tasks (Lacroix et al., 2019; Wang et al., 2020a). Our work focuses on a challenging yet common scenario, where unlabeled data is available but labeled data is scarce. In many real-world scenarios, obtaining even a couple of hundred of labeled examples per class is challenging. Commonly, a target class has a relatively low prior in the examined data, making it a formidable goal to collect enough positive examples for it (Japkowicz and Stephen, 2002). Moreover, sometimes data cannot be labeled via crowd-annotation platforms due to its confidentiality (be it for data privacy reasons or for protecting intellectual property) or since the labeling task requires special expertise. On top of this, often the number of categories to be considered is relatively large, e.g., 50 , thus making even a modest demand of 200 labeled examples per class a task of labeling 10K instances, which is inapplicable in many practical cases (for an extreme example, cf. Partalas et al., 2015). In such limited real-world settings, fine-tuning a large pretrained model often yields far from optimal performance. To overcome this, one may take a gradual approach composed of various phases. One possibility is to further pretrain the model with the self-supervised MLM task over unlabeled data taken from the target task domain (Whang et al., 2019). Alternatively, one can train the pretrained model using a supervised intermediate task which is different in nature from the target-task, and for which labeled data is more readily available (Pruk-sachatkun et al., 2020; Wang et al., 2019a; Phang et al., 2018). Each of these steps is expected to provide a better starting point for the final finetuning phase, performed over the scarce labeled data available for the target task, aiming to end up with improved performance. egy that exploits unsupervised text clustering as the intermediate task towards fine-tuning a pretrained model for text classification. Our work is inspired by the use of clustering to obtain labels in computer vision (Gidaris et al., 2018; Kolesnikov et al., 2019). Specifically, we use an efficient clustering technique, that relies on simple Bag Of Words (BOW) representations, to partition the unlabeled training data into relatively homogeneous clusters of text instances. Next, we treat these clusters as labeled data for an intermediate text classification task, and train the pre-trained model  with or without additional MLM pretraining  with respect to this multi-class problem, prior to the final fine-tuning over the actual target-task labels. Extensive experimental results demonstrate the practical value of this strategy on a variety of benchmark data. We further analyze the results to gain insights as to why and when this approach would be most valuable, and conclude that it is most prominently when the training data available for the target task is relatively small and the classification task is of a topical nature. Finally, we propose future directions. 1 2 Intermediate Training using Unsupervised Clustering A pre-trained model is typically developed in consecutive phases. We release code for reproducing our method. Henceforth, we will refer to BERT as the canonical example of such models. First, the model is pretrained over massive general corpora with the MLM task. 2 We denote the obtained model simply as BERT . Second, BERT is finetuned in a supervised manner with the available labeled examples for the target task at hand. This standard flow is represented via Path-1 in Fig. 1. An additional phase can be added between these two, referred to next as intermediate training , or inter-training in short. In this phase, the model is exposed to the corpus of the target task, or a corpus of the same domain, but still has no access to labeled examples for this task. A common example of such an intermediate phase is to continue to intertrain BERT using the self-supervised MLM task over the corpus or the domain of interest, sometimes referred to as further 1 https://github.com/IBM/ intermediate-training-using-clustering 2 BERT was originally also pretrained over "next sentence prediction"; however, later works (Yang et al., 2019; Liu et al., 2019b) have questioned the contribution of this additional task and focused on MLM. or adaptive pre-training (e.g., Gururangan et al., 2020). This flow is represented via Path-2 in Fig. 1, and the resulting model is denoted BERT IT:MLM , standing for Intermediate Task: MLM. A key contribution of this paper is to propose a new type of intermediate task, which is designed to be aligned with a text classification target task, and is straightforward to use in practice. The underlying intuition is that inter-training the model over a related text classification task would be more beneficial compared to MLM inter-training, which focuses on different textual entities, namely predicting the identity of a single token. Specifically, we suggest unsupervised clustering for generating pseudo-labels for inter-training. As the clustering partition presumably captures information about salient features in the corpus, feeding this information into the model could lead to representations that are better geared to perform the target task. These pseudo-labels can be viewed as weak labels, but importantly they are not tailored nor require a specific design per target task. Instead, we suggest generating pseudo-labels in a way in-dependent of the target classification task. The respective flow is represented via Path-3 in Fig. 1. In this flow, we first cluster to partition the training data into n c clusters. Next, we use the obtained partition as labeled' data in a text classification task, where the classes are defined via the n c clusters, and intertrain BERT to predict the cluster label. In line with MLM, inter-training includes a classifier layer on top of BERT, which is discarded before the fine-tuning stage. The resulting inter-trained model is denoted BERT IT:CLUST . Finally, Path-4 in Fig. 1 represents a sequential composition of Paths 2 and 3. In this flow, we first intertrain BERT with the MLM task. Next, the obtained model is further intertrained to predict the n c clusters, as in Path-3. The model resulting from this hybrid approach is denoted BERT IT:MLM+CLUST . Importantly, following Path-3 or Path-4 requires no additional labeled data, and involves an a-priori clustering of training instances that naturally gives rise to an alternative or an additional inter-training task. As we show in the following sections, despite its simplicity, this strategy provides a significant boost in performance, especially when labeled data for the final fine-tuning is in short supply. We evaluate over 6 topical datasets and 3 non-topical ones (see Table 1), which cover a variety of classification tasks and domains: Yahoo! Answers (Zhang et al., 2015), which separates answers and questions to types; DBpedia (Zhang et al., 2015, CC-BY-SA) which differentiates entity types by their Wikipedia articles; AG's News (Zhang et al., 2015) which categorize news articles; CFPB, which classifies consumer complaints; 20 newsgroups (Lang, 1995), which classifies 20 Usenet discussion groups; ISEAR (Shao et al., 2015, CC BY-NC-SA 3.0), which considers personal reports for emotion; SMS spam (Almeida et al., 2011), which identifies spam messages; Polarity (Pang and Lee, 2005), which includes sentiment analysis on movie reviews, and Subjectivity (Pang and Lee, 2004), which categorizes movie snippets as subjective or objective. A topical dataset splits sentences by a high-level distinction related to what the sentence is about (e.g., sports vs. economics). Non-topical datasets look for finer stylistic distinctions that may depend on the way the sentence is written or on fine details rather than on the central meaning it discusses. It may also separate almost identical sentences; for example, "no" could distinguish between sentences with negative and positive sentiment. When no split is provided we apply a 70% / 10% / 20% train-dev-test split, respectively. 3 To reduce the computational cost over the larger datasets (DBpedia, AG's News, Yahoo! Answers and CFPB) we trim the train/test sets of these datasets to 15 K/ 3 K instances respectively, by randomly sampling from each set. 4 All runs and all methods use only the trimmed versions. In our main set of experiments, we compare the performance of fine-tuning BERT-based models over a target task, for different settings of intermediate training. We consider four BERT-based settings, as described in Section 2 and in Figure 1. Two baselines (i) BERT, without intermediate training, and (ii) BERT IT:MLM intertrained on MLM; and two settings that rely on clustering (i) BERT IT:CLUST , where predicting cluster labels is used for inter-training, and (ii) BERT IT:MLM+CLUST , which combines the two intermediate tasks. Training samples: For each setting, the final fine-tuning for the target task is performed, per dataset, for training budgets varying between 64 and 1024 labeled examples. For each data size x , the experiment is repeated 5 times; each repetition representing a different sampling of x labeled examples from the train set. The samplings of training examples are shared between all settings. That is, for a given dataset and train size the final training for all settings is done with respect to the same 5 samples of labeled examples. Inter-training: Intermediate training, when done, was performed over the unlabeled train set for each dataset (ignoring instances' labels). We studied two implementations for the clustering task: K-means (Lloyd, 1982) and sequential Information Bottleneck (sIB) which is known to obtain better results in practice (Slonim et al., 2002) and in theory (Slonim et al., 2013). Based on initial experiments, and previous insights from works in the computer vision domain (Yan et al., 2020) we opted for a relatively large number of clusters, and rather than optimizing the number of clusters per dataset, set it to 50 for all cases. 5 K-means was run over GloVe (Pennington et al., 2014) representations following word stemming. We used a publicly available implementation of sIB 6 with its default configuration (i.e., 10 restarts and a maximum of 15 iterations for every single run). For sIB clustering, we used Bag of Words (BOW) representations on a stemmed text with the default vocabulary size (which is defined as the 10K most frequent words in the dataset). Our results indicate that inter-training with respect to sIB clusters consistently led to better results in the final performance on the target task, compared to inter-training with respect to the clusters obtained with K-means (see Section 5.1 for details). We also considered inter-training only on representative examples of clustering results  filtering a given amount of outlier examples  but obtained no significant gain (data not shown). Note that the run time of the clustering algorithms is only a few seconds. The run time of the fine-tuning step of the inter-training task takes five and a half minutes for the largest train set (15K instances) on a Tesla V100-PCIE-16GB GPU. 5 Setting the number of clusters to be equal to the number of classes resulted in inferior accuracy. In addition, one may not know how many classes truly exist in the data, so this parameter is not necessarily known in real-world applications. 6 https://github.com/IBM/sib Train Test # classes Yahoo! answers 15K 3K 10 DBpedia 15K 3K 14 CFPB 15K 3K 15 20 newsgroups 10.2K 7.5K 20 AG's news 15K 3K 4 ISEAR 5.4K 1.5K 7 SMS spam 3.9K 1.1K 2 Subjectivity 7K 2K 2 Polarity 7.5K 2.1K 2 Table 1: Dataset details. BERT hyper-parameters: The starting point of all settings is the BERTBASE model (110M pa-rameters). BERT inter-training and fine-tuning runs were all performed using the Adam optimizer (Kingma and Ba, 2015) with a standard setting consisting of a learning rate of 3  10  5 , batch size 64 , and maximal sequence length 128 . In a practical setting with a limited annotations budget one cannot assume that a labeled dev set is available, thus in all settings we did not use the dev set, and fine-tuning was arbitrarily set to be over 10 epochs, always selecting the last epoch. For inter-training over the clustering results we used a single epoch, for two reasons. First, loosely speaking, additional training over the clusters may drift the model too far towards learning the partition into clusters, which is an auxiliary task in our context, and not the real target task. Second, from the perspective of a practitioner, single epoch training is preferred since it is the least demanding in terms of run time. For BERT IT:MLM we used 30 epochs with a replication rate of 5 , and followed the masking strategy from Devlin et al. (2018). 7 Computational budget: Overall we report the results of 1440 BERT fine-tuning runs ( 4 experimental settings  9 datasets  8 labeling budgets  5 repetitions). In addition, we performed 288 inter-training epochs over the full datasets ( 9 datasets  ( 30 BERT IT:MLM epochs + 1 BERT IT:CLUST epoch + 1 BERT IT:MLM+CLUST epoch)). In total, this would equate to about 60 hours on a single Tesla V100-PCIE-16GB GPU. Table 2 depicts the results over all datasets, focusing on the practical use case of a budget of 64 7 In preliminary experiments we found this to be the best configuration for this baseline. samples for fine-tuning (128 for 20 newsgroup, see explanation in Fig. 2). As shown in the table, the performance gains of BERT IT:CLUST are mainly reflected in the 6 topical datasets. For these datasets, BERT IT:CLUST confers a significant benefit in accuracy ( 110% accuracy gain, 33% error reduction). Figure 2 depicts the classification accuracy for the different settings for varying labeling budgets, using sIB for clustering-based inter-training. Over the topical datasets, BERT IT:CLUST and BERT IT:MLM+CLUST clearly outperform BERT and BERT IT:MLM in the small labeled data regime, where the gain is most prominent for the smallest labeled data examined  when only 64 labeled examples are available  and gradually diminishes as more labeled samples are added. We performed paired t-tests to compare BERT IT:CLUST with BERT and BERT IT:MLM , pooling together all datasets and repetitions for a given Dataset BERTaccuracy BERT IT:CLUST accuracy Gain Error reduction Yahoo! Answers 21.2 45.9 117% 31% DBpedia 31.2 67.0 115% 52% CFPB 15.0 27.5 83% 15% 20 newsgroup 13.0 47.2 263% 39% AG's News 61.9 80.7 30% 49% ISEAR 19.0 29.0 53% 12% avg. 7643 Train size 64 128 192 256 384 512 > 512 vs. BERT 1  10  6 1  10  6 6  10  7 2  10  5 2  10  3 9  10  3  vs. BERT IT:MLM 8  10  5 3  10  3 4  10  2     Table 3: Paired t-test p-values (after Bonferroni correction) of classification accuracy for BERT IT:CLUST compared to BERT and to BERT IT:MLM (insignificant results, p  0 . 05 , are denoted by ). labeling budget. As can be seen in Tab. 3, the performance gain, over all datasets, of BERT IT:CLUST over BERT is statistically significant for a budget up to 512 . BERT IT:CLUST is not as successful in the 3 non-topical datasets (cf. Tab. 2 and Fig. 2). A possible reason for the lack of success of inter-training in these three datasets is that their classification task is different in nature than the tasks in the other six datasets. Identifying spam messages, determining whether a text is subjective or objective, or analyzing the sentiment (polarity) of texts, can be based on stylistic distinctions that may depend on the way the sentence is written rather than on the central topic it discusses. Inter-training over BOW clustering seems to be less beneficial when such considerations are needed. We further analyze this in Section 5.4. Nevertheless, it is safe to apply BERT IT:CLUST even in these datasets, as results are typically comparable to the baseline algorithms, neither better nor worse. Both BERT IT:MLM and BERT IT:CLUST expose the model to the target corpus. The performance gains of BERT IT:CLUST over BERT IT:MLM suggest that inter-training on top of the clustering carries an additional benefit. In addition, these inter-training approaches are complementary as seen in Fig. 2, BERT IT:MLM+CLUST outperforms both BERT IT:CLUST and BERT IT:MLM (at the cost of some added runtime). Taken together, our results suggest that in topical datasets, where labeled data is scarce, the pseudo-labels generated via clustering can be leveraged to provide a better starting point for a pre-trained model towards its fine-tuning for the target task. In the literature (Slonim et al., 2002) and on our initial trials, sIB showed better clustering performance, and therefore was chosen over other clustering methods. Next, we analyze whether sIB is also the best fit for inter-training. We compare (see App. C) sIB over BOW representation to two other clustering configurations; K-means over GloVe representations and Hartigan's K-means (Slonim et al., 2013) over GloVe. For most datasets, inter-training over the results of sIB over BOW representations achieved the best results. Our inter-training method relies on BOW-based clustering. Since knowledge of the input words is potentially quite powerful for some text classification tasks, we examine the performance of several BOW-based methods. We used the same training samples to train multinomial Naive Bayes (NB) and Support Vector Machine (SVM) classifiers, using either Bag of Words (BOW) or GloVe (Penning-ton et al., 2014) representations. For GloVe, a text is represented as the average GloVe embeddings of its tokens. This yielded four reference settings: NBBOW , NB GloVe , SVMBOW and SVM GloVe . Overall, all four methods were inferior to BERT IT:CLUST , as shown in App. B. Thus, the success of our method cannot simply be attributed to the information in the BOW representations. The embeddings after BERT IT:CLUST show potential as a better starting point for fine-tuning. Figure 3 depicts t-SNE (van der Maaten and Hinton, 2008) 2D visualizations of the output embeddings over the full train set of several datasets, comparing the [CLS] embeddings before and after inter-training. Manifestly, for topical datasets, the BERT IT:CLUST embeddings, obtained after inter-training with respect to sIB clusters, induce a much clearer separation between the target classes, even though no labeled data was used to obtain this model. Moreover, and perhaps not surprisingly, the apparent visual separation resulting from inter-training is aligned with the performance gain obtained later on in the fine-tuning phase 7644 Figure 3: t-SNE visualizations of model embeddings over the train set, using BERT (top) vs. BERT IT:CLUST (bottom). over the target task (as seen, for instance, in the visualizations of Polarity versus DBpedia data). In addition to the qualitative results of the visualization, we pursue a more quantitative path. We assess whether examples of the same class are more closely represented after inter-training. Formally, given a set of instances' embeddings e 1 , . . . , e n and their corresponding class labels l 1 , . . . , l n  L we compute for each class l  L a centroid c l which is the average embedding of this class. We then compute the average Euclidean Embeddings' Distance ( ED ) from the corresponding centroids: 8 ED ( l, e ) = E ni =0 (cid:107) e i  c i (cid:107) 2 As a sanity check, we apply a significance test to the ED statistic, confirming that representations of same-class examples are close to each other. Specifically, we apply a permutation test (Fisher, 1971), with 1000 repetitions, comparing the class labels to random labels. We find that EDs for both BERT and BERT IT:CLUST are significantly different from random ( p < 0 . 001 ). This implies that both before and after inter-training, same-class representations are close. Next, we compare the representations before and after inter-training. We find that the randomly permuted EDs of BERT IT:CLUST are about 3 times larger than BERT's, despite similar norm values. This means that the post inter-training representations are more dispersed. Hence, to properly compare, we normalize ED by the average of the 8 Macro average results were similar, we hence report only micro average results. Where   S n is a permutation out of S n the set of all permutations. Comparing the Normalized Embeddings' Distance ( NED ) before and after inter-training, we find that in all datasets the normalized distance is smaller after inter-training. In other words, BERT IT:CLUST brings same-class representations closer in comparison to BERT. A natural explanation for the contribution of inter-training to BERT's performance is that the pseudo-labels, obtained via the clustering partition, are informative with regards to target task labels. To quantify this intuition, in Figure 4 we depict the Normalized Mutual Information (NMI) between sIB labels and the target task labels, calculated over the entire training set, versus the gain of using BERT IT:CLUST , reflected as the reduction in classification error rate between BERT and BERT IT:CLUST , at the extreme case of 64 fine-tuning samples. Evidently, in datasets where the NMI is around zero, BERT IT:CLUST does not confer a clear benefit; conversely, where the NMI is relatively high, the performance gains are pronounced as well. Notably, the three datasets with the lowest NMI are those for which inter-training was not beneficial, as discussed in Section 4. Since the partition obtained via clustering is often informative for the target class labels, we examine whether it can be utilized directly, as opposed 7645 to as pseudo-labels for BERT inter-training. To that end, we applied a simple heuristic. Given a labeling budget x , we divide it across clusters, relative to their size, while ensuring that at least one instance within each of the 50 clusters is labeled. We use the budget per cluster to reveal the labels of a random sample of examples in that cluster, and identify each cluster with its most dominant label. Next, given a new test example, we assign it with the label associated with its nearest cluster. Results (see App. B) showed that this rudimentary classifier is generally not on par with BERT IT:CLUST , yet it can be surprisingly effective where the NMI is high and the labeling budget is small. In our work, we transfer a pretrained model to a new domain with little data. Transfer learning studies how to transfer models across domains. It suggests methods such as pivoting (Ziser and Reichart, 2018), weak supervision (Shnarch et al., 2018), data augmentation (Anaby-Tavor et al., 2020) and adversarial transfer (Cao et al., 2018). In Computer Vision, pretrained models are often learnt by image clustering (Caron et al., 2018). In NLP, however, clustering was mainly used for non-transfer scenarios. Ball (2019) relies on pretrained embeddings to cluster labeled and unlabeled data. Then, they fill the missing labels to augment the training data. Clustering itself was improved by combining small amounts of data (Torres and Vaca, 2019; Wang et al., 2016). Pretrained models improved state-of-the-art in many downstream tasks (Nogueira and Cho, 2019; Ein-Dor et al., 2020) and they are especially needed and useful in low resource and limited labeled data settings (Lacroix et al., 2019; Wang et al., 2020a; Chau et al., 2020). There are many suggestions to improve such models, including larger models (Raf-fel et al., 2019), changes in the pretraining tasks and architecture (Yang et al., 2019), augmenting pretraining (Geva et al., 2020), or improving the transfer itself (Valipour et al., 2019; Wang et al., 2019b; Sun et al., 2019; Xu et al., 2020). Two findings on pretraining support our hypothesis on the intermediate task, namely that classification surpasses MLM. Some pretraining tasks are better than others (Lan et al., 2020; Raffel et al., 2019) and supervised classification as additional pre-training improves performance (Lv et al., 2020; Wang et al., 2019a; Pruksachatkun et al., 2020). All these works aim to improve the performance upon transfer, making it more suitable for any new domain. In contrast, we focus on improvement given the domain. With a transferred model, one can further improve performance with domain-specific information. For example, utilizing metadata (Melamud et al., 2019), training on weakly-supervised data (Raisi and Huang, 2018; Meng et al., 2020) or multitasking on related tasks concurrently (Liu et al., 2019a). Given no domain-specific information, it was suggested to further pretrain on unlabeled data from the domain (Whang et al., 2019; Xu et al., 2019; Sung et al., 2019; Rietzler et al., 2020; Lee et al., 2020; Gururangan et al., 2020). This, however, is sometimes unhelpful or even hurts results (Pan, 2019). Transferring a model and retraining with paucity of labels is often termed few-shot learning. Few shot learning is used for many language-related tasks such as named entity recognition (Wang et al., 2020b), relation classification (Hui et al., 2020), and parsing (Schuster et al., 2019). There have also been suggestions other than fine-tuning the model. Koch (2015) suggests ranking examples' similarity with Siamese networks. Vinyals et al. (2016) rely on memory and attention to find neighboring examples and Snell et al. (2017) search for prototypes to compare to. Ravi and Larochelle (2017) don't define in advance how to compare the examples. Instead, they meta-learn how to train the few shot learner. These works addressed the image classification domain, but they supply general methods 7646 which are used, improved and adapted on language domains (Geng et al., 2019; Yu et al., 2018). In conclusion, separate successful practices foreshadow our findings: Clustering drives pretraining on images; supervised classification aids pre-training; and training on unlabeled domain examples is helpful with MLM. We presented a simple approach for improving pretrained models for text classification. Specifically, we show that inter-training BERT over pseudo-labels generated via unsupervised clustering creates a better starting point for the final fine-tuning over the target task. Our analyses suggest that BERT can leverage these pseudo-labels, namely that there exists a beneficial interplay between the proposed inter-training and the later fine-tuning stage. Our results show that this approach yields a significant boost in accuracy, mainly over topical data and when labeled data is scarce. Note that the method does require the existence of an unlabeled corpus, in the order of several thousand examples. We opted here for a practically oriented approach, which we do not claim to be optimal. Rather, the success of this approach suggests various directions for future work. In particular, several theoretical questions arise, such as what else determines the success of the approach in a given dataset; understanding the potential synergistic effect of BOW-based clustering for inter-training; could more suitable partitions be acquired by exploiting additional embedding space and/or more clustering techniques; co-training (Blum and Mitchell, 1998) methods, and more. On the practical side, while in this work we fixed the inter-training to be over 50 clusters and for a single epoch, future work can improve performance by tuning such hyper-parameters. In addition, one may consider using the labeled data available for fine-tuning as anchors for the intermediate clustering step, which we have not explored here. Another point to consider is the nature of the inter-training task. Here, we examined a multi-class setup where BERT is trained to predict one out of n c cluster labels. Alternatively, one may consider a binary inter-training task, where BERT is trained to determine whether two samples are drawn from the same cluster or not. In principle, inter-training BERT over clustering results may be valuable for additional downstream target tasks, that are similar in spirit to standard text classification. Examples include Key-Point Analysis (Bar-Haim et al., 2020) and Textual Entailment (Dagan et al., 2013). The potential value of our approach in such cases is left for future work. Any use of a language model for classification involves some risk of bias, which stems from the pre-training and training data used to construct the model. Here we aim to improve the language model representations by relying on clustering of data from the target domain. We have no reason to believe this process would introduce bias beyond the potential bias that can occur whenever fine-tuning a model, but this is a potential risk, as we did not verify this directly. We thank Assaf Toledo for providing helpful advice on the clustering implementations.
abstain abstain method method result abstain abstain abstain abstain objective abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method method abstain result objective abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain objective abstain method abstain abstain method abstain objective result abstain abstain abstain objective abstain abstain abstain result other abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method method method abstain abstain abstain abstain other abstain abstain method method method abstain method method method abstain abstain result abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method abstain abstain method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain objective other other other other other other other other other abstain other other objective other other other other other other other other other other other other abstain result result objective result abstain abstain abstain abstain result objective abstain method abstain abstain abstain abstain abstain abstain abstain other