summaries: This paper investigates if using large, pretrained models in an active learning setup helps achieve better performance with lesser data when compared to using randomly sampled data. In order to conduct this investigation the authors study the empirical performance of large pre-trained models on some image datasets and a text dataset. In both cases large pre-trained model is finetuned on a small amount of seed data and then an active learning procedure is used (in this paper the AL procedure is an uncertainty sampling procedure) to collect more data. The datasets are chosen to illustrate several conceptual issues (i) distinguishing causal from spurious correlations (ii) measuring robustness to distribution shifts (iii) role of data imbalance.   Experiments are performed to show that using an active learning procedure indeed helps improve performance using only a small amount of actively labeled training dataset.  The paper is well written and the results are convincing and insightful.  The authors describe interesting empirical observations regarding using uncertainty sampling to select examples to fine-tune models that use pretrained embeddings and provide some hypotheses regarding the reasons for these performance improvements. Specifically, (1) from a methodological perspective, they propose using uncertainty sampling (i.e., least confident selection) to select examples for fine-tuning image/NLP pretrained models and (2) from an empirical perspective, they use Waterbirds/Treeperson/iWildCam2020-WILDS for image classification and Amazon-WILDS for review star prediction based on text and compare with random sampling — noting that these are settings where there is known covariate shift between train/test with semantic meaning to induce interpretable spurious associations (e.g., background in images). The proposed method works overall, especially on the image datasets, and they also dig into the types of examples selected — noting that they align with expected ‘difficult’ examples (depending on the setting). The authors set out to investigate if active learning is an emergent property of pre-training. That is if running active learning with pre-trained models gives better result than using the same models without pre-training. They run several experiments on different text and image datasets first showing that active learning performs better than random sampling on pre-trained models and secondly that pre-trained models perform better than un-pretrained ones for active learning. This paper investigates the active learning performance of pre-trained models vs their non-pre-trained counterparts on both vision and NLP tasks. Specifically, the investigation focuses on datasets with spurious correlation, domain shift, and label imbalance. Empirical results generally show that the pre-trained models with the uncertainty acquisition function performs much better than the random baseline and their un-pre-trained counterparts. 
abstractText: An important barrier to the safe deployment of machine learning systems is the risk of task ambiguity, where multiple behaviors are consistent with the provided examples. We investigate whether pretrained models are better active learners, capable of asking for example labels that disambiguate between the possible tasks a user may be trying to specify. Across a range of image and text datasets with spurious correlations, latent minority groups, or domain shifts, finetuning pretrained models with data acquired through simple uncertainty sampling achieves the same accuracy with up to 6× fewer labels compared to random sampling. Moreover, the examples chosen by these models are preferentially minority classes or informative examples where the spurious feature and class label are decorrelated. Notably, gains from active learning are not seen in unpretrained models, which do not select such examples, suggesting that the ability to actively learn is an emergent property of the pretraining process. Figure 1: Active learning can resolve task ambiguity, which is especially salient in few-shot settings. Here, the provided training data leaves the model unsure of the task: is it to predict the shape or the color of the object? Pretraining enables models to disentangle and weigh various competing features, making them good active learners that can choose disambiguating examples (e.g. the blue square), resolving this task ambiguity.
references.authors: Umang Aggarwal Adrian Popescu Céline Hudelot
references.title: Active learning for imbalanced datasets
references.venue: In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,
references.authors: Hunter Jackson Richard Chen Fabian Both Jörg K.H. Franke Heidi V.N. Küsters-Vandevelde W. Vreuls Peter Bult Bram van Ginneken Jeroen A. van der Laak Geert J.S. Litjens
references.title: From detection of individual metastases to classification of lymph node status at the patient level: The camelyon17 challenge
references.venue: IEEE Transactions on Medical Imaging,
references.authors: Sara Beery Elijah Cole Arvi Gjoka
references.title: The iwildcam 2020 competition
references.venue: dataset. ArXiv,
references.authors: William H Beluch Tim Genewein Andreas Nürnberger Jan M Köhler
references.title: The power of ensembles for active learning in image classification
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: ford Ilya Sutskever Dario Amodei
references.title: Language models are few-shot
references.venue: learners. ArXiv,
references.authors: Mathilde Caron Hugo Touvron Ishan Misra Herv’e J’egou Julien Mairal Piotr Bojanowski Armand Joulin
references.title: Emerging properties in self-supervised vision
references.venue: transformers. ArXiv,
references.authors: Gordon A. Christie Neil Fendley James Wilson Ryan Mukherjee
references.title: Functional map of the world
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Jessica Schrouff Martin G. Seneviratne Shannon Sequeira Harini Suresh Victor Veitch Max Vladymyrov Xuezhi Wang Kellie Webster Steve Yadlowsky Taedong Yun Xiaohua Zhai D. Sculley
references.title: Underspecification presents challenges for credibility in modern machine learning
references.venue: ArXiv, abs/2011.03395,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li K. Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: In CVPR,
references.authors: Shrey Desai Greg Durrett
references.title: Calibration of pre-trained transformers
references.venue: In EMNLP,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova. Bert
references.title: Pre-training of deep bidirectional transformers for language understanding
references.venue: In NAACL-HLT
references.authors: Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly Jakob Uszkoreit Neil Houlsby
references.title: An image is worth 16x16 words: Transformers for image recognition
references.authors: Seyda Ertekin Jian Huang Leon Bottou Lee Giles
references.title: Learning on the border: active learning in imbalanced data classification
references.venue: In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management,
references.authors: Chelsea Finn Kelvin Xu Sergey Levine
references.title: Probabilistic model-agnostic meta-learning
references.venue: In NeurIPS,
references.authors: Yarin Gal Riashat Islam Zoubin Ghahramani
references.title: Deep bayesian active learning with image data
references.venue: In International Conference on Machine Learning,
references.authors: Robert Geirhos Jörn-Henrik Jacobsen Claudio Michaelis Richard S. Zemel Wieland Brendel Matthias Bethge Felix Wichmann
references.title: Shortcut learning in deep neural networks. ArXiv
references.authors: Dan Hendrycks Kevin Gimpel
references.title: A baseline for detecting misclassified and out-of-distribution examples in neural networks
references.authors: Dan Hendrycks Kimin Lee Mantas Mazeika
references.title: Using pre-training can improve model robustness and uncertainty
references.venue: In ICML,
references.authors: T.J. Henighan Jared Kaplan Mor Katz Mark Chen Christopher Hesse Jacob Jackson Heewoo Jun Tom B. Brown Prafulla Dhariwal Scott Gray Chris Hallacy Benjamin Mann Alec Radford Aditya Ramesh Nick Ryder Daniel M. Ziegler John Schulman Dario Amodei Sam McCandlish
references.title: Scaling laws for autoregressive generative modeling
references.venue: ArXiv, abs/2010.14701,
references.authors: Neil Houlsby Ferenc Huszár Zoubin Ghahramani Máté Lengyel
references.title: Bayesian active learning for classification and preference learning
references.venue: arXiv preprint arXiv:1112.5745,
references.authors: Jared Kaplan Sam McCandlish T.J. Henighan Tom B. Brown Benjamin Chess Rewon Child Scott Gray Alec Radford Jeff Wu Dario Amodei
references.title: Scaling laws for neural language models
references.authors: Siddharth Karamcheti Ranjay Krishna Li Fei-Fei Christopher D. Manning
references.title: Mind your outliers! investigating the negative impact of outliers on active learning for visual question
references.venue: answering. ArXiv,
references.authors: Pang Wei Koh Shiori Sagawa Henrik Marklund Sang Michael Xie Marvin Zhang Akshay Balsubramani Wei hua Hu Michihiro Yasunaga Richard L. Phillips Sara Beery Jure Leskovec Anshul Kundaje Emma Pierson Sergey Levine Chelsea Finn Percy Liang
references.title: Wilds: A benchmark of in-the-wild distribution shifts
references.venue: In ICML,
references.authors: Alexander Kolesnikov Lucas Beyer Xiaohua Zhai Joan Puigcerver Jessica Yung Sylvain Gelly Neil Houlsby
references.title: Big transfer (bit): General visual representation learning
references.venue: In ECCV,
references.authors: Ranjay Krishna Yuke Zhu Oliver Groth Justin Johnson Kenji Hata Joshua Kravitz Stephanie Chen Yannis Kalantidis Li-Jia Li David A. Shamma Michael S. Bernstein Li Fei-Fei
references.title: Visual genome: Connecting language and vision using crowdsourced dense image annotations
references.venue: International Journal of Computer Vision,
references.authors: Alex Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report,
references.authors: David D Lewis Jason Catlett
references.title: Heterogeneous uncertainty sampling for supervised learning
references.venue: In Machine learning proceedings
references.authors: Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer Veselin Stoyanov
references.title: Roberta: A robustly optimized bert pretraining approach
references.authors: David Lowell Zachary Chase Lipton Byron C. Wallace
references.title: Practical obstacles to deploying active learning
references.venue: In EMNLP,
references.authors: Katerina Margatina Loı̈c Barrault Nikolaos Aletras
references.title: Bayesian active learning with pretrained language models
references.venue: ArXiv, abs/2104.08320,
references.authors: Stephen Mussmann Robin Jia Percy Liang
references.title: On the importance of adaptive data collection for extremely imbalanced pairwise tasks
references.venue: In FINDINGS,
references.authors: Vaishnavh Nagarajan Anders Johan Andreassen Behnam Neyshabur
references.title: Understanding the failure modes of out-of-distribution generalization
references.venue: ArXiv, abs/2010.15775,
references.authors: Jianmo Ni Jiacheng Li Julian McAuley
references.title: Justifying recommendations using distantly-labeled reviews and fine-grained aspects
references.venue: In EMNLP,
references.authors: Ethan Perez Douwe Kiela Kyunghyun Cho
references.title: True few-shot learning with language models
references.venue: ArXiv, abs/2105.11447,
references.authors: Alec Radford Jong Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever
references.title: Learning transferable visual models from natural language supervision
references.authors: Pranav Rajpurkar Robin Jia Percy Liang
references.title: Know what you don’t know: Unanswerable questions for squad
references.venue: In ACL,
references.authors: Tal Reiss Niv Cohen Liron Bergman Yedid Hoshen
references.title: Panda: Adapting pretrained features for anomaly detection and segmentation
references.venue: In CVPR,
references.authors: Shiori Sagawa Pang Wei Koh Tatsunori B. Hashimoto Percy Liang
references.title: Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization
references.authors: Shiori Sagawa Aditi Raghunathan Pang Wei Koh Percy Liang
references.title: An investigation of why overparameterization exacerbates spurious correlations
references.venue: ArXiv, abs/2005.04345,
references.authors: Marten Scheffer Stephen R. Carpenter Jonathan A. Foley Carl Folke Brian H. Walker
references.title: Catastrophic shifts in ecosystems
references.venue: Nature, 413:591–596,
references.authors: Burr Settles
references.title: Active learning literature survey
references.venue: Technical report, University of Wisconsin-Madison Department of Computer Sciences,
references.authors: Burr Settles Mark Craven
references.title: An analysis of active learning strategies for sequence labeling tasks
references.venue: In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing,
references.authors: Artem Shelmanov Dmitri Puzyrev Lyubov Kupriyanova Denis I. Belyakov Daniil Larionov Nikita Khromov Olga Kozlova E. Artemova Dmitry V. Dylov Alexander Panchenko
references.title: Active learning for sequence tagging with deep pre-trained models and bayesian uncertainty estimates
references.venue: In EACL,
references.authors: Alex Tamkin Miles Brundage Jack Clark Deep Ganguli
references.title: Understanding the capabilities, limitations, and societal impact of large language
references.venue: models. ArXiv,
references.authors: Jessica Taylor Eliezer Yudkowsky Patrick LaVictoire Andrew Critch
references.title: Alignment for advanced machine learning
references.authors: Laurens van der Maaten Geoffrey E. Hinton
references.title: Visualizing data using t-sne
references.venue: Journal of Machine Learning Research,
references.authors: Alex Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R Bowman
references.title: Glue: A multi-task benchmark and analysis platform for natural language understanding
references.venue: In International Conference on Learning Representations,
references.authors: Mike Wu Noah D. Goodman
references.title: A simple framework for uncertainty in contrastive learning
references.venue: ArXiv, abs/2010.02038,
references.authors: Michelle Yuan Hsuan-Tien Lin Jordan L. Boyd-Graber
references.title: Cold-start active learning through self-supervised language modeling
references.venue: In EMNLP,
sections.heading: 1 INTRODUCTION
sections.text: Modern pretrained models can be adapted to new tasks with remarkably little data, enabling downstream applications for tasks with only tens or hundreds of examples (Brown et al., 2020; Radford et al., 2021). However, an important but neglected challenge that is especially salient in few-shot settings is task ambiguity, when the desired behavior is not uniquely specified by the provided examples. Task ambiguity can manifest in different ways: For example, the class-relevant features of an input (e.g., the shape of an object in Figure 1) may be spuriously correlated with other features predictive of the training labels (e.g., the color of the object), making the desired task unclear. In addition, task ambiguity may arise from an underdiverse training set, causing models to be unsure of the desired behavior for minority groups or during distribution shifts that occur at test-time.
  We consider whether the task ambiguity problem can be addressed through active learning, where models select informative examples for users to label. In principle, active learning allows models themselves to assist in resolving task ambiguity by identifying examples whose labels would be
  informative; for example, in fig. 1, asking for the label of the blue square helps determine that the desired task is to predict the object shape not the object color. Since it may be challenging in general for users to intuit possible sources of task ambiguity, much less how to address them with example selection, an automated active learning approach may be desirable.
  In traditional settings with smaller, unpretrained models, several challenges often prevent active learning from seeing success in practice, including label noise, unlearnable examples, and a lack of generalizability across models and tasks (Lowell et al., 2019; Karamcheti et al., 2021). However, using pretrained models in few-shot or low-data settings has several key differences from traditional ML pipelines which may make them particularly well-suited for active learning. First, the effect of an individual data point (and the difference between an informative vs. an uninformative one) is magnified, compared to the thousands or millions of examples typically acquired through standard active learning pipelines. Second, pretrained models excel at learning high-level representations of inputs, which better surface relevant features and may encourage active learning to select examples that disambiguate between these high-level features (e.g. shape and color).
  We consider the use of active learning on a range of spuriously correlated, imbalanced, or domain shifted datasets where task ambiguity is salient. We compare an active learning approach with a random-sampling baseline, and compare the difference in performance with and without the use of pretrained models. Our contributions are:
  1. Identifying and motivating task ambiguity as a unified concern across a range of real-world datasets, and an especially salient problem for low-data settings.
  2. Proposing active learning as a potential solution, along with a simple yet practical recipe that does not require adjustment for new datasets or tuning on validation data.
  3. Showing that active learning can enable large gains (up to 6× reduction in data points, +12% absolute gain for the same labeling budget) and presenting scaling trends demonstrating that this is an emergent property of pretraining.
sections.heading: 2 METHOD
sections.text: We study the pool-based active learning setup common in the literature (Settles, 2009), where we have a (possibly pretrained) modelM, a small seed set of training data S = {(xi, yi)}, and a larger pool of unlabeled data P = {xi}. The active learning procedure proceeds as follows: first, finetune M on S until convergence; then, select points xi fromP that are deemed most informative according to an acquisition function a(x;M), obtaining the corresponding labels yi, until some budget k of data points is exhausted. This newly labeled batch B = {(xi, yi)} is then added to the existing data S, and the model is retrained on S for the next acquisition step. This process is repeated for a fixed number of acquisition steps.
  For our acquisition function, we adopt the classic uncertainty sampling approach to active learning, in particular the least confidence heuristic, where we acquire points for which our model is least confident in its predicted label (Settles, 2009). Specifically, treating the outputs of the modelM as a probability distribution1 over possible labels p(y | x;M), we define the acquisition function to be
  a(x;M) = −max i p(yi | x;M) (1)
  Although we could use more complex measures such as the entropy over labels, the least confidence heuristic has shown to be simple and effective in a variety of settings (Settles, 2009; Hendrycks & Gimpel, 2017; Mussmann et al., 2020), and we similarly find good results here (we refer to least confidence sampling as “uncertainty sampling” except in Section 4.3 where we explore other uncertainty-based acquisition functions).
  On top of this standard active learning pipeline, we propose the following change to improve the practical applicability of pretrained models in data-scarce settings:
  1While in general there is no guarantee that this probability distribution will be well-calibrated, recent work has found that pretraining improves model calibration across a variety of settings, including on out-of-domain data (Desai & Durrett, 2020; Hendrycks et al., 2019).
  Removing the need for a separate validation set The active learning cycle begins by finetuning the pretrained model on the seed set. If the size of the seed set is large enough, the seed set may be partitioned into a training set and a validation set, and early stopping may be performed on the validation set. However, in few-shot settings, labeling costs may be high, and the seed set may be too small to meaningfully partition. This has led to a variety of ad-hoc solutions, e.g. finetuning for an arbitrary constant number of gradient steps depending on the task (Kolesnikov et al., 2020).
  Instead of an arbitrary fixed number of finetuning steps, we propose an alternative method to terminate finetuning in the absence of a validation set. Specifically, we found that a simple but effective heuristic was to stop finetuning when the training loss decreases to 0.1% of the original training loss at the start of finetuning. In our experiments, this heuristic performed as well as early stopping on an actual validation set (see Appendix D for more details). By using a standardized recipe across tasks and removing the need for a separate validation set, our active learning pipeline is more robust to the real-world difficulties of deploying active learning in data-scarce settings where use of a validation set is impractical (Lowell et al., 2019; Perez et al., 2021), although further work is needed to capture the full extent of this recipe’s generalizability.
sections.heading: 3 DATASETS
sections.text: We consider a variety of datasets where task ambiguity is especially likely when only a few examples are provided, especially due to spurious correlations, latent minority classes, and domain shifts. These datasets provide an empirical testbed for the ability of pretrained models to choose disambiguating examples using active learning.
sections.heading: 3.1 DISTINGUISHING CAUSAL FROM SPURIOUS FEATURES
sections.text: Spurious correlations arise when multiple features are predictive of the label in a training dataset, yet it is ambiguous which ones are causally linked to the task label (Geirhos et al., 2020). We consider two such datasets, and see whether active learning can choose the disambiguating examples where the spurious features are not copresent with the causal features:
  Waterbirds The Waterbirds dataset (Sagawa et al., 2019) consists of photographs of landbirds or waterbirds digitally edited onto land or water backgrounds. The task is to classify whether the bird is a landbird or a waterbird. In the train set, 77% of the pictures feature landbirds and 23% waterbirds. 95% of both landbirds and waterbirds appear on land and water backgrounds, respectively. In the validation and test sets, this percentage is decreased to 50%, instead. Thus, the image background is a spurious feature the model may come to rely on when making the prediction.
  Treeperson As the Waterbirds dataset was synthetically generated, we also consider a dataset where we perform classification over real, unedited images with spuriously correlated objects. We use the object annotations in Visual Genome (Krishna et al., 2016) to create a new dataset of 8,638 images called Treeperson, for which the task is to predict whether a person is in a given image. While 50% of the images contain a person in this dataset, each image also contains either a tree or a building, and the presence of these objects is spuriously correlated with the presence of people. At train time, 90% of training images with people contain a building, while 90% of training images without people contain a tree. Thus, a model may be incentivized to form representations that classify according to the presence of trees and buildings, rather than the presence of the actual causal variable of interest (people). These values are changed to 50% at test time, removing this correlation to evaluate how well the model learned the actual feature of interest. For more details on this dataset, see Appendix C.
sections.heading: 3.2 MEASURING ROBUSTNESS TO DISTRIBUTION SHIFT
sections.text: Distribution shifts occur when algorithms are evaluated on different data distributions than the ones they were trained on. Examples include changing the location or time of day that photos were taken, or changing the topic or author of a particular textual source. These shifts can reduce performance, and we consider whether active learning can help choose diverse, informative examples that clarify how the model should behave over a range of natural distribution shifts.
  iWildCam2020-WILDS This dataset considers the task of species classification from a database of photos taken from wildlife camera traps (Beery et al., 2020; Koh et al., 2021). The dataset is unbalanced, with most images containing no animal, and the distribution of camera locations and species changes between the in domain (ID) and out-of-domain (OOD) subsets.
  Amazon-WILDS This dataset considers the task of predicting the number of stars associated with the text of a given Amazon review (Ni et al., 2019; Koh et al., 2021). The reviewers are different in the training set versus the test set, and the task is to perform as well as possible on this set of new reviewers. In addition to number of stars, we also consider model performance stratified by different product types, which highlights minority subgroups whose categorization is not visible to the model.
sections.heading: 4 EXPERIMENTS
sections.heading: 4.1 MODELS AND TRAINING
sections.text: Vision For computer vision datasets, we finetune BiT (Kolesnikov et al., 2020), a recentlyproposed family of vision models which have achieved state-of-the-art performance on several vision tasks. We primarily consider the BiT-M-R50x1 model, pretrained on ImageNet-21k (Deng et al., 2009). To explore the effectiveness of larger architectures and pretraining sources, in Section 5.2 we also consider performance achieved by the same-size BiT-S-R50x1, trained on ImageNet1k, and the deeper BiT-M-R101x1 model, also trained on ImageNet-21k. These models have been shown to have emergent few-shot learning abilities, where strong classifiers for new tasks can be obtained by simply finetuning on tens or hundreds of examples with typical gradient descent techniques (rather than meta-learning techniques, for example).
  Text For the text dataset (Amazon), we use RoBERTa-Large (Liu et al., 2019), another pretrained model with similar properties as BiT, and a representative of the BERT (Devlin et al., 2019) family of models which together have obtained state-of-the-art scores on modern NLP benchmarks (Wang et al., 2018).
  Other details, including hyperparameters and seed set/acquisition sizes are deferred to Appendix B.
  Random acquisition baseline As a running baseline, we compare to the same model finetuned with a random acquisition function (equivalent to not doing active learning). That is, a(x;M) = rand(0, 1), so we simply sample a random batch of data from the pool at each acquisition step.
  Comparison with unpretrained models To examine whether effective active learning is an emergent property of the pretraining process, we also compare to the performance observed when applying active learning to a randomly initialized, instead of pretrained, BiT-M-R50x1.
sections.heading: 4.2 ACCURACY PER ACQUISITION
sections.text: For a general measure of success, we plot the accuracy of active learning versus random sampling on the validation datasets as a function of the number of samples acquired during training.
  Waterbirds Waterbirds is evaluated on a balanced dataset where the foreground and background are not correlated. In this setting, uncertainty sampling achieves a +12% improvement in average validation accuracy over random sampling (Figure 2a). This comes primarily from a +31% average increase across the landbird-on-water and waterbird-on-land images (i.e. those without the spurious correlation; Figure 2b). Uncertainty sampling required 5.8x fewer labels than random sampling to achieve random sampling’s final accuracy.
  Amazon In the Amazon dataset, we also see gains from active learning in the OOD setting, including +1% on average across reviewers, and +2.5% on the worst 10th percentile (Figure 2g). This result suggests that our active learning recipe may be of use outside of BiT or computer vision settings more broadly. The confidence not overlapping by the end of training is evidence that
  while the difference between uncertainty and random sampling is not large, it is statistically significant. Uncertainty sampling required 1.3x fewer labels than random sampling to achieve random sampling’s final accuracy.
  iWildCam With the iWildCam dataset, uncertainty sampling achieved a +8% improvement upon random sampling. Uncertainty sampling also required 1.9x fewer labels than random sampling to achieve random sampling’s final accuracy (Figure 2g).
  Treeperson In the Treeperson dataset, uncertainty sampling is +2% improved over random sampling by the end of training (Figure 2c). Uncertainty sampling required 2.5x fewer labels than random sampling to achieve random sampling’s final accuracy.
sections.heading: 4.3 ADDITIONAL ACTIVE LEARNING METHODS
sections.text: We consider two additional active learning methods in addition to least confidence sampling: 1) entropy sampling, which chooses the example that maximizes the entropy of the model’s predictive distribution, and 2) margin sampling, which chooses the example with the smallest difference between the first and second most probable classes (Scheffer et al., 2001; Settles, 2009). We run
  experiments with all methods on the 182-class iWildCam dataset2. All methods significantly outperform random sampling (Figure 3).
sections.heading: 4.4 ADDITIONAL PRETRAINED VISION MODEL
sections.text: We broaden our coverage of computer vision models to include vision transformers (Dosovitskiy et al., 2021), the other major architecture family currently in use. We train ViT-16/B,3 on Treeperson, observing smaller gains for AL than the BiT model (Figure 8). This perhaps reflects the fact that the vision transformer was pretrained on ImageNet-21k for far fewer epochs then BiT (9 vs 70), and is corroborated by much lower oversampling of minority classes than BiT (Figure 9).
sections.heading: 5 ANALYSIS
sections.heading: 5.1 ACTIVE LEARNING SELECTS EXAMPLES THAT RESOLVE TASK AMBIGUITY
sections.text: Overall, we attribute improved performance to pretrained models’ ability to identify and preferentially sample disambiguating examples and latent minority examples that resolve task ambiguity.
  Waterbirds Figure 4a depicts the rate at which uncertainty sampling is acquiring examples of each subgroup compared to the expected rate at which random sampling would acquire examples from those same subgroup. Examples where the bird and background are mismatched are heavily oversampled. We emphasize that these minority examples are not simply members of the minority class (waterbirds). If active learning were simply upsampling minority classes, then all landbird images would be downsampled and all waterbird images would be upsampled. However, this is not the case—instead, the model identifies and preferentially upsamples informative examples where the spurious feature (background) and the causal feature (bird type) disagree.
  Treeperson For Treeperson we see the same pattern as in Waterbirds: the model identifies and upsamples examples where only one of the spurious or causal features is present (Figure 4b).
  Amazon We also see similar behavior in the Amazon dataset, indicating our method’s applicability to multiple modalities and pretrained models. Not only does the model upsample lower star ratings, which are less common, it is also able to upsample rarer product categories—an unseen attribute.
sections.heading: 5.2 PRETRAINING IS CRUCIAL FOR ACTIVE LEARNING TO WORK
sections.text: We also consider the role of the pretraining dataset and model size. We conduct active learning experiments with 3 pretrained models: BiT-S-R50x1, BiT-M-R50x1, BiT-M-R101x1, and their corresponding non-pretrained versions. For all pretrained models (except BiT-M R-101 for Treeperson)
  2Note that least confidence, entropy, and margin sampling are identical in the case of binary classification. 3https://huggingface.co/google/vit-base-patch16-224
  uncertainty acquisition outperformed random acquisition (Figure 7a, Figure 7b, Figure 7c). Importantly, we did not see any gains from AL on the unpretrained models, even when exploring a range of different hyperparameter configurations.
  Scaling trend Interestingly, the BiT-S-R50x1 model, which was pretrained on a smaller dataset than the BiT-M models (ImageNet-1k vs ImageNet-21k) achieves less of a boost from active learning on iWildCam vs the other two models. This suggests that the gains from active learning may continue to grow as pretained models are trained for longer on more data. However, we did not see a difference between BiT-M-50x1 and BiT-M-101x1, which were trained on the same dataset but have different numbers of parameters. We encourage future work that more thoroughly examines these behaviors and scaling laws across a range of models and datasets.
  Impact of pretraining on acquisition patters Pretrained models also acquire disambiguating subgroups much more efficiently than their unpretrained counterparts. See Appendix G for additional figures and results illustrating this point.
sections.heading: 5.3 PRETRAINING PROVIDES A BETTER FEATURE SPACE FOR ACTIVE LEARNING
sections.text: Why does pretraining improve AL for resolving task ambiguity? We investigate the hypothesis that pretrained representations enable AL models to select objects based on higher-level features (e.g. object foreground and background) through various analyses on Waterbirds.
  Linear classifier analysis We investigate the presence of these high-level features by training linear classifiers on the second to last layer of BiT models. The classifiers are trained to predict each image’s bird type and background type (4 classes, each comprising 25% of the data). As shown in Figure 6, these classes are more linearly separable in pretrained models both before any finetuning, as well as after finetuning on a seed set and then acquired examples from the original 2- class Waterbirds dataset. This demonstrates that pretrained models can better identify both the causal and spurious features in the data, potentially explaining why they can choose them preferentially.
  t-SNE visualization We visualize the second-to-last layer embeddings of BiT (without any finetuning) using t-SNE (van der Maaten & Hinton, 2008), shown in Figure 13. We find that examples selected by uncertainty sampling fall closer to the decision boundary for the pretrained model than for the unpretrained model. We also see that the pretrained model exhibits far more separation by class, suggesting it is better able to identify useful features in the input. These conclusions help further characterize how pretraining provides a feature space more amenable to active learning.
sections.heading: 5.4 PINPOINTING THE EFFECT OF TASK AMBIGUITY WITH A DOSE-RESPONSE EXPERIMENT
sections.text: To examine the impact of task ambiguity from the perspective of the unlabeled dataset, we construct variants of the waterbirds dataset where the percentage of mismatched examples range from 95% to 50%. 4 We then proceed with AL and report results in Figure 10.5 We find significantly larger gains on versions of the dataset with smaller fraction of mismatched examples (gains average 4% for 50–70% shift, and 10% for 75–95% shift) keeping. This provides stronger evidence that active learning is actually helping the model identify the true task.
sections.heading: 5.5 FAILURE CASES
sections.text: To assess how similar the datasets needed to be to the training distribution of BiT, we also perform preliminary experiments on Camelyon17-WILDS (Bándi et al., 2019; Koh et al., 2021), which considers tumor identification from tissue patches, and FMoW-WILDS (Christie et al., 2018; Koh et al., 2021), which considers land-use classification from satellite images. Active learning performs comparably or worse than random sampling on these datasets, even when using a pretrained BiT model, suggesting that generalization to domains far from the training distribution (ImageNet-21k) may be challenging. However, Camelyon17-WILDS is also known to exhibit high variance across seeds,6 which may also be a contributing factor.
sections.heading: 6 RELATED WORK
sections.text: Task ambiguity and specification Several works address ambiguity or poor specification in machine learning problems. Taylor et al. (2020) describe the problem of “inductive ambiguity identification,” and describe active learning as a promising potential solution that has failed to see practical success. D’Amour et al. (2020) describe the problem of underspecification, where high variance, instability, and poor model performance result from training overparameterized models on small amounts of data. Geirhos et al. (2020) describes how task ambiguity can arise when both desirable and undesirable features are predictive of the training labels, a problem which several works seek to better characterize and address (Nagarajan et al., 2021; Sagawa et al., 2019; 2020). Finally, Finn et al. (2018) address task ambiguity in few-shot settings via a probabilistic meta-learning algorithm,
  4We construct these datasets using the code at https://github.com/kohpangwei/group_DRO. 5Results are preliminary and will be rerun with more seeds. 6https://wilds.stanford.edu/get_started/
  and perform an active learning experiment in a 1D regression setting. We build on these works by demonstrating that simple uncertainty sampling with pretrained models can be an effective approach to the task ambiguity problem across a wide variety of high-dimensional classification settings.
  Uncertainty and distribution shift In the face of these challenges, several works have tried to quantify how much pretrained models know about problems or their own uncertainty about them. Rajpurkar et al. (2018) propose a question answering dataset with unanswerable questions, where a model must abstain rather than proceeding with an answer. Pretraining can also improve the calibration of model uncertainty (Hendrycks et al., 2019) and pretrained features can be used for outof-distribution detection (Reiss et al., 2021; Wu & Goodman, 2020)—observations that align with our findings that uncertainty sampling can identify minority subgroups in datasets. Furthermore, our observation that upsampling latent minority groups results in better performance aligns well (Sagawa et al., 2020), which found that simply upweighting minority groups performed less well than increasing the relative fraction of minority group examples in the distribution. Importantly, however, our active learning setup does not require these groups to be known in advance.
  Active learning and example selection Active learning (AL) (Lewis & Catlett, 1994; Settles & Craven, 2008; Settles, 2009; Houlsby et al., 2011; Aggarwal et al., 2020) is a well-studied field that investigates how machine learning algorithms might automatically select helpful additional data points to maximize their performance. Such strategies are especially helpful in imbalanced settings (Ertekin et al., 2007; Mussmann et al., 2020) and has been fruitfully applied to deep models (Gal et al., 2017; Beluch et al., 2018), including pretrained models (Yuan et al., 2020; Margatina et al., 2021; Shelmanov et al., 2021). Past work has also considered AL for few-shot learning (Woodward & Finn, 2017). We extend these works by specifically considering AL for resolving task ambiguity, showing that AL can induce something closer to the true task desired by users by selecting examples from unlabeled minority subgroups, as well as examples that disambiguate causal from spurious features. In contrast to prior work, we also demonstrate the causal effect of pretraining on AL by running controlled studies that compare to both random sampling and unpretrained models.
  Pretrained models and their emergent properties Our work contributes to a broader literature on how pretraining enables new kinds of model capabilities (Bommasani et al., 2021; Tamkin et al., 2021), especially when deployed in few-shot settings. For example, (Brown et al., 2020) identify the phenomenon of in-context learning, where tasks can be specified for models through a language modeling prompt, while Caron et al. (2021) discover that a self-supervised vision model implicitly learns high-quality segmentation maps visible through attention scores. Kaplan et al. (2020); Henighan et al. (2020) conduct scaling laws experiments which chart how capabilities emerge with scale. We identify a new model capability that emerges through pretraining: the capacity to actively learn and resolve task ambiguity in conceptually abstract hypothesis spaces.
sections.heading: 7 DISCUSSION AND LIMITATIONS
sections.text: We argue that pretrained models are good active learners, capable of identifying informative examples across a diverse range of settings where task ambiguity makes choosing examples challenging. Despite past work largely considering each separate problems, we find that active learning helps in cases where data is spuriously correlated, undergoes domain shift, or contains unlabeled subpopulations. These behaviors emerge most clearly as a result of large-scale pretraining, suggesting that active learning may be an underappreciated tool for increasing the reliability of systems in realworld settings. Importantly, pretraining appears necessary for these behaviors to manifest—and in some cases, models pretrained for longer on larger datasets appear to actively learn better.
  Our method does suffer from a number of limitations. First, it requires a human in the loop for data acquisition, which significantly increases the time required to train a model compared to random sampling—a cost which must be weighed against potential benefits. Second, our method requires the labeling method to be relatively free of noise—this may be acceptable if annotators are domain experts or are well-trained, but may also increase the cost per acquired example.
  Finally, we note the opportunity for much exciting future work, including deeper investigation of task ambiguity in real-world settings and better understanding how pretraining shapes active learning as models continue to scale.
sections.heading: A CODE RELEASE
sections.text: Our code and training scripts will be released at https://redacted.
sections.heading: B ADDITIONAL EXPERIMENTAL DETAILS
sections.text: Here we provide some additional experimental details.
  All BiT runs use the same default settings specified in the BiT paper (Kolesnikov et al., 2020) and the BiT GitHub repo: https://github.com/google-research/big_transfer
  Settings that are constant across all BiT runs:
  1. Optimizer - SGD with momentum 0.9 2. Learning Rate - Base learning rate 0.003. Linear warm up to this rate, then staircase decay.
  Exact schedule depends on dataset size, but for our few shot setting, this means: (1) linear warm up in the first 100 steps to 0.003, then (2) decay 10 fold every 100 steps; (3) after 500 steps, stop and move on to the next acquisition round.
  3. Data Augmentation - Random cropping and flipping. See our repo/utils/datasets/load for details. Also available in BiT repo.
  4. Batch Size - 32 when training, split into gradient accumulation microbatches of size 8. 5. Early stopping condition - When the training loss reaches below 0.001 times the original
  training loss.
  Settings that varied between image datasets:
  1. Size of Initial Seed Set - 40 for Waterbirds, 40 for Treeperson, 182 for iWildCam, 20 for CIFAR10, 8 for Camelyon17
  2. Size of Training Pool - Entire training set for Waterbirds, Treeperson, Camelyon17. A random subset of 5000 examples and 12000 examples (re-drawn for each acquisition) from the entire training set of CIFAR10 and iWildCam, respectively.
  3. Number of Acquired Examples - 320 for Waterbirds, 320 for Treeperson, 1456 for iWildCam, 180 for CIFAR10, 128 for Camelyon17
  4. Number of Examples Acquired Each Acquisition - 5 for Waterbirds, 5 for Treeperson, 20 for iWildCam, 2 for CIFAR10, 1 for Camelyon17.
  Settings used for Amazon + RoBERTa-Large runs:
  1. Optimizer - AdamW with default hyperparameters (β1 = 0.99, β2 = 0.999, weight decay = 0.1).
  2. Learning Rate - LR = 1e-6. 3. Batch Size - 2 (due to memory considerations).
  Early Stopping Condition When the training loss reaches below 0.001 times the original training loss or when 4000 gradient steps have been taken (whichever comes first).
  4. Size of Initial Seed Set - 10. 5. Size of Training Pool - A random subset of 2000 examples (re-drawn for each acquisition)
  from the entire training set.
  6. Number of Acquired Examples - 90. 7. Number of Examples Acquired Each Acquisition - 2.
sections.heading: C TREEPERSON DATASET
sections.text: The Treeperson dataset is composed of images from Visual Genome (Krishna et al., 2016) with different compositions of detected objects.
  Training set composition by subclass:
  • Person and Building: 3700
  • Person and Tree: 370
  • No Person and Building: 370
  • No Person and Tree: 3700
  The validation set contains 498 examples of each subclass.
  The following annotated objects were used to form the different subclasses:
  • Person: person, people, man, men, woman, women
  • Building: building, buildings
  • Tree: tree, trees, leaf, leaves, grass
  The training set and validation set were drawn randomly from qualifying images in Visual Genome’s training set and validation set, respectively.
sections.heading: D EARLY STOPPING CONDITION
sections.text: At the outset of this work, we explored if we could identify a heuristic for stopping training when there was no validation set present. We compared how the BiT model would perform if it stopped the finetuning step based off of when the validation accuracy plateaued versus when the training loss decayed to be 0.001 of its original value.
  We ran a smaller experiment than the standard Waterbirds parameters we described in Appendix B. Namely, our seed set was of size 32, we acquired 64 examples on top of that, and we acquired one example at a time. We also defined the validation accuracy as having plateaued the fifth time it did not increase.
  We ran twelve paired experiments where for twelve different randomized seed sets, we performed the Waterbirds experiment four times - with random sampling and stopping when validation accuracy plateaued, with random sampling and stopping when training loss decayed, with uncertainty
  sampling and stopping when validation accuracy plateaued, and with uncertainty sampling and stopping when training loss decayed.
  We found that these experiments achieved:
  1. Random sampling + Stop from validation accuracy: average accuracy = 69.11%, average duration = 3.5 hours
  2. Random sampling + Stop from training loss: average accuracy = 69.93%, average duration = 38 minutes
  3. Uncertainty sampling + Stop from validation accuracy: average accuracy = 83.64%, average duration = 4.5 hours
  4. Uncertainty sampling + Stop from training loss: average accuracy = 85.62%, average duration = 74 minutes
  Thus, we concluded that the by stopping our finetuning step just by waiting for the training loss to decay to 0.001 of its original values, we could achieve comparable if not better accuracies, spend a fraction of the time, and remove the need for a labeled validation set.
  E VISION TRANSFORMER MODEL
  Results for Treeperson when using ViT-B/16 are presented in Figure 8. While active learning improves upon random sampling, the gains are not as large as for BiT, perhaps because ViT was pretrained for almost 8x fewer epochs.
sections.heading: F DOSE RESPONSE
sections.text: Results for the Waterbirds dose-response experiment are presented in Figure 10. Even when the training and test distributions are the same, active learning allows the pretrained BiT model to outperform random sampling. However, as the train and test distribution diverge, the benefit of active learning increases, indicating the importance.
sections.heading: G EFFECT OF MODEL SCALING AND PRETRAINING ON ACQUISITION
sections.text: For the Waterbirds model scaling experiment, we track the examples each model acquires. These are presented in Figure 11. The acquisition patterns of all the pretrained models look fairly similar— they upsample both minority (landbird/water-background, waterbird/land-background) subclasses. However, the non-pretrained model is unable to capture that distinction, and is only able to upsample images with a water background, resulting in worse performance. A summary of final class acquisition ratios is available in Figure 12.
  H VISUALIZATION OF IMAGE EMBEDDINGS OF PRETRAINED BIT-M
  As seen in Figure 13b, when performing t-SNE on the Waterbirds image embeddings, we find that the t-SNE of the pretrained model has much more structure than that of the unpretrained model. In the unpretrained model, the image embeddings of the landbird/land background, landbird/water background, and waterbird/water background classes are distributed uniformly about the center of the t-SNE projection. However, we observe much more structure in the pretrained model’s t-SNE. In particular, the most noticible difference is that the landbird/water background and waterbird/land background classes can be found near the other landbird and waterbird images, respectively. This shows that even without any finetuning, the pretrained model already has learned features to help process the type of bird that appears in the image.
  In the t-SNE plots, the black stars represent the embeddings of the first 10 images that were acquired by each model. For the pretrained model, the acquired examples very clearly fall along the waterbird-landbird boundary in the projected feature space. However, in the unpretrained model, the acquired examples are distributed randomly. This suggests that the features acquired during pretraining enables them to select examples that fall near the decision boundary of a new unseen task.
sections.heading: I ROLE OF LABEL IMBALANCE
sections.text: We attempt to isolate the role of label imbalance by investigating CIFAR-10 (Krizhevsky, 2009), a widely-used balanced dataset where task ambiguity is not known to be a common problem. We train uncertainty- and random-sampling models on both the original CIFAR-10 dataset, as well as an imbalanceed variant where half of the classes have 90% of their examples removed. As shown in Figure 14, random sampling slightly outperforms uncertainty sampling on the original CIFAR-10 dataset, but uncertainty sampling performs better in unbalanced settings, both when the validation dataset is imbalanced as well as when it is balanced.