summaries: The paper studies methods to select source models from a diverse set, considering both the quality of the proxy for downstream performance, and the speed of computing the proxy. The paper introduces a new benchmark for evaluating the ability to choose from diverse source models, studies a number of existing methods, and highlights some shortcomings. It then proposes some tweaks to improve these. Finally, the paper proposes a new proxy PARC, which performs well on their benchmark (also using the aforementioned tricks), and also on an extended evaluation on crowd-sourced models. This paper introduced a unified benchmark to test the performance of existing transferability estimation algorithms. The main contribution is firstly its unified benchmark, and then, its experiment models and datasets are large, which show a fair comparison for all methods. Finally, they modified an existing algorithm to propose a new one, which achieves better Person correlation score over others. The paper studies model selection for transfer learning: given a number of source or upstream models, and a new downstream or target data, how can we choose the best model to finetune? The paper offers two main contributions. First, it proposes a specific benchmark with source models, target tasks, and an evaluation metric to compare different approaches. Second, after testing on a number of methods from the recent literature, it identifies some algorithmic aspects that seem to hurt these methods and proposes an alternative approach called Pairwise Annotation Representation Comparison (PARC) which addresses those. Experiments suggest PARC outperforms the original algorithms. The authors introduce the “scalable diverse model selection” task and introduce several tools and benchmarks for evaluating model selection methods in this setting. They show that current transferability and model selection methods fail to beat simple baselines in this new setting. They analyze the reason for this case and provide techniques to improve performance. They develop PARC, a method that outperforms other methods on diverse model selection. The paper proposes a model selection method wherein best experts are selected from a model zoo to fine-tune on the target task. The proposed method "Pairwise Annotation Representation Comparison" is an improvement over RSA [8], instead of using a small dnn trained on target task (or probe network) the authors use spearman correlation between RDM of features and label as model selection score (called PARC score). The proposed method is compared to many baselines NCE, LEEP, RSA, DDS etc. and shows better correlation to fine-tuning accuracy.
abstractText: With the preponderance of pretrained deep learning models available off-the-shelf from model banks today, finding the best weights to fine-tune to your use-case can be a daunting task. Several methods have recently been proposed to find good models for transfer learning, but they either don’t scale well to large model banks or don’t perform well on the diversity of off-the-shelf models. Ideally the question we want to answer is, “given some data and a source model, can you quickly predict the model’s accuracy after fine-tuning?” In this paper, we formalize this setting as “Scalable Diverse Model Selection” and propose several benchmarks for evaluating on this task. We find that existing model selection and transferability estimation methods perform poorly here and analyze why this is the case. We then introduce simple techniques to improve the performance and speed of these algorithms. Finally, we iterate on existing methods to create PARC, which outperforms all other methods on diverse model selection. We have released the benchmarks and method code† in hope to inspire future work in model selection for accessible transfer learning.
authors.name: Daniel Bolya
authors.name: Rohit Mittapalli
references.authors: Alessandro Achille Michael Lam Rahul Tewari Avinash Ravichandran Subhransu Maji Charless C Fowlkes Stefano Soatto Pietro Perona
references.title: Task2vec: Task embedding for meta-learning
references.venue: In ICCV,
references.authors: Pulkit Agrawal Ross Girshick Jitendra Malik
references.title: Analyzing the performance of multilayer neural networks for object recognition
references.venue: In ECCV,
references.authors: Haitham Bou Ammar Eric Eaton Matthew E. Taylor Decibal C. Mocanu Kurt Driessens Gerhard Weiss Karl Tuyls
references.title: An automated measure of mdp similarity for transfer in reinforcement learning
references.venue: In AAAI Machine Learning for Interactive Systems Workshop,
references.authors: Yajie Bao Yang Li Shao-Lun Huang Lin Zhang Lizhong Zheng Amir Zamir Leonidas Guibas
references.title: An information-theoretic approach to transferability in task transfer learning
references.venue: In ICIP,
references.authors: Marius Cordts Mohamed Omran Sebastian Ramos Timo Rehfeld Markus Enzweiler Rodrigo Benenson Uwe Franke Stefan Roth Bernt Schiele
references.title: The cityscapes dataset for semantic urban scene understanding
references.venue: In CVPR,
references.authors: Yin Cui Yang Song Chen Sun Andrew Howard Serge Belongie
references.title: Large scale fine-grained categorization and domain-specific transfer learning
references.venue: In CVPR,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: In CVPR,
references.authors: Aditya Deshpande Alessandro Achille Avinash Ravichandran Hao Li Luca Zancato Charless Fowlkes Rahul Bhotika Stefano Soatto Pietro Perona
references.title: A linearized framework and a new benchmark for model selection for fine-tuning
references.venue: Arxiv,
references.authors: Jeff Donahue Yangqing Jia Oriol Vinyals Judy Hoffman Ning Zhang Eric Tzeng Trevor Darrell
references.title: Decaf: A deep convolutional activation feature for generic visual recognition
references.venue: In ICML,
references.authors: Kshitij Dwivedi Jiahui Huang Radoslaw Martin Cichy Gemma Roig
references.title: Duality diagram similarity: a generic framework for initialization selection in task transfer learning
references.venue: In ECCV,
references.authors: Kshitij Dwivedi Gemma Roig
references.title: Representation similarity analysis for efficient task taxonomy & transfer learning
references.venue: In CVPR, 2019
references.authors: Eric Eaton Terran Lane
references.title: Modeling transfer relationships between learning tasks for improved inductive transfer
references.venue: In ECML PKDD,
references.authors: Mark Everingham Luc Van Gool Christopher Williams John Winn Andrew Zisserman
references.title: The pascal visual object classes (voc) challenge
references.authors: Li Fei-Fei Rob Fergus Pietro Perona
references.title: One-shot learning of object categories
references.authors: David Freedman Robert Pisani Roger Purves
references.title: Statistics (international student
references.venue: 4th edn. WW Norton & Company, New York,
references.authors: F.R.S. Karl Pearson
references.title: Liii. on lines and planes of closest fit to systems of points in space
references.venue: The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science,
references.authors: Ross Girshick Jeff Donahue Trevor Darrell Jitendra Malik
references.title: Rich feature hierarchies for accurate object detection and semantic segmentation
references.venue: In CVPR,
references.authors: Yunhui Guo Honghui Shi Abhishek Kumar Kristen Grauman Tajana Rosing Rogerio Feris
references.title: Spottune: transfer learning through adaptive fine-tuning
references.venue: In CVPR,
references.authors: Agrim Gupta Piotr Dollar Ross Girshick
references.title: Lvis: A dataset for large vocabulary instance segmentation
references.venue: In CVPR,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In CVPR, 2016
references.authors: Minyoung Huh Pulkit Agrawal Alexei A Efros
references.title: What makes imagenet good for transfer learning? Arxiv, 2016
references.authors: Yiding Jiang Dilip Krishnan Hossein Mobahi Samy Bengio
references.title: Predicting the generalization gap in deep networks with margin distributions. ICLR, 2019
references.authors: Yiding Jiang Behnam Neyshabur Hossein Mobahi Dilip Krishnan Samy Bengio
references.title: Fantastic generalization measures and where to find
references.venue: them. ICLR,
references.authors: Aditya Khosla Nityananda Jayadevaprakash Bangpeng Yao Fei-Fei Li
references.title: Novel dataset for fine-grained image categorization: Stanford dogs
references.venue: In CVPR Workshop on Fine-Grained Visual Categorization (FGVC),
references.authors: Alexander Kirillov Ross Girshick Kaiming He Piotr Dollár
references.title: Panoptic feature pyramid networks
references.venue: In CVPR,
references.authors: Simon Kornblith Jonathon Shlens Quoc V Le
references.title: Do better imagenet models transfer better
references.venue: In CVPR,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks
references.venue: Advances in neural information processing systems,
references.authors: Tsung-Yi Lin Priya Goyal Ross Girshick Kaiming He Piotr Dollár
references.title: Focal loss for dense object detection
references.venue: In ICCV,
references.authors: Tsung-Yi Lin Michael Maire Serge Belongie James Hays Pietro Perona Deva Ramanan Piotr Dollár C Lawrence Zitnick
references.title: Microsoft coco: Common objects in context
references.venue: In ECCV,
references.authors: Dhruv Mahajan Ross Girshick Vignesh Ramanathan Kaiming He Manohar Paluri Yixuan Li Ashwin Bharambe Laurens Van Der Maaten
references.title: Exploring the limits of weakly supervised pretraining
references.venue: In ECCV,
references.authors: Cuong Nguyen Tal Hassner Matthias Seeger Cedric Archambeau
references.title: Leep: A new measure to evaluate transferability of learned representations
references.venue: In ICML,
references.authors: Omkar M. Parkhi Andrea Vedaldi Andrew Zisserman C.V. Jawahar
references.title: Cats and dogs
references.venue: In CVPR,
references.authors: Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga Alban Desmaison Andreas Kopf Edward Yang Zachary DeVito Martin Raison Alykhan Tejani Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala
references.title: Pytorch: An imperative style, high-performance deep learning library
references.venue: NeurIPS
references.authors: Matthew E Peters Sebastian Ruder Noah A Smith
references.title: To tune or not to tune? adapting pretrained representations to diverse tasks. Arxiv, 2019
references.authors: Joan Puigcerver Carlos Riquelme Basil Mustafa Cedric Renggli André Susano Pinto Sylvain Gelly Daniel Keysers Neil Houlsby
references.title: Scalable transfer learning with expert models
references.venue: Arxiv, 2020
references.authors: Alec Radford Jong Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark
references.title: Learning transferable visual models from natural language
references.venue: supervision. Arxiv,
references.authors: Benjamin Recht Rebecca Roelofs Ludwig Schmidt Vaishaal Shankar
references.title: Do imagenet classifiers generalize to imagenet
references.venue: In ICML,
references.authors: Shaoqing Ren Kaiming He Ross Girshick Jian Sun
references.title: Faster r-cnn: Towards real-time object detection with region proposal networks
references.venue: NeurIPS,
references.authors: Ali Sharif Razavian Hossein Azizpour Josephine Sullivan Stefan Carlsson
references.title: Cnn features off-the-shelf: an astounding baseline for recognition
references.venue: In CVPR,
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: Arxiv, 2014
references.authors: Jivko Sinapov Sanmit Narvekar Matteo Leonetti Peter Stone
references.title: Learning inter-task transferability in the absence of target task samples
references.venue: In AAMAS,
references.authors: Jie Song Yixin Chen Xinchao Wang Chengchao Shen Mingli Song
references.title: Deep model transferability from attribution
references.venue: maps. Arxiv,
references.authors: Chen Sun Abhinav Shrivastava Saurabh Singh Abhinav Gupta
references.title: Revisiting unreasonable effectiveness of data in deep learning era
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Christian Szegedy Wei Liu Yangqing Jia Pierre Sermanet Scott Reed Dragomir Anguelov Dumitru Erhan Vincent Vanhoucke Andrew Rabinovich
references.title: Going deeper with convolutions
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang Chunfang Liu
references.title: A survey on deep transfer learning
references.venue: In ICANN,
references.authors: Mingxing Tan Quoc Le
references.title: Efficientnet: Rethinking model scaling for convolutional neural networks
references.venue: In ICML,
references.authors: Bart Thomee David A Shamma Gerald Friedland Benjamin Elizalde Karl Ni Douglas Poland Damian Borth Li-Jia Li
references.title: Yfcc100m: The new data in multimedia research
references.venue: Communications of the ACM,
references.authors: Anh T Tran Cuong V Nguyen Tal Hassner
references.title: Transferability and hardness of supervised classification tasks
references.venue: In ICCV, 2019
references.authors: Thomas Unterthiner Daniel Keysers Sylvain Gelly Olivier Bousquet Ilya Tolstikhin
references.title: Predicting neural network accuracy from weights
references.venue: Arxiv,
references.authors: Grant Van Horn Steve Branson Ryan Farrell Scott Haber Jessie Barry Panos Ipeirotis Pietro Perona Serge Belongie
references.title: Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection
references.venue: In CVPR,
references.authors: P. Welinder S. Branson T. Mita C. Wah F. Schroff S. Belongie P. Perona
references.title: Caltech-UCSD Birds 200
references.venue: Technical Report CNS-TR-2010-001, California Institute of Technology,
references.authors: Xi Yan David Acuna Sanja Fidler
references.title: Neural data server: A large-scale search engine for transfer learning data
references.venue: In CVPR,
references.authors: Kaichao You Yong Liu Jianmin Wang Mingsheng Long
references.title: Logme: Practical assessment of pre-trained models for transfer learning
references.venue: In ICML,
references.authors: Amir R Zamir Alexander Sax William Shen Leonidas J Guibas Jitendra Malik Silvio Savarese
references.title: Taskonomy: Disentangling task transfer learning
references.venue: In CVPR,
references.authors: Matthew D Zeiler Rob Fergus
references.title: Visualizing and understanding convolutional networks
references.venue: In ECCV,
references.authors: Bolei Zhou Agata Lapedriza Jianxiong Xiao Antonio Torralba Aude Oliva
references.title: Learning deep features for scene recognition using places database
sections.heading: 1 Introduction
sections.text: Deep Neural Networks (DNNs) have shown to be very capable of solving a wide variety of visual tasks. However, these networks often require large amounts of data and training time to perform well, limiting the accessibility of deep learning for computer vision. One approach to alleviate this problem is to employ transfer learning, commonly by fine-tuning an off-the-shelf model on the desired task.
  With the increasing number of off-the-shelf models available spanning different tasks, datasets, training methods, and architectures, choosing the best model from which to transfer is a challenging endeavor. A common heuristic in computer vision has been to use models pretrained on ImageNet [7] (specifically the ILSVRC challenge data), but more recent work is starting to expose weaknesses in the generalization performance of ImageNet features [39, 28, 40], and many of these works find that other pretraining datasets may perform better for some target tasks. Aside from source dataset, which architecture to select isn’t clear (when comparing similarly fast models). One could theoretically train several transfers from a diverse set of pretrained weights and keep the best performing model, but this isn’t feasible in most practical applications where compute is limited.
  This motivates the need for a model selection method that could query a large bank of existing pretrained models (of which several already exist, e.g., [36, 56, 18]) with a small subset of the target data and return a set of weights which performs well when fine-tuned on all target data. Because of the sheer quantity of pretrained weights available to download off the internet today, such a model bank has the potential to be massive and cover a wide variety of tasks, datasets, and architectures. Any ∗Equal Contribution †https://dbolya.github.io/parc/
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  selection method that intends to operate on such a massive library of weights would thus need two important properties: it would need to be scalable in order to accommodate 100s to 1000s of source models, and it would need to perform well on a diverse set of input weights due to the wide variety of models available. In this paper, we formalize this setting as Scalable Diverse Model Selection.
  Currently, there are two main lines of work that could address this scenario. The first is model selection [11, 46, 10], which attempts to select a viable transfer from a suite of arbitrary pretrained models. However, existing approaches require an initial model trained on the target data to suggest the transfer (which limits accessibility) and are not very successful when comparing across architectures (see Tab. 1). The second line of work is in transferability estimation [4, 52, 34], which attempts to predict how well a source model will transfer to a target task. While this might seem similar to model selection, the two tasks are subtly different in evaluation. Transferability estimation methods usually fix one source model and vary the target task (i.e., “For my source model, which task would it transfer to the best?”), while model selection methods do the opposite: fix a target task and vary the source models (i.e., “For my target task, which source model will transfer the best?”). While it might seem the same methods could work for either case, this turns out to not be the case (see Tab. 2).
  Contributions. We formalize the task of Scalable Diverse Model Selection, which intends to make deep learning for computer vision more accessible. While other papers might have explored aspects of this space already, we standardize it by introducing several tools and benchmarks for evaluating model selection methods in this setting. First, we provide a controlled environment that includes exhaustively trained transfers from 8 source datasets to 6 target datasets across 4 different commonly used architectures for a total of 168 ground truth transfers for analysis (Sec. 3). We show that current state-of-the-art transferability and model selection methods fail to beat simple baselines in this new setting (Tab. 1). We then analyze why this is the case and provide techniques to improve performance (Sec. 4). Using insights from this analysis, we develop PARC, a method that outperforms other methods on this benchmark (Sec. 5). Finally, we show that these results generalize to a larger experiment with an extra dataset and 33 additional off-the-shelf pretrained models downloaded from the internet (for a total of 65 source models and 423 transfers) and by extending PARC to object detection (Sec. 6). We have released all benchmarks and evaluation code at https://dbolya.github.io/parc/ in hopes to further the development of this promising area of research.
sections.heading: 2 Related Work
sections.text: In this paper, we introduce a new task for Scalable Diverse Model Selection, which attempts to make transfer learning more accessible by selecting the best pretrained model for a downstream task from a massive bank of off-the-shelf models. This is adjacent to several fields such as accessible transfer learning, transferability estimation, and Taskonomy model selection.
  Transfer Learning. Transfer learning is the act of using a model trained in one setting to boost the performance or speed up the training of a model in another setting, and is an extensively studied field in deep learning [49]. Several approaches exist for transfer learning from a source model to a target dataset, the most popular of which include fine-tuning a new head to the target domain [9, 43, 60, 39], and fine-tuning the entire network [2, 17]. While there are more sophisticated methods for fine-tuning (e.g., [19, 37]), we study fine-tuning the entire network, as it’s simple and widely adopted.
  Accessible Transfer Leaning. A few works have tried to make transfer learning more accessible. Neural Data Server [57] allows users to augment their own data with similar data indexed from several massive image datasets. While helpful in a limited data environment, this requires the user to have additional compute in order to train with the extra images. Iterating on this idea, Scalable Transfer Learning with Expert Models [38] suggests pretrained models to the user instead, relaxing the compute requirement. However, the focus of the paper is primarily on creating these pretrained “expert” models and not in selecting between them. We argue that there are several “expert” models already widely available and trained with a large amount of data (e.g., [47, 33, 39]), and thus focus on the model selection aspect of this problem. Both of these works use simple baselines to guide their model selection: Neural Data Server [57] uses the accuracy of a logistic classifier on the source features fit to the target task, while Scalable Transfer Learning [38] uses nearest neighbors with hold-one-out cross-validation. We benchmark both baseline techniques in our setting.
  Transferability Estimation. There are several works that attempt to predict how well a model will transfer to new tasks, ranging from methods that attempt to assess a models capacity for transfer [23, 28, 40] to those that attempt to predict transfer accuracy [12, 3, 45]. Others attempt to predict the gap in generalization between training and test time [24, 53, 25]. There has also been a recent line of work that directly attempts to estimate transfer learning accuracy given a source model and target dataset [4, 52, 34]. This line of work is the most applicable to our setting because the only assumption they make on the source model is that it was trained on classification. While not ideal, this allows us to benchmark these methods (H-Score [4], NCE [52], and LEEP [34]) in our setting. Otherwise, Deshpande et al. [8] propose perhaps what is most directly applicable to our work. Their setting is very similar (and fairly concurrent), though not as diverse and with no restrictions on evaluation speed. In addition, LogMe [58] is concurrent work that also focuses on practical transferability estimation, but they don’t release their benchmark and their setting is more narrow.
  Taskonomy Model Selection. The Taskonomy [59] models and dataset has been used to benchmark a previous line of model selection algorithms [11, 46, 10]. Taskonomy attempts to model similarities between tasks by how well models transferring from one task to another perform after fine-tuning. This makes it a natural test-bed for model selection evaluation, as it contains a large number of pretrained transfers to use as a benchmark. However, using Taskonomy as a benchmark is incomplete. All Taskonomy models are trained on the same data (with different labels) and follow roughly the same architecture, only varying the source and target task. This means that the benchmark doesn’t test robustness to source model diversity, which is a core tenant of this work. Furthermore, typical model selection works on Taskonomy require a network trained on the target data to suggest transfers, which should be avoided to make transfer learning more accessible. We benchmark the performance of RSA [11] and DDS [10], as they are the best performing methods on Taskonomy model selection. There are other works in this space such as [6] and [1], but the former scales poorly with the number of sources, and the latter cannot compare across architectures by design, so we don’t include either in our experiments.
sections.heading: 3 Scalable Diverse Model Selection
sections.text: In this work, we address the problem of model selection for transfer learning but through an expressly practical lens. The goal of model selection from a practitioner’s point of view is to find a pretrained off-the-shelf model that will perform well after fine-tuning on their data. In order for a model selection method to perform well in this setting, the models it selects from need to be diverse (i.e., cover a wide variety of source datasets, architectures, and pretraining tasks), and the selection method needs to be scalable (since the number of off-the-shelf models available today is massive and growing).
  In order to provide a realistic transfer learning model selection scenario, we propose to select a model from a large source model bank (over 100 models spanning several datasets and architectures) that transfers well to a target training set after full model fine-tuning. The naive approach to this problem would be to simply fully fine-tune a transfer from each source model to the target training set, but this would of course be computationally infeasible. We’d like to work in the practical setting where we don’t train any extra models, so we’d like a computationally efficient transferability estimation method to predict which source models would transfer well. For the same reason, it’s also infeasible to use the entire target training set for this estimate, since extracting all the features for 100 different source models is akin to training a new model for 100 epochs (though without the backward pass). Thus, we restrict the method to only use a small subset of the target training data for its estimate.
  In this scenario, the target dataset is fixed while the source models can vary in dataset, architecture, and task. Previous work in model selection and transferability estimation typically only vary one of these aspects when evaluating their method (i.e., just task [59, 4, 11, 10], just dataset [34, 52], or just architecture [34], but not all three at once). It’s unclear how well, if at all, these methods benchmarked by varying only one factor will perform when all of these factors can change. Thus, we’ve set out to test these methods in this much more challenging setting.
sections.heading: 3.1 Creating a Diverse Benchmark
sections.text: Because no model selection benchmark currently exists that varies more than one source factor at time, we create our own from 8 source datasets and 6 target datasets across 4 different architectures for classification, varying both source dataset and architecture. We will also vary task in Sec. 6.
  Datasets and Architectures. An ideal selection of source datasets would contain related datasets, so that transfer learning makes sense. Thus, for this benchmark, we choose 6 well-known classification datasets of various difficulties that contain related subthemes: Pets: Stanford Dogs [26] and Oxford Pets [35], Birds: CUB200 [55] and NA Birds [54], and Miscellaneous: CIFAR10 [29] and Caltech101 [14]. We also include VOC2007 [13] and ImageNet 1k [7] as the 7th and 8th source datasets, but not as targets. The other 6 datasets are also included as targets.
  When selecting a model, a practitioner is often interested in the accuracy-speed trade-off, meaning that the benchmark should include architectures at several tiers of evaluation speed. To facilitate this, we include three tiers of architectures: ResNet-50 [22] as the slowest, ResNet-18 [22] and GoogLeNet [48] in the middle, and AlexNet [30] as the fastest.
  Evaluation. The goal in model selection is to find a source model that will transfer well to a target task. Ideally, this would be the best performing model, but that might be unreasonably difficult when there are 100s of models to choose from. Furthermore, practitioners are typically interested in the trade-off between performance and inference speed, which requires us to consider more than just the highest scoring models. What we really need in this case is a score for each model that correlates well with final fine-tuned accuracy on some target data. To benchmark such a score, we use Pearson Correlation [15], a widely adopted correlation metric (with 0 implying no correlation and 100 implying perfect correlation), between the model selection algorithm’s transferability scores and the final fine-tuned accuracy.
  Thus we employ the following procedure to test a model selection method A on our benchmark: for each target dataset Dt (indexed by t ∈ T ), we sample an n image “probe set” Ptn ⊆ Dt. Then, for each source model parameterized by θs (indexed by s ∈ S), we obtain
  αts = A(θs,Ptn) (1) as the predicted score for how well the source model θs will transfer to the target datasetDt. Then, we train each transfer from θs toDt and evaluate it on the test set of t to obtain the final transfer accuracies ωts. Finally, we compute Pearson correlation (denoted by pearsonr) between the predicted and final transfer accuracy and average it over all target datasets:
  Mean PC (Varying Source) = 1 |T | ∑ t∈T pearsonr({αts : s ∈ S}, {ωts : s ∈ S}) (2)
  Because there can be a large amount of variance in the probe set sampled, we further report mean and standard deviation over 5 different randomly sampled probe sets.
  Note that we explicitly use Pearson Correlation because it incorporates all data points, if all you care about is selecting the most accurate model (without limits on speed or other model parameters), other metrics such as top-k accuracy may be more suitable. Thus, we include some extra metrics in the Appendix.
  Implementation Details. For simplicity, images from all datasets are resized to 224× 224. When constructing the probe sets, we ensure that there are at least 2 examples of each class (necessary for several methods) and randomly subsample classes if this results in more than n = 500 images. We train all source models and transfers using SGD with no weights frozen (i.e., full fine-tuning) and employ grid search to find optimal hyperparameters for each target dataset, architecture pair. Previous work in this space assume that each target model is trained with exactly the same hyperparameters. However, in a practical setting we expect some tuning to be done when training on the target dataset, so we have done the same. This makes the selection task more challenging but also more aligned with best case transfer outcomes. All models are trained on Titan Xp GPUs and all transferability methods are evaluated on the CPU. Note that times should be taken as lower bounds, since they’re evaluated with expensive hardware. Many would-be practitioners don’t have such hardware at their disposal.
sections.heading: 3.2 Benchmarking Existing Work
sections.text: We use this benchmark to evaluate several recent model selection and transferability estimation methods. We also include some typical baselines to contextualize the performance of these methods.
  Probability-Based Methods. We test two very recent transferability estimation methods, NCE [52] and LEEP [34]. Both operate similarly: given a source model parameterized by θ that predicts
  probabilities pθ(z | x) over the source classes z, input target images x and estimate a joint probability p̂(z, y) between the source classes z and the target classes y. Then p̂θ(y | x) can be estimated as
  p̂θ(y | x) = ∑ zi p̂(y | zi) pθ(zi | x) (3)
  Transferability is then computed by aggregating this probability distribution for all target images x. NCE and LEEP differ mainly in how they produce p̂(z, y): NCE (as extended in [34]) produces this by counting when the source model predicts z for a target image with label y, while LEEP incorporates the entire distribution pθ(z | x). The former naturally requires more data.
  Feature-Based Methods. We also include several recent works from the Taskonomy [59] model selection literature: H-Score [4], RSA [11], and DDS [10]. All of these methods compare the penultimate layer’s features fθ(x) of a source model parameterized by θ evaluated on target images x to a different set of features. H-score compares the covariance of the features with the covariance of their mean over the target classes y. Note that H-Score scales poorly with latent feature size, since they invert the covariance of the features (which can be as big as 4096× 4096 for AlexNet [30]). RSA [11] and DDS [10], on the other hand, compare the source model’s extracted features to that of a “probe” model already trained on the target data. They assume that if two images are far apart in the probe model’s feature space, then they should also be far apart in the feature space of a good source model. RSA and DDS vary in how they construct these distances and how they aggregate them to produce final scores: RSA uses one minus the correlation coefficient between each pair of images for distance and Spearman correlation for the final score, while DDS tests a large number of combinations, of which we choose the best performing in [10] (cosine distance and z-score). Note that both of these methods require training an additional model before they can begin recommending transfers, and the architecture used for this probe model highly impacts performance (see Tab. 1). Because these methods requires extra training time and expert knowledge of what architecture to use, we include them only for reference. While 3 extra hours of training time on a Titan Xp might not seem like much, it drastically limits accessibility on cheaper hardware, where training can take days.
  Baseline Methods. Finally, we include three fairly standard baselines. The first two are simple classifiers learned on top of the penultimate layer’s features fθ(x) and tested on the probe set to get an estimate of final fine-tuning accuracy (outputting accuracy as the score). For these methods, we include k = 1 nearest neighbors with leave one out cross-validation (denoted as k-NN CV, used in [38]) and a logistic classifier trained on one half of the probe set and evaluated on the other half (used in [57]). We also include a simple heuristic that rates performance as the number of layers in the source network `s plus the log of the total number of images in both the source and target sets:
  heuristicts = `s + log ( |Ds|+ |Dt| ) (4)
  This captures the intuition that more layers and more data is better (with log data scaling).
  Results. In Tab. 1 we list the Mean Pearson Correlation (computed as described in Sec. 3.1) and average time taken per transfer (not including feature extraction) for each method. We also include important properties of each method that can impact its scalability (training time, time taken) and whether it supports a diverse set of models (input requirements, being agnostic to source / target task).
  From this experiment, we can see that current works fare poorly when applied to this new benchmark. In fact, none of these methods perform better than the simple k = 1 nearest neighbors baseline. Some even have close to no correlation (LEEP, NCE, H-Score). Furthermore, many of these methods are extremely expensive to compute (e.g., H-Score inverts a very large matrix when evaluating on AlexNet). RSA and DDS also require training of a target model with a manually selected architecture and high variability if poorly chosen (see Table 1). In the next section, we explore why these methods, which were not designed for our challenging transfer setting, may fail to generalize.
  A Note on Scalability. This benchmark includes 168 transfers, which should be enough to test the scalability of model selection approaches. However, when evaluating current methods, we run the method on each source-target transfer and thus the “scalability” is linear per transfer. No current work exists to make this process sub-linear (e.g., through some kind of a weight embedding that you can binary search through), so instead we focus on the runtime speed of these methods. However, the end-goal of Scalable Diverse Model Selection is really to have sub-linear scaling, so we hope that future work can become even more scalable in that sense.
sections.heading: 4 Analyzing Failure Modes of Existing Selection Methods
sections.text: In this section, we explore what causes existing algorithms to fail in Sec. 3 and provide simple techniques to improve these methods on the challenging task of diverse model selection.
  Small Probe Set Size. One potential reason why these methods fail to generalize to this setting is that, for efficiency, we use a probe set of n = 500 target train samples to compute transferability, while the final transfer model fine-tunes on all the data. In Fig. 1, we vary n and show relative correlation (a) and the ratio of time taken (b) for each method. NCE, LEEP, and H-Score all gain a significant boost from the extra data, implying that perhaps the restriction of a probe set is a major limiting factor for their performance. On the other hand, RSA, DDS, and the baseline methods seem to be able to perform well with even fewer samples, which is a boon to scalability.
  Robustness to Evaluation Mode. There’s a subtle, but important, point about evaluation that’s often overlooked in transferability estimation. In model selection, you assume that you have a fixed
  target dataset and would like to select the best source model for transfer. Transferability estimation often answers the opposite question: given a fixed source model, which target dataset will it best transfer to? The former evaluates following Eq. 2, while the latter considers the following correlation:
  Mean PC (Varying Target) = 1 |S| ∑ s∈S pearsonr({αts : t ∈ T}, {ωts : t ∈ T}) (5)
  One might assume that the same method would work well in both situations, but this is not the case.
  In Tab. 2, we test different evaluation settings by swapping what set correlation is being computed over (i.e., S for source and architecture, T for target, and subsets of S for the rest). For each nonbaseline method, we’ve marked their native evaluation mode (using source dataset for Taskonomy). Most methods work well using their original evaluation protocol, but are not robust to other settings. Since each evaluation mode compares outputs across different factors, we hypothesize that the poor performance may be due to miscalibration and propose ways to mitigate this in the following sections.
  Dimensionality Reduction. For the feature based methods, there’s a lot of potential variation in the number of features between architectures. For instance, the number of features |fθ(x)| for AlexNet is 4096, while for ResNet-18, it’s 1024. Not only does this cause some methods to vary wildly in evaluation time (e.g., H-Score), but it also can be a source of miscalibration. To address this, we apply PCA dimensionality reduction [16] to the features, fθ(x), down to a fixed dimension f before computing each selection method. The results for this experiment are displayed in Fig. 2. All nonbaseline feature-based methods receive a significant boost in performance from this dimensionality
  reduction step. RSA and DDS even outperform the baselines after dimensionality reduction. For subsequent experiments, we will use the best value for f found in Fig. 2a for each method.
  Capacity to Change. The goal of a scalable diverse model selection algorithm is predict a score that correlates well with full fine-tuning on the target set. However, all methods so far have used the source model as fixed features or probabilities, without considering the potential for those features to change after fine-tuning. Thus, we observe poor performance on more difficult target datasets (such as NA Birds, see the Appendix), where models with a high capacity of learning are required. While modeling the capacity for a network to learn is out of the scope of this work, we can substitute that with a simple heuristic. A strong indicator of how much information a CNN can learn that applies to most off-the-shelf architectures (e.g., [30, 44, 48, 22, 50]) is simply the depth of the model (i.e., number of layers). In Tab. 3, we compare the performance on our benchmark for each method before and after adding this heuristic. To integrate this intuition, we first normalize the predicted scores αts for each method by their mean µt and standard deviation σt over all transfers and then add the relative source model depth `s over the maximum depth possible, `max (i.e., `max = 50 for our experiments):
  (αts) ′ =
  αts − µt
  σt + `s `max
  (6)
  This change results in a significant boost in performance for all methods and will be denoted as +` for future experiments. We explore other ways to incorporate this model capacity in the Appendix.
sections.heading: 5 Pairwise Annotation Representation Comparison (PARC)
sections.text: Following the insights gained in Sec. 4, we devise a new method for the task of scalable diverse model selection. RSA [11] with dimensionality reduction performs extremely well (see Fig. 2), however it requires a “probe model” trained on the target data. To alleviate this restriction, we perform an intuitive modification: we replace the probe model with the ground truth labels. Features that work well for a target task should consider two images dissimilar if they were annotated differently.
  More formally, given a probe set Pn of target images x and labels y and a model parameterized by θ, PARC produces two distance matrices Dθ, Dy of shape n× n as
  Dθ = 1− corrcoef(fθ(x)) Dy = 1− corrcoef(g(y)) (7) where corrcoef computes pairwise Pearson product-moment correlation between the features of each pair of images (as used in RSA) and g maps the labels y to some vector representation. For classification, g maps y to the corresponding one-hot vector, but in general, g can be any function that maps the annotations to a vector. For instance, with semantic segmentation as the target task, g could produce a pixel-wise average of the annotations, and similar extensions exist for other computer vision tasks. We explore this further in Sec. 6.2.
  Then, to compute the final PARC score, like RSA we simply compute the Spearman correlation between the two distance matrices for all pairs of images:
  PARC(θ,Pn) = spearmanr({Dθ[i, j] : i < j}, {Dy[i, j] : i < j}) (8)
  Results. We apply all the same improvements discussed in Sec. 4 and report the performance of PARC on our benchmark in Tab. 4. With both the dimensionality reduction and heuristic ensemble, PARC outperforms every other method (even with the same improvements). PARC observes a much larger boost from the heuristic ensemble than RSA, likely because RSA is allowed to use a ResNet-50 model fine-tuned on the target set. Thus, it already has some information about how features can change over fine-tuning built in (which is what the heuristic is trying to accomplish). PARC, on the other hand, isn’t allowed this extra information, and thus adding the heuristic helps with calibration tremendously. We include more results for PARC in the Appendix.
sections.heading: 6 Extended Benchmarks
sections.text: Our benchmark in Sec. 3 describes a more practically useful setting than previous works, but it doesn’t fully encapsulate scalable diverse model selection. Ideally, we could predict transferability between any source model (for any dataset, architecture, or task) to any target dataset or task. In this section, we explore arbitrary crowd-sourced source models and object detection as the target task.
sections.heading: 6.1 A Crowd-Sourced Benchmark
sections.text: In this experiment, we test how well PARC and other model selection methods perform on an even more diverse source model bank. To facilitate this, we collect 33 models from online model banks (i.e., [56, 18]) that span several source datasets ([7, 13, 32, 20, 51, 61, 33, 5]) and model architectures ([22, 42, 21, 27, 31]) covering many different pretext tasks (see the Appendix for full details). We combine this with the original 32 models we trained for a total of 65 source models and we add VOC2007 [13] multi-class classification as an additional target task, resulting in 423 transfers total. We place no restrictions on how the original models were trained, but we do normalize their features.
  Results for this benchmark are available in Tab. 5. We only test on the subset of methods that support multi-class classification out of the box and apply all improvements from Sec. 4. While PARC outperforms the other methods, we note that none of the methods perform very well here (all having around 50% correlation with transfer accuracy). This indicates that selecting from an extremely diverse set of source models is still a very challenging task and warrants further study.
sections.heading: 6.2 Transferability to Detection
sections.text: So far, we’ve only considered classification as the target task. However, as mentioned in Sec. 5, we can apply PARC to other tasks by summarizing the annotations y with a vector g(y), e.g., by averaging the annotations pixel-wise. To average “pixel-wise” for object detection, we count all pixels belonging to boxes for each class, and then normalize by the total area of all boxes in the image. To extend 1-NN, we measure the L1 distance between pairs of these aggregate label vectors.
  In Tab. 6, we display the performance of each method for predicting transfers from 6 source detectors (Faster and Mask R-CNN [42, 21] on Cityscapes [5] and COCO [32], and Retinanet [31] and YOLOv3 [41] on just COCO) transferring to VOC2007 detection [13]. We compute Pearson correlation between the predicted scores and the mAP of each fine-tuned model, and we provide RSA and DDS a Faster R-CNN model trained on VOC2007. Because there are only 6 transfers, this is a much easier experiment than our main benchmark (and thus we found the techniques discussed in Sec. 4 to not
  be important). While it doesn’t perform as well as RSA or DDS here, PARC requires no additional model trained on the target data. Yet, it is still highly correlated with final fine-tuned mAP.
sections.heading: 7 Conclusion and Limitations
sections.text: In this work, we introduce Scalable Diverse Model Selection, create several benchmarks to test this task, analyze existing methods in this setting, address multiple techniques to improve performance, and finally iterate on existing methods to create a new approach that works well. While the techniques we found combined with PARC improve performance on this setting, there are a few limitations of our work. First, we assume that the model selection method is applied to every source model, which could get extremely expensive. Subsequent work could try relaxing this assumption. Second, lowering the barrier to entry to transfer learning makes deep learning more accessible, but the model returned by these algorithms isn’t guaranteed to be the best, so more work would likely need to be done to temper the expectations of non-experts. Finally, selecting from any arbitrary set of models in a scalable way still remains challenging (especially on crowd-sourced models) and thus is still an open problem. Many factors can affect fine-tuned performance, but we only consider source feature quality and architecture capacity in this paper. We address this latter point with a simple heuristic, which is slightly unsatisfying. For instance, an ideal system could have some comprehensive learnability score for each architecture instead. Several papers look at factors that affect fine-tuning performance in isolation, but none combine everything into one recommendation system. We hope that this paper can be the first step in creating such a diverse model selection algorithm. We believe that a robust system to recommend pretrained models for transfer learning would be an incredible boon for the accessibility of deep learning and hope that future work can study this task in further detail.
title: Scalable Diverse Model Selection for Accessible Transfer Learning