summaries: # Summary  This paper proposes a method for using intermediate representations from a pre-trained model for better transfer to other tasks. A linear classifier is trained on features from multiple layers. A feature selection strategy is chosen, where first a linear classifier is trained on all features with group-lasso regularization, and then another classifier is trained only on features whose regularization score exceeds some threshold. This approach is evaluated in a low-resource transfer scenario of image classification, and compared both to training on all features without the two-step feature selection but with regularization, and to the standard fine-tuning approach. The proposed method works slightly better than fine-tuning with a ResNet model, and slightly worse with a ViT model, although the comparison is not completely fair, as discussed. There are different performances on different categories of images.    This paper explores the utility of intermediate layers for linear probing in transfer learning. Specifically, authors proposed Head-to-Toe probing (HEAD2TOE), that selects features from all layers of the source model to train a classification head for the target-domain. Experiments on the VTAB benchmark shows that Head2Toe matches performance obtained with fine-tuning on average, but critically, for out-of-distribution transfer, Head2Toe outperforms fine-tuning. This paper propose Head2Toe, a method that exploits intermediate representations of DNNs to improve performance, and OOD generalization of transfer learning. The key idea of the Head2Toe is to augment traditional linear probing with intermediate representations. It uses group lasso as regularization to learn weights of different features, and then select features based on the learned weights and validation performance.Experiments on VTAB benchmark show that Head2Toe outperforms linear probes and is competitive compared with fine-tuning. An important finding is that Head2Toe improves OOD generalization. The main focus of this paper is the study the use of intermediate layers in a deep pretrained model on downstream tasks. Authors argue that the the fact that the good performance while fine tuning on downstream tasks even if data scarce is due to the prior existence of useful representations deep in the model which are brought up during training. Authors propose a new approach. Head2toe, which consist in utilizing intermediate representations, while freezing the weights of the network for training for downstream tasks Experimentation is carried out starting with a ResNet-50 and ViT-B/16 models on a variety of datasets collected on the VTAB collection, showing that the approach where Head2toe  outperforms linear finetunning (training a classification head on top of the model, while freezing the rest of the weights) and matches the performance of finetunning the whole model.
abstractText: Transfer-learning methods aim to improve performance in a data-scarce target domain using a model pretrained on a data-rich source domain. A cost-efficient strategy, linear probing, involves freezing the source model and training a new classification head for the target domain. This strategy is outperformed by a more costly but state-of-the-art method—fine-tuning all parameters of the source model to the target domain—possibly because fine-tuning allows the model to leverage useful information from intermediate layers which is otherwise discarded by the later pretrained layers. We explore the hypothesis that these intermediate layers might be directly exploited by linear probing. We propose a method, Head-to-Toe probing (HEAD2TOE), that selects features from all layers of the source model to train a classification head for the target-domain. In evaluations on the Visual Task Adaptation Benchmark (VTAB), Head2Toe matches performance obtained with fine-tuning on average, but critically, for out-of-distribution transfer, Head2Toe outperforms fine-tuning.
references.authors: Zaid Alyafeai Maged Saeed AlShaibani Irfan Ahmad
references.title: A survey on transfer learning in natural language processing
references.venue: arXiv preprint arXiv:2007.04239,
references.authors: Andreas Argyriou Theodoros Evgeniou Massimiliano Pontil
references.title: Multi-task feature learning
references.venue: Advances in Neural Information Processing Systems. MIT Press,
references.authors: Zahra Atashgahi Ghada Sokar Tim van der Lee Elena Mocanu Decebal Constantin Mocanu Raymond N.J. Veldhuis Mykola Pechenizkiy
references.title: Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders
references.venue: ArXiv, abs/2012.00560,
references.authors: R. Baldock Hartmut Maennel Behnam Neyshabur
references.title: Deep learning through the lens of example
references.venue: difficulty. ArXiv,
references.authors: Muhammed Fatih Balın Abubakar Abid James Zou
references.title: Concrete autoencoders: Differentiable feature selection and reconstruction
references.venue: In Proceedings of the 36th International Conference on Machine Learning,
references.authors: Hakan Bilen Andrea Vedaldi
references.title: Universal representations: The missing link between faces, text, planktons, and cat breeds
references.venue: arXiv preprint arXiv:1701.07275,
references.authors: Avrim L. Blum Pat Langley
references.title: Selection of relevant features and examples in machine learning
references.venue: Artificial Intelligence,
references.authors: Vernica Boln-Canedo Noelia Snchez-Maroo Amparo Alonso-Betanzos
references.title: Feature Selection for High-Dimensional Data
references.authors: Giorgos Borboudakis I. Tsamardinos
references.title: Forward-backward selection with early dropping
references.venue: J. Mach. Learn. Res.,
references.authors: Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly Jakob Uszkoreit Neil Houlsby
references.title: An image is worth 16x16 words: Transformers for image recognition at scale
references.venue: In International Conference on Learning Representations,
references.authors: Vincent Dumoulin Neil Houlsby Utku Evci Xiaohua Zhai Ross Goroshin Sylvain Gelly Hugo Larochelle
references.title: Comparing transfer and meta learning approaches on a unified few-shot classification benchmark
references.venue: arXiv preprint arXiv:2104.02638,
references.authors: Nikita Dvornik Cordelia Schmid Julien Mairal
references.title: Selecting relevant features from a multidomain representation for few-shot classification
references.venue: In European Conference on Computer Vision,
references.authors: Cheng Fu Hanxian Huang Xinyun Chen Yuandong Tian Jishen Zhao
references.title: Learn-to-share: A hardware-friendly transfer learning framework exploiting computation and parameter sharing
references.venue: In ICML,
references.authors: Jie Gui Zhenan Sun Shuiwang Ji Dacheng Tao Tieniu Tan
references.title: Feature selection based on structured sparsity: A comprehensive study
references.venue: IEEE Transactions on Neural Networks and Learning Systems,
references.authors: Demi Guo Alexander M. Rush Yoon Kim
references.title: Parameter-efficient transfer learning with diff pruning
references.venue: In ACL/IJCNLP,
references.authors: Yunhui Guo Humphrey Shi Abhishek Kumar Kristen Grauman Tajana Simunic Rogério Schmidt Feris
references.title: Spottune: Transfer learning through adaptive fine-tuning
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Isabelle Guyon Jason Weston Stephen D. Barnhill Vladimir Naumovich Vapnik
references.title: Gene selection for cancer classification using support vector machines
references.venue: Machine Learning,
references.authors: Xiaofei He Deng Cai Partha Niyogi
references.title: Laplacian score for feature selection
references.venue: In Proceedings of the 18th International Conference on Neural Information Processing Systems,
references.authors: Timothy Hospedales Antreas Antoniou Paul Micaelli Amos Storkey
references.title: Meta-learning in neural networks: A survey
references.venue: arXiv preprint arXiv:2004.05439,
references.authors: Neil Houlsby Andrei Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin de Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly
references.title: Parameter-efficient transfer learning for nlp
references.authors: Yigitcan Kaya Sanghyun Hong T. Dumitras
references.title: Shallow-deep networks: Understanding and mitigating network overthinking
references.venue: In ICML,
references.authors: Ron Kohavi George H. John
references.title: Wrappers for feature subset selection
references.venue: Artif. Intell.,
references.authors: Simon Kornblith Jonathon Shlens Quoc V. Le
references.title: Do better imagenet models transfer better
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Wei-Hong Li Xialei Liu Hakan Bilen
references.title: Improving task adaptation for cross-domain few-shot learning
references.venue: arXiv preprint arXiv:2107.00358,
references.authors: Wei-Hong Li Xialei Liu Hakan Bilen
references.title: Universal representation learning from multiple domains for few-shot classification
references.venue: arXiv preprint arXiv:2103.13841,
references.authors: Lu Liu William Hamilton Guodong Long Jing Jiang Hugo Larochelle
references.title: A universal representation transformer layer for few-shot image classification
references.venue: In International Conference on Learning Representations,
references.authors: Michael C Mozer Paul Smolensky
references.title: Skeletonization: A technique for trimming the fat from a network via relevance assessment
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Basil Mustafa Carlos Riquelme Joan Puigcerver andAndr’e Susano Pinto Daniel Keysers Neil Houlsby
references.title: Deep ensembles for low-data transfer
references.venue: learning. ArXiv,
references.authors: Feiping Nie Heng Huang Xiao Cai Chris Ding
references.title: Efficient and robust feature selection via joint `2,1-norms minimization
references.venue: Advances in Neural Information Processing Systems,
references.authors: Matthew E. Peters Mark Neumann Mohit Iyyer Matt Gardner Christopher Clark Kenton Lee Luke Zettlemoyer
references.title: Deep contextualized word representations
references.authors: Matthew E. Peters Sebastian Ruder Noah A. Smith
references.title: To tune or not to tune? adapting pretrained representations to diverse tasks
references.venue: In RepL4NLP@ACL,
references.authors: Sylvestre-Alvise Rebuffi Hakan Bilen Andrea Vedaldi
references.title: Learning multiple visual domains with residual adapters
references.venue: arXiv preprint arXiv:1705.08045,
references.authors: Cédric Renggli André Susano Pinto Luka Rimanic J. Puigcerver Carlos Riquelme Ce Zhang Mario Lucic
references.title: Which model to transfer? finding the needle in the growing haystack
references.authors: Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein Alexander C. Berg Li FeiFei
references.title: Imagenet large scale visual recognition challenge
references.venue: International Journal of Computer Vision (IJCV),
references.authors: Jürgen Schmidhuber
references.title: Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-.
references.venue: hook. PhD thesis, Technische Universität München,
references.authors: Jürgen Schmidhuber Jieyu Zhao Marco Wiering
references.title: Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement
references.venue: Machine Learning,
references.authors: Surat Teerapittayanon Bradley McDanel H.T. Kung
references.title: Branchynet: Fast inference via early exiting from deep neural networks. 2016
references.venue: 23rd International Conference on Pattern Recognition (ICPR),
references.authors: Sebastian Thrun Lorien Pratt
references.title: Learning to learn: Introduction and overview
references.venue: In Learning to learn,
references.authors: Eleni Triantafillou Hugo Larochelle Richard Zemel Vincent Dumoulin
references.title: Learning a universal template for few-shot dataset generalization
references.venue: In International Conference on Machine Learning,
references.authors: Paul A. Viola Michael J. Jones
references.title: Rapid object detection using a boosted cascade of simple features
references.venue: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
references.authors: Songtao Wu Shenghua Zhong Yan Liu
references.title: Deep residual learning for image steganalysis
references.venue: Multimedia Tools and Applications,
references.authors: Ming Yuan Yi Lin
references.title: Model selection and estimation in regression with grouped variables
references.venue: Journal of The Royal Statistical Society Series B-statistical Methodology,
references.authors: Xiaohua Zhai J. Puigcerver Alexander Kolesnikov P. Ruyssen Carlos Riquelme Mario Lucic Josip Djolonga André Susano Pinto Maxim Neumann A. Dosovitskiy L. Beyer Olivier Bachem M. Tschannen Marcin Michalski O. Bousquet S. Gelly N. Houlsby
references.title: The visual task adaptation
references.venue: benchmark. ArXiv,
references.authors: Dinghuai Zhang Kartik Ahuja Yilun Xu Yisen Wang Aaron C. Courville
references.title: Can subnetwork structure be the key to out-of-distribution generalization
references.venue: In ICML,
references.authors: Zheng Zhao L. Wang Huan Liu
references.title: Efficient spectral feature selection with minimum redundancy
references.venue: In AAAI,
references.authors: Wangchunshu Zhou Canwen Xu Tao Ge Julian McAuley Ke Xu Furu Wei
references.title: Bert loses patience: Fast and robust inference with early
references.venue: exit. ArXiv,
references.authors: Zhi-Hua Zhou Jianxin Wu Wei Tang
references.title: Ensembling neural networks: Many could be better than all
references.venue: Artificial Intelligence,
references.authors: Zhuangdi Zhu Kaixiang Lin Jiayu Zhou
references.title: Transfer learning in deep reinforcement learning: A survey
references.venue: arXiv preprint arXiv:2009.07888,
references.authors: Fuzhen Zhuang Zhiyuan Qi Keyu Duan Dongbo Xi Yongchun Zhu Hengshu Zhu Hui Xiong Qing He
references.title: A comprehensive survey on transfer learning
references.venue: Proceedings of the IEEE,
sections.heading: 1 INTRODUCTION
sections.text: Tranfer learning is a widely used method for obtaining strong performance in a variety of tasks where training data is scarce—see Zhu et al. (2020); Alyafeai et al. (2020); Zhuang et al. (2020) for recent application-specific surveys. A well-known recipe for transfer learning involves the supervised or unsupervised pretraining of a model on a source task with a large training dataset (also referred to as upstream training). After pretraining, the model’s output head is discarded, and the rest of the network is used to obtain a feature embedding, i.e., the output of what was formerly the penultimate layer of the network. When transferring to a target task, a new output head is trained on top of the feature extractor (downstream training). This approach makes intuitive sense: if a linear combination of embedding features performs well on the source task, we expect a different linear combination of features to generalize to the target domain, provided the source and target tasks are similar.
  This approach of training a new output head, referred to as linear probing (LINEAR), often yields significant improvements in performance on the target task over training the network from scratch (Kornblith et al., 2019). An alternative to LINEAR is fine-tuning (FINETUNING), which uses targetdomain data to adapt all weights in the feature extractor together with the new output head. This procedure requires doing forward and backward passes through the entire network at each training step and therefore its per-step cost is significantly higher than LINEAR. Furthermore, since the entire network is fine-tuned, the entire set of new weights needs to be stored for every target task, making FINETUNING impractical when working on edge devices or with a large number of target tasks. However, FINETUNING is often preferred over LINEAR since it consistently leads to better performance on a variety of target tasks even when data is scarce (Zhai et al., 2019).
  FINETUNING’s superior generalization in the low-data regime is counterintuitive given that the number of model parameters to be adapted is often large relative to the amount of available training data. How does FINETUNING learn from few examples successfully? We conjecture that FINETUNING better leverages existing internal representations rather than discovering entirely new representations; FINETUNING exposes existing features buried deep in the net for use by the classifier. Under this hypothesis, features needed for transfer are already present in the pretrained network and might be identified directly without fine-tuning the backbone itself.
  In this work, we propose and explore methods for selecting useful features embedded in internal layers of a pretrained net, concatenating them to the embedding produced by the pretrained net, and then applying the LINEAR transfer approach to the augmented representation (Fig. 1-left). Our approach leads to significant improvements over LINEAR as shown in Fig. 1-right. Our key contributions are as follows:
  1. We perform analyses to better understand two factors that affect the benefit of incorporating intermediate representations: feature selection and the degree to which a target domain is out-of-distribution (OOD).
  2. We introduce head-to-toe probing (HEAD2TOE), a simple recipe for selecting relevant features from intermediate representations for few-shot generalization. On the VTAB collection of data sets, we show that HEAD2TOE outperforms LINEAR and matches the performance of more computationally costly FINETUNING.
  3. Critically, HEAD2TOE outperforms FINETUNING on OOD target domains. If a practitioner can make an educated guess about whether a target domain is OOD with respect to a source, using HEAD2TOE improves on the state of the art for transfer learning
sections.heading: 2 PRELIMINARIES
sections.text: Source domain and backbone models. In our experiments, we use source models pretrained on ImageNet-2012 (Russakovsky et al., 2015), a large scale image classification benchmark with 1000 classes and over million natural images. We benchmark HEAD2TOE using convolutional (ResNet-50, Wu et al., 2018) and attention-based (ViT-B/16, Dosovitskiy et al., 2021) architectures pretrained on ImageNet-2012.
  Target domains. In this work, we focus on target tasks with few examples (i.e., few-shot) and use Visual Task Adaptation Benchmark-1k (Zhai et al., 2019) to evaluate different methods. Visual Task Adaptation Benchmark-1k consists of 19 different classification tasks, each having between 2 to 397 classes and a total of 1000 training instances. The domains are grouped into three primary categories: (1) natural images (natural), (2) specialized images using non-standard cameras (specialized), and (3) rendered artificial images (structured).
  Characterizing out-of-distribution domains. Consider the relationship between source and target domains. If the domain overlap is high, then features extracted for linear classification in the source domain should also be relevant for linear classification in the target domain, and LINEAR should yield performance benefits. If the domain overlap is low, then constraining the target domain to use the source domain embedding may be harmful relative to training a model from scratch on
  the target domain. Therefore, we might quantify the source-target distribution shift in terms of how beneficial LINEAR is relative to training a model from scratch (denoted SCRATCH):
  DomainAffinity = AccLINEAR − AccSCRATCH.
  In Fig. 2, we sort the 19 VTAB-1k target tasks by their domain affinity to ImageNet-2012—from low to high—for a pretrained ResNet-50 backbone. The Figure shows examples of the three target domains with the most and least distribution shift (left and right sides, respectively). These examples seem consistent with intuitive notions of distribution shift.
  Baselines. Fig. 2 also presents transfer test accuracy of LINEAR, FINETUNING, and SCRATCH baselines. Consistent with the literature, FINETUNING performs as well or better than the other two baselines. For in-distribution targets (right side of graph), LINEAR matches FINETUNING; for OOD targets (left side of graph), LINEAR is worse than FINETUNING. With distribution mismatch, the source network may filter information available in lower layers because it is not needed for the source task but is essential for the target task. Observing FINETUNING performs better than SCRATCH even in OOD tasks, we hypothesize that intermediate features are key for the FINETUNING, since if learning novel features was possible using limited data, training the network from scratch would work on-par with FINETUNING. Motivated by this observation, HEAD2TOE probes the intermediate features of a network directly and aims to eliminate the need for fine-tuning.
sections.heading: 3 HEAD2TOE UTILIZATION OF PRETRAINED BACKBONES
sections.heading: 3.1 YOUR REPRESENTATIONS ARE RICHER THAN YOU THINK
sections.text: In this section, we conduct a simple experiment to demonstrate the potential of using representations from intermediate layers. We concatenate the feature embedding of a pretrained ResNet-50 backbone (features from the penultimate layer) with features from one additional layer and train a linear head on top of the concatenated features. When extracting features from convolutional layers, we reduce the dimensionality of the convolutional stack of feature maps using strided average pooling, with a window size chosen so that the resulting number of features is similar to the number of embedding features (2048 for ResNet-50).
  In order to estimate an upper bound on how much the performance can be improved over LINEAR by including a single intermediate layer, we use an oracle to select the layer which yields the largest boost in test performance (for each target task separately). Percentage improvement over LINEAR using this ORACLE is shown in Fig. 3-left. We observe a negative correlation (-0.745, Spearman)
  between the domain affinity of a target task and the accuracy gain. As predicted earlier, adding intermediate representations does not improve in-domain generalization because feature embedding already contains the most useful features. In contrast, generalization on out-of-domain tasks are improved significantly when intermediate features are used.
  In Fig. 3-top-right, we show test accuracy as a function of the ResNet-50 layer whose internal representation is used to augment the feature embeddings to boost LINEAR transfer. Figures for remaining tasks can be found in Appendix D. Different tasks on VTAB benchmark prefer inclusion of different layers to obtain the best generalization accuracy, emphasizing the importance of selecting the right set of features for optimal generalization. Overall, the ORACLE that selects the layer with best test performance for each task yields an average of 3.5% improvement on VTAB-1k benchmark. One possible explanation for the improvement in performance with the augmented representation is simply that it has more degrees of freedom (4096 features instead of 2048). To demonstrate that the improvement is due to inclusion of intermediate layers and not simply due to increased dimensionality, we formed a 4096-element feature representation by using a second feature embedding obtained from a second ResNet-50 backbone pretrained on ImageNet-2012 (CONTROL). Note that this experiment bears similarity to ensembling (Zhou et al., 2002), which is known to bring significant accuracy gains on its own (Mustafa et al., 2020). Utilizing a second backbone doubles the amount of resources required, yet it falls 1% shy of the Oracle performance demonstrating the extent to which using intermediate representations can be helpful for generalization.
sections.heading: 3.2 HEAD2TOE
sections.text: Motivated by our observations in the previous section, we hypothesize that we can attain—or possibly surpass—the performance of FINETUNING without changing the backbone itself by using LINEAR augmented with well-chosen intermediate activations. However, this approach introduces the problem of selecting features. The number of possible features to include is large relative to the amount of training data available, leading to the potential of overfitting. In this section, we explore various regularization and feature selection methods, which we then incorporate into our method, HEAD2TOE.
  Notation. Our method applies to any network with any type of layers, but here, for simplicity and without loss of generality, we consider a network with L fully connected layers, each layer receiving input from the layer below:
  z` = h`−1W` ; h` = f(z`) (1)
  where the subscript denotes a layer index, h0 = x is the input, f is the activation function, W` is the weight matrix of layer `, and zL is the logit vector used for classification. When transferring a pretrained network to a target task using LINEAR, we discard the last layer of the pretrained network and train a new set of linear weights, W ′L, such that predictions (logits) for the new task are obtained by z′L = hL−1W ′ L.
  Head2Toe. Consider a simple scheme that augments the backbone embedding with activations from all layers of the network, such that:
  z′L = hallWall ; hall = concat(a1(h1), a2(h2), ..., aL(hL)) (2)
  where a`(.) denotes a fixed function to reduce the dimensionality of the activation vector at a given layer `. Such functions become valuable when considering network architectures that generate large number of intermediate features. For example, for convolutional networks, where the representations at each layer are structured as multi-channel spatial arrays of detectors, we perform two-dimensional average pooling as explained in Section 3.1 and Section 5. Even with dimensionality reduction, hall can exceed a million elements, and Wall is underconstrained by the training data, leading to overfitting. Further, without care Wall may become so large as to be impractical for deploying this model.1. We can address these issues by selecting a subset of features before training the targetdomain classifier. Equivalently, we can zero out the non-selected rows of Wall. We will show that appropriate selection of a subset of features leads to better generalization than using all features.
  Feature selection based on group lasso. Group lasso (Yuan & Lin, 2006) is a popular method for selecting relevant features in multi-task adaptation settings (Argyriou et al., 2007; Nie et al., 2010). When used as a regularizer on a weight matrix W , the group-lasso regularizer encourages the `2 norm of the rows of the matrix to be sparse, and is defined as:
  |W |2,1 = |s|1 = ∑ i |si| ; si = √∑ j w2ij (3)
  To determine which features are most useful for the task, the linear head is trained with group-lasso regularization on Wall. In contrast to the approach taken by Argyriou et al. (2007) and Nie et al. (2010), which use costly matrix inversions, we incorporate the regularization as a secondary loss and train with stochastic gradient descent. Following training, the `2 norms of the rows of Wall, s, are used to decide which features are most relevant. We refer to si as the relevance score for feature i. We pick a fraction F of all features with the greatest relevance and train a new linear head to obtain the final logit mapping. Feature selection alone provides strong regularization, therefore during the final training we don’t use any additional regularization.
  We make two comments on this. First, because the initial round of training Wall with the group-lasso regularizer is used only to rank features by importance, the method is quite robust to the regularization coefficient; it simply needs to be large enough to distinguish the contribution of the individual features. Second, interpreting si as the importance of feature i depends on all features having the same dynamic range. This constraint will often be satisfied naturally due to the normalization layers found in the architecture itself (we read out representations after normalization). To cover the remaining cases (where no normalization exists between two layers) we normalize the representations of each layer to a unit norm.
  Selection of F and the cost of HEAD2TOE. The fraction F determines the total number of features retained. One would expect the optimal value to depend on the target task. We select F based on validation performance. This validation procedure is inexpensive compared to the cost of the initial phase of the algorithm (i.e., training of Wall to obtain s) due to the reduced number of features in the second step. Overall, HEAD2TOE together with its validation procedure requires 18% more operations compared to training Wall alone (details shared in Appendix B).
  Verifying HEAD2TOE. In Fig. 4-left, we demonstrate the effectiveness of group lasso on identifying relevant intermediate features of a ResNet-50 for transfer from ImageNet. We rank all features
  1For example, using a pooling size of 2, ResNet-50 generates 1.7 million features and storing Wall requires 6.6e8 parameters (2.6GB for float32) for SUN-397.
  0 5000 10000 15000 20000 25000 Offset
  0.54
  0.56
  0.58
  0.60
  Te st
  A cc
  ur ac
  y Linear
  Oracle
  102 103 104 #Features Selected 0.46
  0.48
  0.50
  0.52
  0.54
  0.56
  0.58
  0.60
  Te st
  A cc
  ur ac
  y
  Linear
  Oracle
  feature-wise layer-wise layer-wise (group)
  0
  100
  200
  300
  400 Feature (Acc: 63.8%)
  .ro ot b1 u1 b1 u2 b1 u3 b2 u1 b2 u2 b2 u3 b2 u4 b3 u1 b3 u2 b3 u3 b3 u4 b3 u5 b3 u6 b4 u1 b4 u2 b4 u3
  pr el
  og its 0 pr el og its lo gi ts
  0
  500
  1000
  1500
  2000 Layer (Acc: 44.1%)
  Figure 4: (left) Over all VTAB tasks, average accuracy of HEAD2TOE when selecting 2048 consecutive features sorted by their relevance score, starting with an index specified by offset. (center) Average accuracy over all VTAB tasks as a function of the number of features included. In this experiment, we only use features from the first layer of each of the 18 ResNet blocks and adjust pooling to have around 2048 features for each layer, totalling 29800 features. Results show that selecting layers performs worse than selecting features when adapting to a target domain. (right) Distribution of 2048 intermediate features retained from a ResNet-50 when using feature-wise and layer-wise scores on the SVHN transfer task.
  by their relevance score, si, and we select groups of 2048 consecutive features beginning at a particular offset in this ranking. Offset 0 corresponds to selecting the features with largest relevance. We calculate average test accuracies across all VTAB tasks. As the figure shows, test accuracy decreases monotonically with the offset, indicating that the relevance score predicts the importance of including a feature in the linear classifier.
  HEAD2TOE selects individual features independent of the layer in which they are embedded. We compare this feature-wise strategy to selecting layers as whole (i.e., selecting all features in a layer or none). One might expect layer-wise selection to be superior because it involves fewer decisions and therefore less opportunity to overfit to the validation set that is being used for selection. Further, layer-wise selection may be superior because the relevance of one feature in a layer may indicate the relevance of others. To obtain a layer-wise relevance score, we compute the mean relevance score of all features in a layer and then rank layers by relevance. We also run an alternative layer selection algorithm, in which we group weights originating from the same layer together and select layers using the `2 norm of the groups (layer-wise (group)). Fig. 4-center compares feature-wise and layer-wise selection methods, matched for number of features retained. Feature-wise selection performs better than layer-wise selection on average and the best performance is obtained when around 1000 features are kept. We share figures for all 19 datasets in Appendix E. Fig. 4-right shows the distribution of features selected over different layers for both strategies.
  Note that HEAD2TOE’s use of a fixed backbone means that as we search for features to include, the extracted features themselves are not changing. Consequently, we can calculate them once and reuse as necessary, instead of recalculating at every step, as required for FINETUNING. Furthermore, since the backbone is frozen, the only additional weight storage required for each target task is the output head. Such properties are key for transfer-learning methods to be practical when using models with billions of parameters.
sections.heading: 4 RELATED WORK
sections.text: Transfer Learning is studied extensively in the literature and used widely in practice. Most similar to our work is ELMo (Peters et al., 2018), which averages intermediate representations of a language model (2 LSTM embeddings) using a learned linear combination (a softmax). ELMo requires embeddings to be same size and is most similar to the suboptimal layer-selection baseline shown in Fig. 4-center. Peters et al. (2019) compared FINETUNING with LINEAR and reported a small gap in performance for ELMo and BERT models. Kornblith et al. (2019) showed that ImageNet-2012 performance correlates highly with LINEAR and FINETUNING. Using a large pool of pretrained models, Renggli et al. (2020) examined various metrics to decide which backbone to transfer. Given the ever increasing size and never saturating performance of pretrained models, the importance of reducing the cost of FINETUNING models is stated in the literature regularly. Methods like featurewise transformations (Bilen & Vedaldi, 2017), residual adapters (Houlsby et al., 2019; Rebuffi et al.,
  2017), Diff-pruning (Guo et al., 2021) and selective fine-tuning (Guo et al., 2019; Fu et al., 2021) are proposed in order to reduce the cost of storing fine-tuned models. Zhang et al. (2021) showed that selecting a sub-network of a pretrained model can increase OOD generalization. However, none of these methods match the simplicity of training (and storing) a linear classifier. Teerapittayanon et al. (2016), Kaya et al. (2019), and Zhou et al. (2020) studied intermediate representations to reduce ”overthinking” and thus provide better early-exit criteria. Similarly Baldock et al. (2021) showed a correlation between early classification of a sample and how easy its classification is.
  Alternatives to transfer learning exist to learn from few labeled examples. Meta-learning (Thrun & Pratt, 1998; Schmidhuber, 1987; Schmidhuber et al., 1997) is used (among other things) to tackle the few-shot learning problem (i.e learning from limited data) from a learning-to-learn perspective; see Hospedales et al. (2020) for a comprehensive survey. In-between the representation learning and learning-to-learn paradigms lie approaches that use multi-domain training as an inductive bias, either without (Dvornik et al., 2020; Triantafillou et al., 2021; Li et al., 2021b;a) or with (Liu et al., 2021) meta-learning. However, in the large-scale setting FINETUNING remains a top-performing approach to few-shot classification (Dumoulin et al., 2021).
  Feature selection aims to find the most relevant features from the input and is studied extensively in the machine-learning literature. Approaches can be grouped according to whether labeled data is used—supervised (Nie et al., 2010) or unsupervised (He et al., 2005; Balın et al., 2019; Atashgahi et al., 2020)—or what high-level approach is taken—filter methods (Blum & Langley, 1997), wrapper methods (Kohavi & John, 1997), or embedded methods (Yuan & Lin, 2006). Most relevant to our work are embedded supervised methods as they have good scaling properties while achieving the best performance which is vital in our setting with over a million features. Embedded supervised feature selection methods use a cost function to iteratively refine the subset of features selected and popular approaches include forward selection (Viola & Jones, 2001; Borboudakis & Tsamardinos, 2019), backward selection (pruning) (Mozer & Smolensky, 1989; Guyon et al., 2004) and regularization/feature-ranking based methods (Yuan & Lin, 2006; Blum & Langley, 1997; Zhao et al., 2010). Most relevant to our work is Argyriou et al. (2007); Nie et al. (2010), both of which uses `2,1 regularization to select features, however their approach requires matrix inversions which is not practical in our setting. We point interested readers to the survey of Gui et al. (2017) and book of Boln-Canedo et al. (2015) for a detailed discussion on the topic.
sections.heading: 5 EVALUATING HEAD2TOE
sections.text: We evaluate HEAD2TOE on the VTAB-1k benchmark using two popular vision architectures, ResNet-50 (Wu et al., 2018) and ViT-B/16 (Dosovitskiy et al., 2021), both pretrained on ImageNet2012. ResNet-50 consists of 50 convolutional layers. To utilize the intermediate convolutional features, we aggregate spatial information using average pooling on non-overlapping patches, as explained in Section 3. We adjust the pooling size to target a fixed dimensionality of the representation. For example, for a feature map of shape 20 × 20 × 128 and a target size of 512, we average-pool disjoint patches of size 5 × 5, resulting in 4 features per channel and 1024 features in total. This helps us to balance different layers in terms of number features they contribute to the concatenated embedding. We normalize features coming from each layer to a unit norm. This scaling preserves the relative magnitude of features within a layer while accounting for inter-layer differences and works better than normalizing each feature separately or not normalizing at all. ViT-B/16 consists of 12 multi-headed self-attention layers. When we aggregate the output of the self-attention layers, we perform 1-D average pooling over the patch/token dimension choosing the pooling size to match a target number of features as before. Given that token dimension is permutation invariant, 1-D average pooling might not be the best choice here and better aggregation function can provide further gains. We share a detailed list of intermediate representations utilized for each architecture in Appendix H.
  HEAD2TOE selects a subset of features and trains a linear classifier without regularization on top of the selected features. We compare HEAD2TOE with regularization baselines that utilize all features without doing feature selection. These baselines are denoted as +All-`1, +All-`2 and +All-`2,1 according to the norm they use.
  We perform five-fold cross validation for each task and method in order to pick the best hyperparameters. All methods search over the same learning rates and training steps (two values of
  each). Methods that leverage intermediate features (i.e., regularization baselines and HEAD2TOE) additionally search over regularization coefficients and the size of the aggregated representation at each layer. The FINETUNING baseline searches over 4 hyper-parameters; thus the comparison of HEAD2TOE, which searches over 24 values, to fine-tuning might seem unfair. However, this was necessary due to fine-tuning being significantly more costly than training a linear classifier, even with intermediate features. We repeat each evaluation using 3 different seeds and report median values and share standard deviations in Appendix C. More details on hyper-parameter selection and best values can be found in Appendix A.
sections.heading: 5.1 RESNET-50
sections.text: The top half of Table 1 presents results on the 19 VTAB-1k target domains when transferring from a pretrained ResNet-50 architecture. On average, HEAD2TOE slightly outperforms all other methods, including FINETUNING (see rightmost column of Table). HEAD2TOE, along with the regularization baselines that use intermediate layers, is far superior to LINEAR, indicating the value of the intermediate layers. And HEAD2TOE is superior to the regularization baselines, indicating the value of explicit feature selection. Among the three categories of tasks, HEAD2TOE excels relative to the other methods for the specialized category, but does not outperform FINETUNING for the natural and structured categories.
  Does HEAD2TOE select different features for each task? Which layers are used more frequently? In Appendix F, we show the distribution of features selected across different layers and the amount of intersection among features selected for different tasks and seeds. We observe a high variety among layers and features motivating the importance of performing feature selection for each task separately. HEAD2TOE exceeds FINETUNING performance, but requires only 0.5% of FLOPs during training on average. Similarly, the cost of storing the adapted model is reduced to 1% on average. We discuss HEAD2TOE’s computational and storage costs in detail in Appendix B.
sections.heading: 5.2 VIT-B/16
sections.text: Results for ViT-B/16 are shared in the bottom half of Table 1. As with the ResNet-50 architecture, HEAD2TOE achieves the best accuracy among methods that keep the backbone fixed: HEAD2TOE improves accuracy over LINEAR by about 10% on average (in absolute performance), and HEAD2TOE outperforms the regularization baselines that include intermediate features but that do not explicitly select features. The story for HEAD2TOE versus FINETUNING is a bit more complicated. The FINETUNING recipe used by Dosovitskiy et al. (2021) includes many tweaks such as selective data-augmentation, image upsampling, learning rate warm-up, and gradient clipping. In order to make a direct comparison with LINEAR, in the next-to-last row of the Table we report the performance of FINETUNING without all of these tweaks; HEAD2TOE obtains about a 1% absolute accuracy improvement over FINETUNING in this setting. However, when some of the tweaks are incorporated into FINETUNING—in particular gradient clipping—FINETUNING achieves a 3% improvement over HEAD2TOE. Since it is a linear classifier, adding gradient clipping to HEAD2TOE doesn’t improve the results. That being said, HEAD2TOE retains its competitive advantage in terms of resource costs and its accuracy would likely similarly benefit from the development of more sophisticated tricks (notably moving beyond the naive 1-D pooling of ViT features), much like FINETUNING has gained over time.
sections.heading: 5.3 SCALING HEAD2TOE
sections.text: In Fig. 5 we vary the number of intermediate features used for each of our pretrained backbones. We observe that including all layers always performs better on average. However when varying the number of target features for each layer, we observed 2 distinct sub-group of target tasks that behave differently as the number of features increases. This observation informed our discussion to include both small and large target sizes in our validation hyper parameter search.
sections.heading: 6 CONCLUSION
sections.text: In this work, we introduced HEAD2TOE, an approach that extends linear probing (LINEAR) by selecting the most relevant features among a pretrained network’s intermediate representations. We show that doing so greatly improves performance over LINEAR and allows the approach to reach a performance competitive with—and in some cases superior to—FINETUNING. Our findings challenge the conventional belief that FINETUNING is required to achieve good performance on OOD tasks. While more work is needed before HEAD2TOE can realize the full computational benefits of linear probing, our work paves the way for applying new and more efficient feature selection approaches and for experimenting with HEAD2TOE probing in other domains such as regression, video classification, object detection, reinforcement learning, and language modelling domains.
sections.heading: B COST OF HEAD2TOE
sections.text: We evaluate different values of F and pick the value with best validation performance. Cost of HEAD2TOE consists of three parts: (1) CI : Cost of calculating the representations using the pretrained backbone. (2) Call: cost of training the initial head Wall (in order to obtain si’s) (3)∑
  f CF=f : total cost of validating different values of F . Cost of validating a fraction value f , assuming equal number of training steps, is equal to Cf = Call ∗ f . Therefore relative cost of searching for F is equal to the sum of fractions validated (in comparison to the initial training of (s)).
  In Table 4, we compare cost of running HEAD2TOE adaptation with FINETUNING. HEAD2TOE uses the backbone once in order to calculate the representations and then trains the Wall, whereas FINETUNING requires a forward pass on the backbone at each step. Therefore we calculate the cost
  of finetuning for t steps as CI · t. Similarly, cost of HEAD2TOE is calculated as CI + Call · t. The overall relative cost of Call increases with number of classes C and number of features considered
  N . As shown in Table 1-top, HEAD2TOE obtains better results than FINETUNING, yet it requires 0.5% of the FLOPs needed on average.
  After adaptation, all methods require roughly same number of FLOPs for inference due to all methods using the same backbone. The cost of storing models for each task becomes important when the same pre-trained backbone is used for many different tasks. In Table 4 we compare the cost of storing different models found by different methods. A finetuned model requires all weights to be stored which has the same cost as storing the original network, whereas LINEAR and HEAD2TOE requires storing only the output head: Wlinear. HEAD2TOE also requires to store the indices of the features selected using a bitmap. Even though HEAD2TOE considers many more features during adaptation, it selects a small subset of the features and thus requires a much smaller final classifier (on average 1% of the FINETUNING). Note that hyper parameters are selected to maximize accuracy, not the size of the final model. We expect to see greater savings with an efficiency oriented hyper-parameter selection.
sections.heading: C STANDARD DEVIATIONS FOR TABLE 1
sections.text: In Table 5, we share the standard deviations for the median test accuracies presented in Table 1. On average we observe LINEAR obtains lower variation as expected, due to the limited adaptation and convex nature of the problem. Head2Toe seem to have similar (or less) variation then the other regularization baselines that use all features.
sections.heading: D ADDITIONAL PLOTS FOR EXPERIMENTS USING SINGLE ADDITIONAL LAYER
sections.text: Test accuracies when using a single additional intermedieate layer from a pretrained ResNet-50 backbone are shown in Fig. 6. natural datasets (except SVHN) are highly similar to upstream dataset (ImageNet-2012) and thus adding an extra intermediate layer doesn’t improve performance much. However performance on OOD tasks (mostly of the tasks in structured category) improves significantly when intermediate representations are used, which correlates strongly with datasets in which HEAD2TOE exceeds FINETUNING performance.
sections.heading: E ADDITIONAL PLOTS FOR LAYER/FEATURE-WISE SELECTION COMPARISON
sections.text: We compare layer-wise selection strategy discussed in Section 3 to HEAD2TOE in Fig. 7. For allmost all datasets, feature-wise selection produces better results. Retinopathy and Flowers-102 are the only two datasets where the layer-wise strategy performs better.
sections.heading: F ADDITIONAL PLOTS FOR RESNET-50 RESULTS
sections.text: Improvement of HEAD2TOE over fine-tuning test accuracy for ResNet-50 backbone is shown in Fig. 8. Similar to earlier plots, we observe a clear trend between being OOD and improvement in accuracy: HEAD2TOE obtains superior few-shot generalization for most of OOD tasks. We also share the distribution of features selected for each task in Fig. 10. Since different tasks might have different number features selected, we normalize each plot such that bars add up to 1. Overall, features from later layers seem to be preferred more often. Early layers are preferred, especially for OOD tasks like Clevr and sNORB. We observe a diverse set of distributions for selected features, which reflects the importance of selecting features from multiple layers. Even when distributions match, HEAD2TOE can select different features from the same layer for two different tasks. Therefore, next, we compare the indices of selected features directly to measure the diversity of features selected across tasks and seeds.
  Transfer across Tasks In Fig. 9-left we investigate how features selected using a task i performs when evaluated on a task j. Each pixel represents the average accuracy over 3 seeds. For each column we subtract the diagonal term (i.e. self-transfer, i = j) to obtain delta accuracy for each
  dataset. For most tasks, using a separate task for feature selection hurts performance. Results in Flowers-102 and Pets get better when other datasets like sNORB-Elev is used. Crucially no single dataset (rows of Fig. 9-left) seems to get best transfer, which highlights the importance of doing the feature selection during the adaptation (not beforehand). Note that, in practice (and in the literature) evaluation for each task is done in isolation, i.e. the data from other tasks are not available.
  Similarity of features selected In Fig. 9-right we investigate the intersection of features selected by HEAD2TOE. We select 2048 features for each task from the pool of 29800 features (same experiments as in Fig. 4-center). For each task HEAD2TOE selects a different subset of features. We calculate the fraction of features shared between each subset. For each target task we run 3 seeds resulting in 3 sets of features kept. When comparing the similarity across seeds (diagonal terms), we average over 6 possible combinations among these 3 sets. Similarly, when comparing two different datasets we average over 9 possible combinations. Results show that features selected by complementary tasks like dSprites-Loc and dSpr-Orient (or sNORB tasks) have high overlap. Both dSprites and sNORB images are generated artificially and have similar characteristics. This is reflected in the heat-map (about 40% overlap). Similarly, features selected for CIFAR-100 and Caltech101 tasks seem to have high overlap.
  Features selected also vary across different runs (seeds). This variation seem to be higher for highly in-domain datasets like Flowers102 and Pets, which seems to correlate with HEAD2TOE’s poor performance. Similarly dSprites-Loc seems to have higher variance across different seeds, which correlates with the outlier performance of HEAD2TOE on this task when compared with FINETUNING (see Fig. 8). We believe that understanding and reducing the inconsistency among different runs can be a promising direction to improve head-to-toe utilization performance.
  Apart from a small fraction of datasets, most datasets seem to share less than 20% of the features, which highlights the importance of doing the feature selection for each target task separately.
  Effect of Training Data In Fig. 11, we compare the performance of HEAD2TOE with other baselines using reduced training data. Fraction (df )=1 indicates original tasks with 1000 training samples. For other fractions we calculate number of shots for each class as int(1000 ∗ fd/C) where C is the number of classes and then sample a new version of the task accordingly. For SUN-397 task, number of shots are capped at 1 and thus fractions below 0.75 lead to 1-shot tasks and thus results are all the same. Overall we observe the performance of HEAD2TOE improves with amount of data available, possibly due to the reduced noise in feature selection.
sections.heading: G ADDITIONAL PLOTS FOR SCALING BEHAVIOUR OF HEAD2TOE
sections.text: Scaling behaviour of HEAD2TOE when using different feature target size and number of layers over 19 VTAB-1k tasks is shown in Fig. 12.
  0.0
  0.2
  0.4
  KITTI-Dist sNORB-Azim sNORB-Elev
  0.0
  0.2
  0.4
  Clevr-Dist DMLab dSpr-Loc dSpr-Orient
  0.0
  0.2
  0.4
  EuroSAT Resisc45 Retinopathy Clevr-Count
  Pets Sun397 SVHN Camelyon
sections.heading: H DETAILS OF INTERMEDIATE REPRESENTATIONS INCLUDED
sections.text: ResNet-50 has 5 stages. First stage includes a single convolutional layer (root) followed by pooling. We include features after pooling. Remaining stages include 3,4,6 and 3 bottleneck units (v2) each. Bottleneck units start with normalization and activation and has 3 convolutional layers. We includes features right after the activation function is applied, resulting in 3 intermediate feature sets per unit. Output after 5 stages are average-pooled and passed to the output head. We include features before and after pooling. with the output of the final layer (logits), total number of locations where features are extracted makes 52.
  ViT-B/16 models consists of multiple encoder units. Each attention head consists of a self attention module and followed by 2 MLPs. For each encoder unit, we include features (1) after layer-norm (but before self-attention) (2) features after self-attention (3-4) features after MLP layers (and after gelu activation function). Additionally we use patched image (i.e. tokenized image input), pre-logits and logits.
  103 104 0.83
  0.84
  0.85
  0.86
  0.87
  Ta rg
  et S
  iz e
  Caltech101
  103 104 0.41
  0.42
  0.43
  0.44
  0.45 CIFAR-100
  103 104
  0.50
  0.55
  0.60
  Clevr-Dist
  103 104
  0.40
  0.45
  0.50
  0.55 Clevr-Count
  103 104 0.730
  0.735
  0.740
  Retinopathy
  103 104
  0.42
  0.44
  0.46
  Ta rg
  et S
  iz e
  DMLab
  103 104
  0.3
  0.4
  dSpr-Orient
  103 104
  0.45
  0.50
  0.55
  0.60
  dSpr-Loc
  103 104
  0.62
  0.63
  0.64
  0.65
  DTD
  103 104
  0.935
  0.940
  0.945
  EuroSAT
  103 104
  0.82
  0.84
  0.86
  Ta rg
  et S
  iz e
  KITTI-Dist
  103 104 0.78
  0.79
  0.80
  0.81
  Flowers102
  103 104 0.905
  0.910
  0.915
  0.920
  Pets
  103 104
  0.83
  0.84
  0.85
  Camelyon
  103 104 0.70
  0.72
  0.74
  0.76
  Resisc45
  103 104
  0.4
  0.5
  0.6
  0.7
  Ta rg
  et S
  iz e
  sNORB-Azim
  103 104 0.32
  0.34
  0.36
  0.38
  0.40
  sNORB-Elev
  103 104 0.26
  0.27
  0.28
  0.29
  0.30
  Sun397
  103 104
  0.5
  0.6
  0.7
  0.8
  SVHN
  #Target Features Per Layer
  n_layers
  1/16 1/4 all
  103 104 0.750
  0.775
  0.800
  0.825
  Ta rg
  et S
  iz e
  Caltech101
  103 104 0.48
  0.50
  0.52
  0.54
  CIFAR-100
  103 104 0.475
  0.500
  0.525
  0.550
  0.575
  Clevr-Dist
  103 104 0.400
  0.425
  0.450
  0.475
  Clevr-Count
  103 104
  0.73
  0.74
  0.75 Retinopathy
  103 104 0.38
  0.40
  0.42
  0.44
  Ta rg
  et S
  iz e
  DMLab
  103 104 0.24
  0.26
  0.28
  0.30
  0.32 dSpr-Orient
  103 104 0.30
  0.35
  0.40
  0.45
  0.50 dSpr-Loc
  103 104
  0.55
  0.60
  DTD
  103 104 0.92
  0.93
  0.94
  0.95
  EuroSAT
  103 104
  0.78
  0.80
  0.82
  0.84
  Ta rg
  et S
  iz e
  KITTI-Dist
  103 104
  0.65
  0.70
  0.75
  0.80
  Flowers102
  103 104
  0.85
  0.86
  0.87
  0.88
  Pets
  103 104
  0.83
  0.84
  0.85
  Camelyon
  103 104
  0.70
  0.75
  0.80 Resisc45
  103 104 0.3
  0.4
  0.5
  0.6
  Ta rg
  et S
  iz e
  sNORB-Azim
  103 104 0.33
  0.34
  0.35
  0.36
  0.37
  sNORB-Elev
  103 104 0.26
  0.28
  0.30
  Sun397
  103 104 0.4
  0.5
  0.6
  0.7
  0.8 SVHN
  #Target Features Per Layer
  n_layers
  1/16 1/4 all
  Figure 12: Scaling Behaviour over 19 VTAB-1k tasks when varying feature target size and number of layers utilized for (top) ResNet-50 and (bottom) ViT-B/16