summaries: The paper introduces a representation consolidation method to properly aggregate the pre-trained knowledge from multiple teachers for transfer learning. It claims that a generalist teacher is necessary for preserving the transferability of distilled representations, and task-specific teachers contribute to better performance in the same-domain downstream tasks. The introduced method performs on par with multi-task training while neglecting the need for teacher datasets. - The authors propose a method of learning a consolidated image feature representation from a collection of related task-specific teachers that transfer well on novel recognition tasks.  - To achieve it, the authors utilize multi-teacher multi-task model distillation framework that jointly distills one or several task-specific teachers with a generalist one (trained with Imagenet dataset).   - Each teacher is set to operate on a different set of classes, and a multi-head student is trained to emulate all teachers.  - Experimental results show that the proposed method doesn’t need to revisit original training data of each teacher, but gains better performance.  The paper proposes a task-agnostic Knowledge Distillation method that includes a generalist model as an additional teacher, to limit student forgetting and representation collapse. The authors call this method representation consolidation and claim it can benefit both related and unrelated downstream tasks. The authors experiment with ImageNet and iFood image classification tasks, the accuracy of the proposed method is generally higher than using Knowledge Distillation only. This paper studies the problem of multi-model consolidation, which aims to merge (consolidate) the knowledge from multiple models into one model so that we can better transfer this merged model for downstream tasks. Different from the traditional knowledge distillation or multi-teacher distillation, the proposed method aims to transferability of merged models for the downstream tasks, whilst achieving good performance on source tasks. The key technique (technical contribution) is to add a generalized model during the knowledge distillation, which can be regarded as s "special" teacher model.    The authors conduct extensive experiments on multiple datasets and most of the experiments are under a few-shot linear probe transfer learning setting.
abstractText: A library of diverse expert models transfers better to a novel task than a single generalist model. However, growing such a library indefinitely is impractical. Hence, we explore the problem of learning a consolidated image feature representation from a collection of related task-specific teachers that transfer well on novel recognition tasks. This differs from traditional knowledge distillation in which a student model is trained to emulate the input/output functionality of a teacher. Indeed, we observe experimentally that standard distillation of task-specific teachers, or using these teacher representations directly, reduces downstream transferability compared to a task-agnostic generalist model. We show that a simple multihead, multi-task distillation method using an unlabeled proxy dataset and adding a generalist teacher is sufficient to consolidate representations from task-specific teacher(s). We improve downstream performance, outperforming the teacher (or best of all teachers) as well as the strong baseline of ImageNet pre-trained features. Our method almost reaches the performance of a multi-task joint training oracle, reaping the benefit of the teachers without replaying their training data.
references.authors: Alessandro Achille Michael Lam Rahul Tewari Avinash Ravichandran Subhransu Maji Charless C Fowlkes Stefano Soatto Pietro Perona
references.title: Task2vec: Task embedding for meta-learning
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Sravanti Addepalli Gaurav Kumar Nayak Anirban Chakraborty Venkatesh Babu Radhakrishnan
references.title: Degan: Data-enriching gan for retrieving representative samples from a trained classifier
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Gustavo Aguilar Yuan Ling Y. Zhang Benjamin Yao Xing Fan Edward Guo
references.title: Knowledge distillation from internal representations
references.venue: In AAAI,
references.authors: Victor Besnier Himalaya Jain Andrei Bursuc M. Cord
references.title: P’erez. This dataset does not exist: Training models from generated images
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Kartikeya Bhardwaj Naveen Suda R. Marculescu
references.title: Dream distillation: A data-independent model compression framework
references.venue: In Joint Workshop on On-Device Machine Learning & Compact Deep Neural Network Representations, NeurIPS Workshop,
references.authors: Lukas Bossard Matthieu Guillaumin Luc Van Gool
references.title: Food-101 – mining discriminative components with random forests
references.venue: In European Conference on Computer Vision,
references.authors: R. Caruana
references.title: Multitask learning
references.venue: In Encyclopedia of Machine Learning and Data Mining,
references.authors: Rudrasis Chakraborty Chun-Hao Yang B. Vemuri
references.title: A mixture model for aggregation of multiple pre-trained weak classifiers
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW),
references.authors: Akshay Chawla Hongxu Yin Pavlo Molchanov J. Álvarez
references.title: Data-free knowledge distillation for object detection
references.venue: In WACV,
references.authors: Hanting Chen Yunhe Wang Chang Xu Zhaohui Yang Chuanjian Liu Boxin Shi Chunjing Xu Chao Xu Qi Tian
references.title: Dafl: Data-free learning of student
references.authors: Gong Cheng Junwei Han Xiaoqiang Lu
references.title: Remote sensing image scene classification: Benchmark and state of the art
references.venue: Proceedings of the IEEE,
references.authors: Yi-Min Chou Yi-Ming Chan Jia-Hong Lee Chih-Yi Chiu Chu-Song Chen
references.title: Unifying and merging well-trained deep neural networks for inference stage
references.venue: In Proceedings of the 27th International Joint Conference on Artificial Intelligence,
references.authors: M. Cimpoi Subhransu Maji I. Kokkinos S. Mohamed A. Vedaldi
references.title: Describing textures in the wild
references.venue: IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Aditya Deshpande Alessandro Achille Avinash Ravichandran Hao Li Luca Zancato Charless Fowlkes Rahul Bhotika Stefano Soatto Pietro Perona
references.title: A linearized framework and a new benchmark for model selection for fine-tuning
references.venue: arXiv preprint arXiv:2102.00084,
references.authors: Gongfan Fang J. Song Chengchao Shen X. Wang D. Chen Mingli Song
references.title: Data-free adversarial distillation
references.venue: ArXiv, abs/1912.11006,
references.authors: Zhiyuan Fang Jianfeng Wang Lijuan Wang L. Zhang Yezhou Yang Zicheng Liu
references.title: Seed: Self-supervised distillation for visual representation
references.authors: J. Gao Zijian Guo Z. Li R. Nevatia
references.title: Knowledge concentration: Learning 100k object classifiers in a single cnn
references.authors: Robin Geyer Luca Corinzia Viktor Wegmayr
references.title: Transfer learning by adaptive merging of multiple models
references.venue: In MIDL,
references.authors: G. Griffin Alex Holub P. Perona
references.title: Caltech-256 object category dataset
references.venue: Technical report, California Institute of Technology,
references.authors: Neel Guha Ameet S. Talwalkar Virginia Smith
references.title: One-shot federated learning
references.venue: Machine Learning on Devices Workshop at NeurIPS,
references.authors: S. Guo Weilin Huang X. Zhang Prasanna Srikhanta Yin Cui Y. Li M. Scott Hartwig Adam Serge J. Belongie
references.title: The imaterialist fashion attribute dataset
references.venue: IEEE/CVF International Conference on Computer Vision Workshop (ICCVW),
references.authors: Matan Haroush Itay Hubara E. Hoffer Daniel Soudry
references.title: The knowledge within: Methods for data-free model compression
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Kaiming He X. Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Geoffrey E. Hinton Oriol Vinyals J. Dean
references.title: Distilling the knowledge in a neural network
references.venue: ArXiv, abs/1503.02531,
references.authors: Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui C. Sun Alexander Shepard Hartwig Adam P. Perona Serge J. Belongie
references.title: The inaturalist species classification and detection dataset
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Wenpeng Hu Z. Lin B. Liu Chongyang Tao Zhengwei Tao Jinwen Ma Dongyan Zhao Rui Yan
references.title: Overcoming catastrophic forgetting for continual learning via model adaptation
references.venue: In ICLR,
references.authors: Parneet Kaur Karan Sikka Weijun Wang serge Belongie Ajay Divakaran
references.title: Foodx-251: A dataset for fine-grained food classification
references.authors: J. Kirkpatrick Razvan Pascanu Neil C. Rabinowitz J. Veness G. Desjardins Andrei A. Rusu K. Milan John Quan Tiago Ramalho Agnieszka Grabska-Barwinska D. Hassabis C. Clopath D. Kumaran R. Hadsell
references.title: Overcoming catastrophic forgetting in neural networks
references.venue: Proceedings of the National Academy of Sciences,
references.authors: Animesh Koratana Daniel Kang Peter Bailis M. Zaharia
references.title: Lit: Learned intermediate representation training for model compression
references.venue: In ICML,
references.authors: Jonathan Krause Michael Stark Jia Deng Li Fei-Fei
references.title: 3d object representations for fine-grained categorization
references.venue: In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia,
references.authors: Sang-Woo Lee Lee Jin-Hwa Kim Jaehyun Jun Jung-Woo Ha Byoung-Tak Zhang
references.title: Overcoming Catastrophic Forgetting by Incremental Moment Matching (IMM)
references.venue: In Advances In Neural Information Processing Systems
references.authors: Zhizhong Li Derek Hoiem
references.title: Learning without forgetting
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Iou-Jen Liu J. Peng A. Schwing
references.title: Knowledge flow: Improve upon your teachers
references.venue: 7th International Conference on Learning Representations,
references.authors: Raphael Gontijo Lopes Stefano Fenu T. Starner
references.title: Data-free knowledge distillation for deep neural networks. ArXiv
references.authors: Liangchen Luo M. Sandler Zi Lin A. Zhmoginov Andrew G. Howard
references.title: Large-scale generative data-free distillation
references.venue: ArXiv, abs/2012.05578,
references.authors: Subhransu Maji Esa Rahtu Juho Kannala Matthew B. Blaschko A. Vedaldi
references.title: Fine-grained visual classification of aircraft
references.venue: ArXiv, abs/1306.5151,
references.authors: P. Micaelli A. Storkey
references.title: Zero-shot knowledge transfer via adversarial belief matching
references.venue: In NeurIPS,
references.authors: G.K. Nayak K.R. Mopuri V. Shaj R.V. Babu A. Chakraborty
references.title: Zero-shot knowledge distillation in deep networks
references.venue: In International Conference on Machine Learning,
references.authors: Maria-Elena Nilsback Andrew Zisserman
references.title: Automated flower classification over a large number of classes
references.venue: Sixth Indian Conference on Computer Vision, Graphics & Image Processing,
references.authors: Tribhuvanesh Orekondy B. Schiele Mario Fritz
references.title: Knockoff nets: Stealing functionality of blackbox models
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Seonguk Park Nojun Kwak
references.title: Feature-level ensemble knowledge distillation for aggregating knowledge from multiple networks
references.venue: In ECAI,
references.authors: Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga Alban Desmaison Andreas Kopf Edward Yang Zachary DeVito Martin Raison Alykhan Tejani Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala
references.title: Pytorch: An imperative style, high-performance deep learning library
references.venue: In Advances in Neural Information Processing Systems
references.authors: F. Pedregosa G. Varoquaux A. Gramfort V. Michel B. Thirion O. Grisel M. Blondel P. Prettenhofer R. Weiss V. Dubourg J. Vanderplas A. Passos D. Cournapeau M. Brucher M. Perrot E. Duchesnay
references.title: Scikit-learn: Machine learning in Python
references.venue: Journal of Machine Learning Research,
references.authors: Ameya Prabhu P. Torr P. Dokania
references.title: Gdumb: A simple approach that questions our progress in continual learning
references.venue: In ECCV,
references.authors: Joan Puigcerver Carlos Riquelme Basil Mustafa Cedric Renggli André Susano Pinto Sylvain Gelly Daniel Keysers Neil Houlsby
references.title: Scalable transfer learning with expert models
references.authors: Sylvestre-Alvise Rebuffi Alexander Kolesnikov G. Sperl Christoph H. Lampert
references.title: icarl: Incremental classifier and representation learning
references.venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Yonglong Tian Dilip Krishnan Phillip Isola
references.title: Contrastive representation distillation
references.venue: In International Conference on Learning Representations,
references.authors: J. Vongkulbhisal Phongtharin Vinayavekhin M.V. Scarzanella
references.title: Unifying heterogeneous classifiers with distillation
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: C. Wah S. Branson P. Welinder P. Perona S. Belongie
references.title: The Caltech-UCSD Birds-200-2011 Dataset
references.venue: Technical Report CNS-TR-2011-001, California Institute of Technology,
references.authors: Guodong Xu Ziwei Liu Xiaoxiao Li Chen Change Loy
references.title: Knowledge distillation meets selfsupervision
references.venue: In ECCV,
references.authors: Shoukai Xu Haokun Li Bohan Zhuang Jing Liu Jiezhang Cao Chuangrun Liang Mingkui Tan
references.title: Generative low-bitwidth data free quantization
references.venue: In ECCV,
references.authors: I.Z. Yalniz H. Jégou K. Chen Manohar Paluri D. Mahajan
references.title: Billion-scale semi-supervised learning for image classification
references.venue: ArXiv, abs/1905.00546,
references.authors: Jingwen Ye Yixin Ji X. Wang Xin Gao Mingli Song
references.title: Data-free knowledge amalgamation via group-stack dual-gan
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Hongxu Yin Pavlo Molchanov Zhizhong Li Jose M. Alvarez Arun Mallya Derek Hoiem N. Jha J. Kautz
references.title: Dreaming to distill: Data-free knowledge transfer via deepinversion
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Jaemin Yoo Minyong Cho T. Kim U. Kang
references.title: Knowledge extraction with no observable data
references.venue: In NeurIPS,
references.authors: Friedemann Zenke Ben Poole S. Ganguli
references.title: Continual learning through synaptic intelligence
references.venue: Proceedings of machine learning research,
references.authors: X. Zhao S. Schulter G. Sharma Yi-Hsuan Tsai Manmohan Chandraker Ying Wu
references.title: Object detection with a unified label space from multiple datasets
references.venue: In ECCV,
references.authors: B. Zhou À. Lapedriza A. Khosla A. Oliva A. Torralba
references.title: Places: A 10 million image database for scene recognition
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
sections.heading: 1 INTRODUCTION
sections.text: A promising approach to scale transfer learning to diverse downstream vision tasks is to maintain a library of diverse experts pre-trained on different tasks. (Deshpande et al., 2021) When presented with a novel downstream task, one can select an appropriate expert and fine-tune the representation with small amounts of task-specific data. This strategy has many practical benefits: fine-tuning a task-relevant pre-trained representation is fast, there is no need to store or revisit expert pre-training data. How is such a library of expert representations populated and maintained? Previous work (e.g., Puigcerver et al. (2020); Achille et al. (2019); Deshpande et al. (2021)) has assumed a static collection of experts created by training on domain-specific datasets or domain-specific subsets of general dataset. Instead, we would like to automatically enrich the diversity of the expert library by accumulating knowledge from transferred downstream tasks, so that the overall system performance continually increases over time (life-long meta-learning).
  One approach to growing the library would be to add fine-tuned downstream task-specific models back into the library as a candidate expert for transfer on future tasks. Such a naı̈ve approach clearly does not scale and, at a minimum, requires developing techniques for selecting experts that is sub-linear in the size of the library (Puigcerver et al., 2020; Achille et al., 2019; Deshpande et al., 2021). More importantly, as our experiments show in Sec. 4, task-specific models fine-tuned on small amounts of data do not provide transferable representations. Task-specific models tend to overspecialize and degrade under-utilized features in their representations, and thus under-perform on new tasks compared to generic pre-trained models.
  To address this, we introduce representation consolidation where the goal is to consolidate knowledge from multiple task-specific teacher models into a single expert student representation which transfers to downstream tasks better than any of the individual teacher representations. There are three factors that make the problem of representation consolidation unique relative to the existing literature: (1) we assume a representation is to be consolidated from a collection of multiple taskspecific models, (2) we avoid the need to revisit task-specific training data of these models, and (3) we focus on improving downstream transfer instead of simply replicating performance on the upstream teacher’s task(s). To the best of our knowledge, no prior work has analyzed the downstream transfer aspect of distilled domain-expert representations.
  To carry out representation consolidation, we utilize multi-teacher multi-task model distillation (see Fig. 1). We use a simple yet effective method to jointly distill one or several task-specific teacher with a generalist one. Each teacher may operate on a different set of classes, and a multi-head student is trained to emulate all teachers. Previous work on knowledge distillation has focused on the student model’s performance on the teacher’s task. Instead, we evaluate how well the student representation generalizes to new downstream tasks (whether related or unrelated to the teachers’ tasks). In this new setting we demonstrate several surprising results:
sections.heading: 2 REPRESENTATION CONSOLIDATION
sections.text: Problem statement. We start with a collection of one or more task-specific image classification models {Mit}Ni=1 as teachers, trained on corresponding datasets {Dit } belonging to some domain (satellite images, images of flowers, etc.). We assume models consist of a feature extractor or backbone φit (·), composed with a classifier head hit (·) so thatMit = hit (φit (·)). We consolidate the knowledge of these task-specific teachers into a single student representation φs(·) using a proxy dataset Dproxy (e.g., ImageNet), and then evaluate φs(·) by training a SVM on top of it (or fine-tuning it) on a given downstream Djd chosen from some set {D j d}. Our goal is that the resulting downstream model hjd(φ j d(·)) achieves good performance, where φ j d denotes the student φs as-is or potentially after tuning on Djd . Fig. 1 highlights how this differs from standard distillation in which the student model hs(φs(·)) is simply evaluated on the same task its teachers were once trained to perform.
  Forgetting and representation collapse during distillation. Though our problem statement is novel, can existing approaches solve our problem statement? We observe (Sec. 4) that neither the task-specific teachers nor students distilled with standard or state-of-the-art methods provide sufficiently transferable representations. They under-perform general pre-training representations (e.g. with ImageNet) when evaluated on downstream tasks from the teachers’ domain, and drastically under-perform on tasks outside the teachers domain. Surprisingly, this holds true even though we use ImageNet-pretrained weights to initialize the teacher and student networks. Thus, we argue that training task-specific teachers or distilling from them suffers from catastrophic forgetting of general knowledge that is crucial for transfer learning.
  Intuitively, transfer performance depends on how distinguishable different classes are represented in the penultimate layer feature space. In standard distillation, the student only learns from taskspecific teachers trained on smaller datasets and is only required to discriminate the classes in those datasets. This is well suited for the traditional distillation problem that evaluates the student on these same tasks. But for representation consolidation, distinguishing unknown downstream classes requires preserving general features that may not be relevant to the teachers’ specific task. Our strategy is thus to ensure that the student maintains general features as it learns task-specific ones.
  Method formulation. A key challenge to our approach is avoiding replay of the images used to train the task-specific teachers. To overcome this, we propose using a proxy dataset and a generalist model that transfers well (e.g. one pre-trained on ImageNet). This model is referred to as a generalist as it is not task specific and contains a wide variety of classes, and use it to help avoid forgetting. More specifically, we use multi-head multi-task distillation. We put N + 1 heads, h0s , . . . , h N s , on top of the student backbone φs(·). In addition to the task-specific teachers hit (φit (·)), i ∈ {1, . . . , N} used in traditional distillation, we also include the generalist teacher trained on ImageNet (denoted as h0t (φ 0 t (·))). The rationale behind this construction is that we learn task knowledge from the task-specific model on the proxy dataset, but also force the student to retain knowledge from the generalist teacher to prevent over-specialization. We learn the student by optimizing the loss:
  L = ∑
  x∈Dproxy
  ( λ0 Ldistill ( h0t (φ 0 t (x)), h 0 s (φs(x)) )︸ ︷︷ ︸ generalist teacher term + N∑ i=1 λi Ldistill ( hit (φ i t (x)), h i s(φs(x)) )︸ ︷︷ ︸ traditional distillation ) (1)
  with a distillation loss Ldistill. Our key difference from traditional distillation is the generalist teacher term. Setting λ0 = 0 while keeping all other conditions (proxy dataset, network initialization, etc.) yields a standard multi-teacher distillation baseline. Note that this is also distinct from ensemble knowledge distillation, where one averages the homogeneous output of every model and distills the mean onto a single student head, since our teacher outputs are heterogeneous (different tasks).
  Our method is agnostic to the underlying distillation method. We can use the standard knowledge distillation (KD) loss (Hinton et al., 2015) which is cross-entropy with temperature T = 2, i.e. Ldistill(pt, ps) = − ∑C c=1 p (c) t log(p (c) s ) where c indexes the C classes, and pt =
  softmax ( hit (φ i t (x))/T ) , ps = softmax ( his(φs(x))/T ) . Alternatively, we can use a state-of-theart distillation method, e.g., CRD (Tian et al., 2020). CRD uses the KD loss and adds a contrastive embedding loss between hiCRD,t(φ i t (x)) and h i CRD,s(φs(x)) with a pair of trainable linear embedding layers hiCRD,t and h i CRD,s that are used only in training and discarded afterwards. The teacher φ i t ’s
  are trained on completely different tasks, so for CRD we use different embedding layers for each teacher-student-head pair.
  We initialize the student backbone φs and its 0-th head h0s using the generalist model’s weights φ0t , h 0 t , whereas other heads are randomly initialized. Since it is important to maintain pre-trained model’s representational power, we simply set the loss weights λ0 = 1 and λi = 1N for 1 ≤ i ≤ N . This forces the learned representation φs to balance between learning general and taskspecific features, which benefits future downstream transfer. After training the student we evaluate the resulting representation φs(·) on multiple downstream tasks {Did}. For each task j we can either fine-tune the whole model hjd(φ j d(·)) or keep the student representation fixed and only learn the classifier head hjd(φs(·)) which is often referred to as a linear probe.
sections.heading: 3 EXPERIMENTAL SETUP
sections.text: Datasets and downstream tasks. We utilize datasets from a variety of domains to generate teachers and downstream tasks: Cars196 (Krause et al., 2013), Resisc45 (Cheng et al., 2017) (remote sensing images), iFood (Kaur et al., 2019) and Food101 (Bossard et al., 2014), iFashion (Guo et al., 2019), DTD (Cimpoi et al., 2014) (describable textures), iNaturalist (Horn et al., 2018) (species classification, 2019 challenge version), CUB Birds (Wah et al., 2011), Flowers (Nilsback & Zisserman, 2008), Caltech256 (Griffin et al., 2007), and Aircrafts (Maji et al., 2013). Among these, iFood and Food101 are the same domain, and Birds and Flowers are subdomains of iNaturalist. We checked for near-duplicates between these datasets using perceptual hash (Buchner, 2021), and found negligible duplication: 1 out of 134k iNaturalist (50% classes we used) is a duplicate of CUB, and 8 out of 130k/100k images are duplicates between iFood and Food101.
  To evaluate if a consolidated student representation has learned features relevant to a specific domain, we require downstream tasks that are related (in the same domain) to that of each task-specific teacher. Except for Food101 and iFashion, we split each dataset at random into two disjoint sets which each contain only 50% of the classes. We take the first half of the dataset, named “t-split”, and use as Dit , to train a task-specific teacher. We use the second half of each dataset, named “dsplit”, as one of the downstream tasks Djd . We primarily evaluate on few-shot downstream transfer, where we randomly sample 5 training images from each available class in Djd , but always use the entire test set for evaluation. We use all samples for those classes with < 5 images. For iFashion, which is a multi-task multi-label dataset, we use all images but with a subset of labels (e.g., those related to clothing category) as Dit for teacher training. We then use all images with a disjoint set of labels (e.g., those related to sleeve style) as a downstream task Djd for evaluating transfer. For iFashion’s few-shot scenario, we randomly subsample 1000 images for downstream training, and evaluate on all test data. We do not train any teacher on Food101 so the complete set of classes are used as a downstream task.
  During distillation and consolidation, we use ImageNet (Russakovsky et al., 2015) (ILSVRC12) as the proxy dataset Dproxy for most experiments. However, we also show results with Places365standard (Zhou et al., 2018) or iNaturalist (Horn et al., 2018) as the proxy.
  Compared methods and Criteria. As a reminder, our goal is not to duplicate upstream teacher predictions, but rather to improve transfer performance of distilled representations. Therefore, we evaluate each student’s φs using its performance when transferred to various downstream datasets {Djd} and compare multiple methods for initializing the downstream representation φ j d:
  (1) ImageNet-pretrained φ0t , a strong baseline for transfer learning. (2) ImageNet-pretrained φ 0 t fine-tuned on the soft-labels produced by the ImageNet-pretrained model with batchnorms in test mode. This baseline aims to isolate the effect of soft-labels / self-distillation on model performance. (3) The task-specific teacher φ1t (or one of the teachers when N > 1) without further distillation. (4) Distillation with N teachers using either standard knowledge distillation (KD) or contrastive representation distillation (CRD). This distills only task-specific teacher(s) on Dproxy, without the ImageNet teacher h0t (φ 0 t (·)). (5) Our consolidated representation φs which includes h0t (φ0t (·)) as a teacher, using either KD or CRD as the underlying distillation loss. (6) A multi-task learning oracle (MTL) that jointly trains on ImageNet and teacher data Dt rather than on unlabeled proxy data.
  For comparison fairness, (2-6) are all ImageNet pre-trained (i.e. initialized with (1)) before further training. Both (4) and (5) are using the same proxy dataset (i.e. ImageNet for most experiments).
  We use the transfer accuracy on downstream tasks Djd to measure each representation’s power. We primarily use the few-shot linear probe (train a 5-shot linear SVM as hjd over fixed φ) on D j d’s training set (single training run for full dataset, few-shot performance averaged over 50 random subsampling trials). We also verify our results hold when fine-tuning the student representation hjd(φd(·)) on few-shot or full dataset splits.
  Implementation details. We will release our code to reproduce this paper upon acceptance. For experimental fairness, we use the same network (ResNet50 (He et al., 2016)) / hyperparameters / ImageNet initialization for the baselines and ours. For more details, please see appendix A.
sections.heading: 4 RESULTS
sections.text: Exp. 0: Motivational analysis – traditional distill vs. representation consolidation To highlight the difference between representation consolidation and traditional distillation, we use either iFood or Resisc45 (t-split) as D1t to train teachers, and run distillation/consolidation with N = 1. We test on iFood (d-split), Food101 (full), and Resisc45 (d-split) as downstream tasks (t-split and d-split are disjoint 50% classes of each dataset; see Section 3)
  On downstream tasks Djd , we follow representation learning’s evaluation protocol (few-shot linear probe): train linear SVM hd on φ(x), x ∈ Djd , evaluate on D j d’s test set. We also evaluate each network following the traditional distillation evaluation protocol, i.e. directly evaluate h1t (φ 1 t (·)) or h1s (φs(·)) on the upstream task D1t ’s test set. The results are shown in Table 2. If we only focus on the upstream task D1t , then traditional (proxy) distillation almost matches the teacher’s performance, and representation consolidation performs worse than both. However, if we instead focus on the downstream transfer performance onDjd , we see the opposite trend. In Table 2a with iFood representations, for the teacher-related downstream tasks Food101 and iFood d-split (the disjoint classes split from the same dataset as D1t ), we see a benefit of consolidation over both taskspecific teacher and generic pre-trained features. On the unrelated downstream task (Resisc45), where the generalist teacher excels, we notice that our method almost matches the generalist performance even though we improve over it for the food tasks. We see a similar trend in Table 2b: the consolidated student outperforms the teacher and the generalist on the teacher-related task (Resisc45 d-split) but still retains the generalist teacher’s performance on unrelated food tasks. This clearly demonstrates the significant difference between upstream and downstream transfer.
  We perform nearly the same as the MTL oracle which trains also on the teacher datasets Dt, indicating that while we avoid replayingDt, our simple consolidation method withDproxy can extract nearly all of its benefit. Note that although we have carefully optimized the MTL oracle for downstream performance, adding state-of-the-art MTL techniques could outperform our MTL implementation.
  We only demonstrate that it is not trivial to outperform our method using original teacher training data.
  Exp. 1: Improving student representation when N = 1. We show that this advantage of consolidation over baselines holds for a wide range of upstream and downstream datasets. We also compare to CRD to show state-of-the-art distillation often underperforms even when specifically targeting representation learning. Fig. 2 summarizes few-shot SVM accuracy using different representations relative to ImageNet-pretrained (See appendix for the full Fig. 6 and raw numbers).
  The conclusions are similar regardless of using KD or CRD – consolidation outperforms (Cars196, Resisc45, iFood, CUB, Aircrafts, iNaturalist teachers) or matches (iFashion, DTD, Flowers, Caltech256 teachers) ImageNet pre-trained model performance on related downstream tasks, and matches its performance on unrelated ones. In contrast, both traditional distillation methods (1) underperform consolidation on related downstream tasks for all teachers except Cars196, iFashion, and Aircraft, and (2) drastically underperform both consolidated and ImageNet features on unrelated downstream tasks – although CRD is better than KD, it still degrades performance. Notably, traditional distillation methods underperform ImageNet even on related downstream tasks for Resisc45, DTD, Caltech256, and (for KD only) Flowers teachers. We match the MTL oracle performance across the board, reaping the teacher dataset’s benefit without replaying it.
  Exp. 2: Consolidating representations withN > 1. We can merge multiple task-specific teacher representations that are related to get a better one. In addition, our method is not constrained to merging models with the same architecture or φit ’s feature space dimensions like Geyer et al. (2019). To illustrate this, we split the “t-split” of Cars196 and iFood into five random splits containing 10% of the original classes. We train a ResNet18 teacher on each of the five splits. Then, we use either traditional distillation or representation consolidation to merge these models with a ResNet50 generalist teacher. We compare the resulting representations with the teacher model’s and ImageNet. Fig. 3a shows this result. For downstream tasks, we obtain better performance than using only one of the five teachers on both related and unrelated tasks, especially for iFood and Food101 where the teachers themselves underperform on related downstream task. This shows that our method can benefit from even teachers whose representation is weaker, as long as they have domain knowledge.
  Full comparison with traditional distillation and representation consolidation from only one of the five teachers are in the appendix – we gain performance on similar Djd by using five task-specific teachers instead of one, and we outperform both traditional distill methods on related downstream
sections.heading: Accuracy (rel. to ImageNet-pretrained model)
sections.text: tasks for iFood/Food101. We also show results when using all ResNet50 teachers for the five splits in the appendix. The gaps are closer but the results are similar and the conclusions are identical.
  Finally, we explore merging models from different domains to form a multi-domain consolidated student. See Fig. 3b. We observe that we can outperform ImageNet on all related downstream tasks, but the performance gain is smaller than representation consolidation in just one domain. We show in the appendix that ours+KD outperforms KD on most downstream tasks except Cars196.
  Exp. 3: Fine-tuning downstream. We also verify that our conclusions generalize to fine-tuning as well, especially without few-shot sampling. Fig. 4 shows an excerpt of our results that include full dataset transfer. See appendix for fine-tuning using few-shot and full d-splits, whose conclusions are the same as this section’s. Fine-tuning allows the representation to change into one that better suits the downstream task, so the benefit of teachers’ domain knowledge shrinks compared to fixed φs with linear SVM h j d. Despite this, the conclusions are the same as the fixed φs scenario.
  Exp. 4: Influence of the Loss weights. Fig. 5a shows the effect of the choice of loss weight on our method – when we use λ0 = 12 , λi = 3 2N (i > 0), we gain performance on the related downstream task but lose performance on unrelated ones, whereas using λ0 = 32 , λi = 1
  2N (i > 0) gives us the opposite. We can potentially use this trade-off to suppress unreliable teachers’ influence.
  Exp. 5: Proxy data choice. Fig. 5b shows results replacing ImageNet with Places365 or iNaturalist as the Dproxy dataset. Places365 yields similar performances as using ImageNet, while a narrower-scope iNaturalist is weaker on unrelated downstream tasks. Jointly training on Places365 labels is not helpful, meaning that the improvement comes from the generalist teacher, not the data.
sections.heading: 5 RELATED WORK
sections.text: Our goal is to maximize the transferability of a representation consolidated from multiple teachers. This downstream transfer aspect has received little attention in the literature (Liu et al., 2019; Geyer et al., 2019; Tian et al., 2020), but this problem formulation is closely related to prior work on multi-model merging and distillation with proxy data.
  Multi-model merging. It is useful to combine separate models that perform different tasks into a single model for efficiency and performance benefits. Knowledge Concentration (Gao et al., 2017) combine teachers trained on subsets of 100k classes in the EFT dataset (Gao et al., 2017) into one improved single model, using handcrafted sparse connections for the final student layers. Chou et al. (2018) merge CNNs by combining the kernel weights of different models by clustering and lookup, followed by fine-tuning. Zhao et al. (2020) merge object detection models when some classes may be the background to other models. Vongkulbhisal et al. (2019) combine models with overlapping classes using the combined train set, by deriving soft labels from intra-model class correspondence. Ye et al. (2020) progressively train a GAN to regenerate proxy data for all teachers, and combine teachers layer by layer. Chakraborty et al. (2018) and Park & Kwak (2020) aggregate ensembles of members trained on the same task. In one-shot federated learning (Guha et al., 2019), multiple clients with their own private data train a model each, and finally merge the models using an ensemble or distillation in a way that protects their data privacy.
  Unlike our approach, these methods merge models in order to perform exactly the same task as the teacher models, are not concerned with the performance of the student when transferred to a downstream task, and often (Gao et al., 2017; Chou et al., 2018; Zhao et al., 2020; Vongkulbhisal et al., 2019; Chakraborty et al., 2018; Park & Kwak, 2020) require revisiting the original training images. We have shown that these teachers themselves have poor transferability compared to a simple pre-trained baseline, and learning only from them (even using state-of-the-art CRD (Tian et al., 2020)) yields suboptimal transfer learning performance.
  Multi-model merging for transfer. Knowledge Flow (Liu et al., 2019) connects the student to multiple teachers’ intermediate layers to kick-start target task training, and gradually penalizes its reliance on teacher models over time. Geyer et al. (2019) uses IMM (Lee et al., 2017) to merge multiple models using their diagonal approximated Fisher information matrix (FIM) to balance different
  teacher’s weight importance, merge, and fine-tune. Computing the FIM requires reprocessing the original teacher training data (unlike our approach that only needs generic proxy data). Furthermore, this method requires homogeneous architectures for students and teachers, while ours works on any architecture combinations. These methods directly optimize performance on the downstream task and thus require re-running for each different target dataset. Our approach is more efficient, as it only requires consolidating teachers once to improve the pre-trained representation independently of the downstream task. Finally, we note that the representations learned were not compared to a strong baseline (i.e., pre-training on ImageNet), which we argue is a prerequisite for being useful in realworld applications. One of our main contributions is observing the need for including a generalist teacher, an insight which is orthogonal to, and could be combined with these previous approaches.
  Distillation with representation losses (“representation distillation”) tries to capture additional structure of feature representations by aligning student and teacher feature activations during distillation. Koratana et al. (2019) compress models by adding L2 losses between intermediate representations of the teacher and students. Aguilar et al. (2020) use KL divergence and cosine similarity to make the attention and representation of intermediate features of the student and teacher similar. Tian et al. (2020) adds contrastive representation learning loss to the penultimate layer to preserve feature structures, by maximizing each image’s student and teacher features’ mutual information. Despite their focus on yielding better representations, these methods fully rely on the teacher’s transferability. We show that this yields downstream performance worse than generalist baselines.
  Proxy data (“data-free”) distillation transfers the input-output function of a teacher network to a student network without using the teacher’s training data. (Yalniz et al., 2019; Orekondy et al., 2019) opt to use a large general proxy dataset to query the teacher, and their teacher outputs to on this data to train the student. Other methods (Nayak et al., 2019; Chen et al., 2019; Haroush et al., 2020; Chawla et al., 2021) generate proxy data directly from the trained models, and use this data to train the students. Further, Micaelli & Storkey (2019); Yin et al. (2020) also encourage generating samples the student and teacher disagree on. Lastly, other methods require the original dataset to compute meta-data information such as feature cluster mean. (Lopes et al., 2017; Bhardwaj et al., 2019) Some train a GAN from the teachers to maximize chosen class predictions (Fang et al., 2019; Yoo et al., 2019; Ye et al., 2020), sometimes also batchnorm statistics (Luo et al., 2020; Xu et al., 2020b), and sometimes on proxy data instead. (Addepalli et al., 2020; Besnier et al., 2020) Some work combines self-supervised learning with distillation (Tian et al., 2020; Xu et al., 2020a; Fang et al., 2021) to improve the representation performance on the original task.
  These works try to emulate the original data for standard distillation, and do not concern the student’s transferability or merging multiple teachers. Our simple method already performs comparably to a multi-task oracle that uses the original teacher dataset, indicating that these more complex methods that improve upstream task distillation may be unnecessary when the goal is downstream transfer.
  Incremental learning is also related to our overall goal of growing a library of expert representations. These methods continually learn tasks or classes but often with limited access to past training data (Li & Hoiem, 2018; Rebuffi et al., 2017; Kirkpatrick et al., 2017; Zenke et al., 2017; Hu et al., 2019; Yin et al., 2020; Prabhu et al., 2020). Our approach addresses many of the same challenges by consolidating knowledge in the form of feature representations without revisiting old data used to train teachers, but our “increments” are whole tasks given in the form of trained models instead of data, whose labels may overlap. Multi-task learning (Caruana, 1998) assumes all datasets are available and trains jointly on them whereas we assume task-specific datasets absent. We perform comparably to this oracle.
sections.heading: 6 SUMMARY
sections.text: In this paper, we show that traditional distillation can result in a representation suboptimal for downstream task transfer learning, because it only focuses on preserving the end-to-end input-output mapping of the old task. We propose representation consolidation with the generalist model as an additional teacher. Our method preserves the wide-range transferability of the strong ImageNet baseline and improve the performance for both related and unrelated downstream tasks over traditionally distilled networks. We show that we can merge multiple models in the same domain to get a better representation than any single model.
sections.heading: B COMPLETE FIGURES FOR SECTION 4 EXPERIMENTS
sections.text: We now provide the full graphs and table for our experiments. Note that we have summarized these results and all conclusions in the main paper.
  Exp. 1: Improving student representation when N = 1. Fig. 6 and Table 3 show full results for Fig. 2. Please see the main paper for analysis and conclusions.
  Exp. 2: Consolidating representations with N > 1. Figs. 7, 8, 9 shows full results for Fig. 3.
  For Figs. 7 and 8 same-domain model merging, in addition to the main paper results merging ResNet18 φit and ResNet50 φ 0 t , we show similar results for all teachers being ResNet50 in Fig. 8. We also show comparison to traditional distill with 5 teachers, representation consolidation with only 1 teacher, and the teacher itself (best out of 5 according to related Djd performance). The conclusions are the same, and merging five teachers using representation consolidation outperforms all baselines on both related and unrelated downstream tasks (except Cars196 with related Djd against traditional distill).
  For multi-domain model merging in Fig. 9, we compare to traditional distill with multi-task learning. We outperform or match it on downstream Djd except for Cars196. We also explore using a concatenation of multiple large unlabeled datasets as Dproxy. With a more diverse proxy, the performance
sections.heading: Accuracy (rel. to ImageNet-pretrained model)
sections.heading: Teacher: iNaturalist (t-split) model
sections.heading: Accuracy (rel. to ImageNet-pretrained model)
sections.heading: Teacher: Cars196+Resisc45+iFood (t-split) model
sections.heading: Accuracy (rel. to ImageNet-pretrained model)
sections.heading: Accuracy (rel. to ImageNet-pretrained model)
sections.heading: Accuracy (rel. to ImageNet-pretrained model)
sections.text: drops a little for related Djd and stays similar for unrelated D j d , suggesting we are somewhat insensitive to choice of datasets, but a more diverse proxy data may not provide better model merging performance.
sections.heading: Accuracy (rel. to ImageNet-pretrained model)
sections.heading: Accuracy (rel. to ImageNet-pretrained model)
sections.text: Exp. 3: Fine-tuning downstream. Figs. 10, 11 show full results for Fig. 4. These extensive results show that in both few-shot and full-shot scenarios, and both single-domain and multi-domain scenarios, we have the same conclusions as the fixed representation experiments.
  Exp. 5: Proxy data choice. Fig. 12 shows full results for Fig. 5b. This shows similar results with different teacher Dt, i.e. Places365 performs similarly to ImageNet as the proxy, but the narrowerscope iNaturalist underperforms.
sections.heading: C RAW NUMBER ACCURACY TABLES FOR ALL EXPERIMENTS
sections.text: • Table 4: fixed representation few-shot results for Figs. 2, 3 (Figs. 6,7,8,9 in the appendix). • Table 5: fixed representation few-shot results for Fig. 5. • Table 6: fine-tuning few-shot results for Fig. 10 in the appendix. • Table 7: fine-tuning full-shot results for Fig. 4 (Figs. 11 in the appendix.)
sections.heading: D LIMITATIONS
sections.text: We assume we have perfect knowledge of which tasks form the same domain and which tasks do not belong to a domain. Our performance drops when teachers from different domains are consolidated. In future work, we plan to automatically determine how to cluster a large amount of teachers into domains. In addition, one of our contributions assume the existence of a strong representation baseline such as the ImageNet pre-trained model, which is true for fields like images and language, but not for others (e.g., 3D reconstruction). Our method also takes the teachers as is and learn from them, and any mistakes made by the teachers can be propagated during the consolidation. Possible mitigations include using a better teacher that makes less such mistakes, or using regularization on both teacher and student training, such as making similar inputs map to similar outputs.