summaries: This work proposes a Hybrid Neural Pareto Front (HNPF) framework to solve multi-objective optimization problems. The proposed method needs to first generate a huge number of feasible solutions to cover the whole decision space (e.g., random sampling in this work), and then use a two-stage approach to select the Pareto optimal solutions. In the first stage, it builds a Fritz-John Condition (FJC) based neural network to verify the weak Pareto optimality for given solutions. In the second stage, it further filters the truly strong Pareto optimal subset from the obtained solutions. Experiments have been conducted on several low-dimensional problems to validate the proposed method's performance.

summaries: This paper has two contributions. First it argues for the need to benchmark multi-objective problems  using analytic functions for convex and non-convex conditions. Second, it presents a two-stage method for finding the Pareto front.

summaries: This paper intends to find points on the Pareto frontier of a multiobjective optimization problem. Despite other baseline methods for this problem, they claim that their proposed algorithm is suitable for non-convex optimization, and their solutions are evenly spread across the frontier. Also, their algorithm can handle constraints for the MOO problems similar to solutions proposed in existing operation research methods. Empirical evaluations show that the proposed algorithm can find Pareto points and omit non-dominated points in the final stage.

summaries: The authors present key shortcomings of MTL solvers in addition to Hybrid Neural Pareto Front (HNPF) that aims to handle non-convex functions and constraints.  The authors claim the following contributions: - New strategy for weak Pareto front identification based on Fritz-John conditions. - New Pareto filter to remove dominated points from the weak Pareto front.

abstractText: Pareto solutions are optimal trade-offs between multiple competing objectives over the feasible set satisfying imposed constraints. Fixed-point iterative strategies do not always converge and might only return one solution point per run. Consequently, multiple runs of a scalarization problem are required to retrieve a Pareto front, where all instances converge. Recently proposed Multi-Task Learning (MTL) solvers claim to achieve Pareto solutions combining Linear Scalarization and domain decomposition. We demonstrate key shortcomings of MTL solvers, that limit their usability for real-world applications. Issues include unjustified convexity assumptions on practical problems, incomplete and often wrong inferences on datasets that violate Pareto definition, and lack of proper benchmarking and verification. We propose a two stage Pareto framework: Hybrid Neural Pareto Front (HNPF) that is accurate and handles non-convex functions and constraints. The Stage-1 neural network efficiently extracts the weak Pareto front, using Fritz-John Conditions (FJC) as the discriminator, with no assumptions of convexity on the objectives or constraints. An FJC guided diffusive manifold is used to bound the error between the true and the Stage-1 extracted weak Pareto front. The Stage-2, low-cost Pareto filter then extracts the strong Pareto subset from this weak front. Numerical experiments demonstrates the accuracy and efficiency of our approach.

authors:
Ananth Balashankar
Alyssa Lees
Chris Welty
Lakshminarayanan Subramanian

title: What is fair? exploring pareto-efficiency for fairness constrained classifiers

title: K-d trees for semidynamic point sets

venue: In Proceedings of the sixth annual symposium on Computational geometry,

authors:
Michele Benzi
Andrew J Wathen

title: Some preconditioning techniques for saddle point problems

venue: In Model order reduction: theory, research aspects and applications,

authors:
Michele Benzi
Gene H Golub
Jörg Liesen

title: Numerical solution of saddle point problems

venue: Acta numerica,

authors:
Roberto Calandra
Jan Peters
MP

title: Deisenrothy. Pareto front modeling for sensitivity analysis in multi-objective bayesian optimization

venue: In NIPS Workshop on Bayesian Optimization,

title: Approximation with artificial neural networks

venue: Faculty of Sciences, Etvs Lornd University, Hungary,

authors:
Indraneel Das
John E Dennis

title: Normal-boundary intersection: A new method for generating the pareto surface in nonlinear multicriteria optimization problems

venue: SIAM journal on optimization,

authors:
Kalyanmoy Deb
Amrit Pratap
Sameer Agarwal
TAMT Meyarivan

title: A fast and elitist multiobjective genetic algorithm: Nsga-ii

venue: IEEE transactions on evolutionary computation,

title: The mnist database of handwritten digit images for machine learning research [best of the web

venue: IEEE Signal Processing Magazine,

authors:
Joydeep Dutta
C Yalçın Kaya

title: A new scalarization and numerical method for constructing the weak pareto front of multi-objective optimization problems

authors:
Carlos M Fonseca
Peter J Fleming

title: Multiobjective optimization and multiple constraint handling with evolutionary algorithms. ii. application example

venue: IEEE Transactions on systems, man, and cybernetics-Part A: Systems and humans,

title: Multiple objective decision making-methods and applications: A state-of-the art survey: Ching-lai hwang and abu syed md

venue: masud springer, berlin,

authors:
A Ghane-Kanafi
E Khorram

title: A new scalarization method for finding the efficient frontier in non-convex multi-objective problems

venue: Applied Mathematical Modelling,

authors:
Massimiliano Gobbi
F Levi
Gianpiero Mastinu
Giorgio Previati

title: On the analytical derivation of the pareto-optimal set with applications to structural design

venue: Structural and Multidisciplinary Optimization,

authors:
Daniel Hernández-Lobato
Jose Hernandez-Lobato
Amar Shah
Ryan Adams

title: Predictive entropy search for multi-objective bayesian optimization

venue: In International Conference on Machine Learning,

authors:
Cho-Jui Hsieh
Nagarajan Natarajan
Inderjit Dhillon

title: Pu learning for matrix completion

authors:
Nazan Khan
David E Goldberg
Martin Pelikan

title: Multi-objective bayesian optimization algorithm

venue: In Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Francesco Levi
Massimiliano Gobbi

title: An application of analytical multi-objective optimization to truss structures. In 11th AIAA/ISSMO multidisciplinary analysis and optimization conference

authors:
Xi Lin
Hui-Ling Zhen
Zhenhua Li
Qing-Fu Zhang
Sam Kwong

title: Pareto multi-task learning

authors:
Xiao Lin
Hongjie Chen
Changhua Pei
Fei Sun
Xuanji Xiao
Hanxiao Sun
Yongfeng Zhang
Wenwu Ou
Peng Jiang

title: A pareto-efficient algorithm for multiple objective optimization in e-commerce recommendation

venue: In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys),

title: The mythos of model

venue: interpretability. Queue,

authors:
Pingchuan Ma
Tao Du
Wojciech Matusik

title: Efficient continuous pareto exploration in multi-task learning

authors:
Debabrata Mahapatra
Vaibhav Rajan

title: Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization

authors:
R Timothy Marler
Jasbir S Arora

title: Survey of multi-objective optimization methods for engineering

venue: Structural and multidisciplinary optimization,

authors:
Natalia Martinez
Martin Bertran
Guillermo Sapiro

title: Minimax pareto fairness: A multi objective perspective

venue: In Proceedings of the 37th International Conference on Machine Learning (ICML),

authors:
Achille Messac
Amir Ismail-Yahaya
Christopher A Mattson

title: The normalized normal constraint method for generating the pareto frontier

title: Nonlinear multiobjective optimization, volume 12

venue: Springer Science & Business Media,

authors:
A Jemshia Miriam
R Saminathan
S Chakaravarthi

title: Non-dominated sorting genetic algorithm (nsga-iii) for effective resource allocation in cloud

venue: Evolutionary Intelligence,

authors:
Daniel Mueller-Gritschneder
Helmut Graeb
Ulf Schlichtmann

title: A successive approach to compute the bounded pareto front of practical multiobjective optimization problems

venue: SIAM Journal on Optimization,

authors:
Aviv Navon
Aviv Shamsian
Gal Chechik
Ethan Fetaya

title: Learning the pareto front with hypernetworks

title: Manuale di economica politica, societa editrice libraria

venue: milan. English translation as Manual of Political Economy,

authors:
Behzad Pirouz
Esmaile Khorram

title: A computational approach based on the "-constraint method in multi-objective optimization problems

venue: Advances and Applications in Statistics,

authors:
Maziar Raissi
Paris Perdikaris
George E Karniadakis

title: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

venue: Journal of Computational Physics,

authors:
Ozan Sener
Vladlen Koltun

title: Multi-task learning as multi-objective optimization

venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS),

authors:
Nidamarthi Srinivas
Kalyanmoy Deb

title: Muiltiobjective optimization using nondominated sorting in genetic algorithms

venue: Evolutionary computation,

authors:
Masahiro Tanaka
Hikaru Watanabe
Yasuyuki Furukawa
Tetsuzo Tanino

title: Ga-based decision support system for multicriteria optimization

venue: IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century,

authors:
Ma Guadalupe Castillo Tapia
Carlos A Coello Coello

title: Applications of multi-objective evolutionary algorithms in economics and finance: A survey

venue: IEEE Congress on Evolutionary Computation,

authors:
Trisna Trisna
Marimin Marimin
Yandra Arkeman
T Sunarti

title: Multi-objective optimization for supply chain management problem: A literature review

venue: Decision Science Letters,

authors:
Ana Valdivia
Javier Sánchez-Monedero
Jorge Casillas

title: How fair can we go in machine learning? assessing the boundaries of accuracy and fairness

venue: International Journal of Intelligent Systems,

authors:
Susan Wei
Marc Niethammer

title: The fairness-accuracy pareto front

venue: arXiv preprint arXiv:2008.10797,

authors:
Han Xiao
Kashif Rasul
Roland Vollgraf

title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

venue: arXiv preprint arXiv:1708.07747,

title: Compromise programming

venue: Multiple criteria decision making,

authors:
Qingfu Zhang
Aimin Zhou
Shizheng Zhao
Ponnuthurai Nagaratnam Suganthan
Wudong Liu
Santosh Tiwari

title: Multiobjective optimization test instances for the cec 2009 special session and competition

heading: 1 INTRODUCTION

text:
Multi-Objective Optimization (MOO) problems arise frequently across diverse fields such as engineering (Marler & Arora, 2004), finance (Tapia & Coello, 2007), and supply chain management (Trisna et al., 2016). Such problems share a common requirement to satisfy multiple competing objectives under a set of constraints imposed by physical or economic limits. A Pareto optimal solution Pareto (1906) for an MOO problem is defined as the solution point away from which no single objective can be improved without diminishing at least one other objective. A Pareto front is then defined as the set of all such optimal points that satisfy this definition.
Since practical MOO problems contain competing objectives and additional domain specific constraints, a Pareto solver should be able to handle both functions and constraints. Moreover, problems in classification and recommendation are non-convex (Hsieh et al., 2015), necessitating solvers that are robust for non-convex problems. Although many saddle point solvers have been proposed in the literature (Benzi et al., 2005; Benzi & Wathen, 2008), they cannot be generalized to handle constraints without specialized pre-conditioners designed to handle non-linear constraints. These existing solvers are based upon fixed-point iterations, returning one solution per run with specialized local initialization to generate an even spread of Pareto points across the feasible set of saddle points.
Recently proposed Multi-Task Learning (MTL) solvers (Sener & Koltun, 2018; Lin et al., 2019a; Mahapatra & Rajan, 2020; Ma et al., 2020; Navon et al., 2021) criticize Linear Scalarization (LS) since it retrieves a subset of the Pareto optimal set. MTL solvers claim to resolve this issue by decomposing the functional/variable domain (into cones or rays) while using LS in each of the subdomains. However, the stationary points of the LS problem (now decomposed) remain unchanged (see Appendix B), and therefore any fixed-point iterative strategy will still retrieve only a subset of the Pareto optimal set. Additionally, the theorems presented in MTL works rely upon convexity assumption that are never justified through numerical experiments. Furthermore, a notable absence of benchmarks against known analytical forms makes it difficult to assess reported results, verify
optimality, and A/B test alternative methods. This is in contrast to studies in Operations Research (OR) (Das & Dennis, 1998; Ghane-Kanafi & Khorram, 2015; Pirouz & Khorram, 2016) in which such comparative benchmarking and verification is well established. Although accurate and verifiable, existing OR methods tend to generate Pareto points with low density and limited scalability, with compute times ranging from hours to days as variable dimensionality increases.
While lack of formal guarantees for non-convex cases certainly does not preclude use of MTL methods for such problems, rigorous numerical evaluation becomes particularly important. However, the true Pareto front (closed-form analytical solution) is often unknown for real MOO problems, hence it is challenging to compare the accuracy of a Pareto solver on such problems. We follow the OR literature in advocating that the correctness of any proposed Pareto solver should first be tested on constructed benchmark problems with known analytic solutions. This is also consistent with broader ML community practice of first evaluating proposed methods in controlled synthetic or simulated conditions to verify correctness, before applying these systems on real datasets. In general, rigorously evaluating a new method on a diverse set of complicated benchmarking scenarios can provide valuable insights prior to venturing into the unknown and builds user trust in the model.
We propose a two-stage framework: Hybrid Neural Pareto Front (HNPF) for extracting Pareto optimal solution sets. Stage 1 consists of an interpretable neural network that extracts a weak Pareto solution manifold as the output, given a dataset as input. Following this, Stage 2 is a low-cost Pareto filter to remove dominated points from the weak Pareto set. The network loss function uses a discriminator based on Fritz-John Conditions (FJC) to account for multiple objectives and constraints. An approximate weak Pareto manifold is extracted as a weighted output of the softmax function from the last layer of the network. The softmax activation classifies weak Pareto vs. non-Pareto data points. HNPF extracts this weak Pareto front as a continuous manifold approximated by the Stage-1 neural network. Numerical experiments show the computational efficiency of HNPF vs. OR approaches and accuracy comparisons against MTL methods. HNPF produces only Pareto points (no false positives) with an even spread and high density. Furthermore, HNPF is scalable (compared to OR methods) with both increasing dimensions of the variable domain, and the number of functions and constraints. Table 1 summarizes key properties of HNPF vs. existing methods. Our key contributions are: 1. A neural manifold extraction strategy for weak Pareto front identification based on Fritz-John
conditions as the discriminator, for both convex and non-convex scenarios, supporting constraints. 2. The final neural net layer is interpretable as a continuous approximation of the weak Pareto mani-
fold. The extracted manifold error is bounded below w.r.t. the true manifold upon convergence. 3. The necessity of Pareto filter to remove dominated points from the weak Pareto set. Design of a
computationally efficient Pareto filter to extract the non-dominated Pareto optimal set. 4. Raising community awareness about existence of benchmarks from literature in OR field.

heading: 2 PROBLEM STATEMENT

text:
A general MOO problem can formulated as:
optimize F (x) = (f1(x), f2(x), . . . , fk(x)) (1)
s.t. x 2 H = {x 2 Rn|G(x) = (g1(x), g2(x), . . . , gm(x)  0} in n variables (x1, . . . , xn), k objective functions (f1, . . . , fk), and m constraint functions (g1, . . . , gm). Here, H is the feasible set i.e. the set of input values x that satisfy the constraints G(x). For a MOO problem there is typically no single global solution, and it is often necessary to determine a set of points that all fit a predetermined definition for an optimum. See Appendix A for definitions.

heading: 2.1 FRITZ JOHN CONDITIONS (FJC)

text:
Let the objective and constraint function in Eq. 1 be differentiable once at a decision vector x̃⇤ 2 H. The Fritz-John necessary conditions (FJC) for x̃⇤ to be weak Pareto optimal is that vectors must exists for 0  2 Rk, 0  µ 2 Rm and ( , µ) 6= (0, 0) (not identically zero) s.t. the following holds:
kX
i=1
irfi(x̃ ⇤) +
mX
j=1
µjrgj(x̃ ⇤) = 0, µjgj(x̃ ⇤) = 0, 8j = 1, . . . , m (2)
Gobbi et al. (2015) presented an L matrix form of FJC as follows:
L =
 rF rG
0 G
[(n + m) ⇥ (k + m)] (3)
rFn⇥k = [rf1, . . . ,rfk], rGn⇥m = [rg1, . . . ,rgm], Gm⇥m = diag(g1, . . . , gm)
In L matrix, for x⇤ to be Pareto optimal, is to show the existence of 2 Rk+m in Eq. 2 such that L · = 0 s.t. L = L(x̃⇤), 0, 6= 0 (4)
The non-trivial solution ( not identically zero) for Eq. 4 is:
det(LT L) = 0 (5)
Remark. Note that det(L) = 0 is equivalent to det(LT L) = 0 (see Appendix G for derivation).
The weak Pareto front is characterized by the set of points such that matrix L is low rank. This ensures that points identified are either inside the feasible set or at boundaries dictated by the constraints. For e.g. if µ1 = 0 for any i, then P i ifi = 0 must be satisfied for the corresponding internal point x ⇤ to be Pareto. Similarly if µ1 6= 0, µj 6=1 = 0 in the aforementioned case, then g1 = 0 holds true for the corresponding boundary point x⇤ to be Pareto. All Pareto points satisfy rfi = 0 for at least one i whether they lie inside the feasible set or on the boundary, i.e. all points x⇤ need to be local optimizers (stationary points)1 for at least one fi. The rank of the matrix LT L determines the dimension of the Pareto manifold. FJC written as det(LT L) is independent of the preference parameters i, µj . Thus, Eq. 5 is an oracle serving as a discriminator in HNPF to identify a weak Pareto front. Remark. For a multi-objective optimization (MOO) problem scalarization results in a singleobjective optimization (SOO) problem. However, the stationary points of this resulting SOO problem can only be a small subset of the weak Pareto solution set. The reader is referred to Appendix B for a visual explanation of stationarity w.r.t. an SOO problem arising from an MOO problem.

heading: 3 RELATED WORKS

text:
Generic and Enhanced Scalarization: One common approach is to convert an MOO problem into a Single Objective Optimization (SOO) problem via Linear Scalarization (LS). These include: Balashankar et al. (2019); Lin et al. (2019a); Martinez et al. (2020); Valdivia et al. (2021); Wei & Niethammer (2020). Enhanced scalarization approaches (Das & Dennis, 1998; Ghane-Kanafi & Khorram, 2015; Pirouz & Khorram, 2016) rely upon localization of the objective space to handle nonconvex functions and constraints. Although accurate and complete, these approaches suffer from low computational scalability and low density of Pareto points on the solution manifold. For example, the 30 dimensional benchmark (Case V in Section 5) shows enhanced scalarization methods (mCHIM and PK) generating a Pareto set in approximately 18 hours.
Multi Task Learning: MTL methods rely upon a utopia/ideal point (similar to Das & Dennis (1998)) but are shown to be scalable for high-dimensional MOO problems. MOOMTL (Sener & Koltun, 2018) uses a multi-gradient descent approach, but does not guarantee an even spread of solution points along the Pareto front. PMTL (Lin et al., 2019a) attempts to address this issue by dividing the functional domain into equal spaced cones, with increasing computational cost as the number of cones increase. EPO (Mahapatra & Rajan, 2020) uses preference rays along specified weights to find Pareto points in the vicinity of the rays. EPSE (Ma et al., 2020) uses a combination Hessian of the functions and Krylov subspace to find Pareto solutions. PHN (Navon et al., 2021) is initialized either by LS or EPO and is the first neural framework that can approximate the front as a neural manifold.
1A stationary point w.r.t. gradient descent fixed point iteration is the set of points corresponding to the local or global minima, maxima or saddle points of an objective function.
The reader is further referred to Appendix O for a detailed review of these methods including other from the Bayesian, Genetic and Fairness literature.
Current Limitations: Firstly, these aforementioned point-based approaches extract one solution point at a time, given the optimization problem converges. However, for practical applications, with non-convex objectives and constraints, ensuring optimality is non-trivial. Secondly, multiple runs with different trade-off parameters must be performed in order to extract the weak Pareto solution set, resulting in substantial computational overhead. Finally, the Pareto solution set can still form a non-convex manifold even when the objectives are convex due to the presence of non-convex constraints (Case III in Section 5). These challenges prove to be major obstacles in the deployment of MOO solution methods as practical tools for Pareto set extraction.

heading: 4 HNPF FRAMEWORK

text: We draw inspiration from three seminal works: 1) Das & Dennis (1998) proposed to break the functional domain boundary into uniform and evenly spaced segments to identify weak Pareto points with guarantees. Motivated by this, we first identify the weak Pareto front using a neural network (Stage 1). 2) Messac et al. (2003) proposed the first Pareto filter to obtain the set of strong Pareto points from the aforementioned weak Pareto set. The filter uses an all-pair comparison criterion to reject dominated points from the weak Pareto set. This motivates our low-cost Pareto filter (Stage 2) using a plane search strategy to avoid an expensive all-pair comparison. 3) Gobbi et al. (2015) presented a matrix form of the Fritz-John conditions satisfying the existence of weak Pareto points.

heading: 4.1 STAGE 1: NEURAL NET FOR WEAK PARETO FRONT

text:
Given FJC, one can brute force classify the data points as weak Pareto or not. However, our intent is to extract an approximate Pareto front as an indicator function (M̃ : Rn ! R) parameterized by the neural network. The Stage-1 neural network (Fig. 1) smoothly approximates the true weak Pareto manifold M(X⇤) as M̃(X̃) (see Appendix F for details on notation). The last layer has two neurons with softmax activation for binary clas-
sification of Pareto vs. non-Pareto points, distinguishes weak Pareto vs. sub-optimal points, in the feasible set H. Our network loss is representation driven, since the FJC guided discriminator (Eq. 5), described by the objectives and constraints, explicitly classifies the input data points X points.
The Pareto front is a lower dimensional manifold ( (n 1)) in an n-dimensional variable domain. The Pareto vs. non-Pareto classes are not comparable since the measure of the Pareto front is zero. In other words, the volume of a surface (or anything lower) is zero. To volumize the Pareto front and to make a Binary Cross Entropy measure computable, we use a diffusive indicator function. The volume can then be made tending to zero by choosing a smaller ✏ approximating the lower dimensional weak-Pareto manifold. We use a slightly relaxed criterion with user-tunable threshold 0  ✏  1 as the classification margin. Any point below this value will be classified as weak Pareto. The FJC for weak Pareto optimality then requires that the D = det(LT L) = 0. Therefore, |D|  ✏ (True or False) naturally provides us with binary labels for the softmax activated output layer. A binary cross entropy loss ensures that the distribution of the extracted manifold M̃(X̃) matches the distribution of the weak Pareto front satisfying FJC within the specified ✏ tolerance. This is similar to recently popularized Physics Informed Neural Networks (PINNs) (Raissi et al., 2019) where a known analytical form of the regression function is supplied as a loss term, instead of explicit labels.
Error bound and Network Interpretability. For a user-prescribed relaxation margin 0  ✏  1, the approximation error between the network extracted manifold M̃(X̃) and the true solution M(X⇤) is bounded below by kM̃(X̃) M(X⇤)k2  ✏, upon convergence. See Appendix F for proof and Appendix H for a discussion on interpretability as a mathematical model. Remark. For a n-dimensional variable domain, HNPF (Stage 1) uses the Fritz-Johns conditions in the determinant form to extract a n-dimensional, diffusive, indicator function using a binary cross entropy loss (weak Pareto vs. non-Pareto). The indicator function regresses to 1 at the weak Pareto front and 0 everywhere else with values between 0 and 1 in the ✏ neighborhood of this weak Pareto front (diffusive indicator). In this respect, Fritz-Johns condition is an oracle to gather the weak Pareto set where the determinant form is used instead of user provided labels to identify the front.

heading: 4.2 STAGE 2: PARETO FILTER FOR STRONG PARETO SET

text:
A Pareto filter is an algorithm that, given a set of weak Pareto points P of cardinality |P| = P in objective space, retains a subset of non-dominated points. This corresponds to the strong Pareto set s.t. none of the points are dominated, i.e. the filter eliminates all dominated points from P . The state of the art Pareto filter (Messac et al., 2003) requires an all-pairs comparison (O(P!) time complexity), that becomes computationally expensive as the set P grows in size. However, since Stage-1 of HNPF generates weak Pareto points with high density, the filter proves to be quite expensive. Remark. A fixed point iterative approach might converge to the stationary points (local criterion) of an MOO problem. However, it is not necessary that all such stationary points are non-dominated. A filter is therefore necessary to remove the dominated points based upon a global criterion. See Section 5.1 and 5.4 for numerical experiments where a filter is necessary.
We present an efficient Pareto filter for finding the strong Pareto set which is computationally scalable to arbitrary dimensions. The algorithm is based on a plane search strategy, inspired by Kd-Trees (Bentley, 1990), well known for efficient data partitioning and storage. The compute cost is guided by the number of functions while being linearly proportional to the number of points. The inputs to the algorithm (Fig. 2) are the number of functions k and their respective global minima and maxima. These are known a priori for benchmark problems, or can be computed from the data itself in O(kP ) time i.e. a linear search over the k functions for P points. f(min), f(max),
discretization level h of the function space and the weak Pareto points P . The output is the strong Pareto set {x⇤}. Refer to Appendix N for algorithmic details and usage illustration of the filter. Time complexity: The three nested for loops carry the load. In the worst case that the points in the weak Pareto set are all strong Pareto, then the cardinality P of the set P remains unchanged. Let z = (fi(max) fi(min))/h denote the number of chunks into which the function space is divided. The worst case complexity of the proposed Pareto filter is O(kzP). For scenarios, where the strong Pareto set is a subset of the original weak set P , the complexity reduces in the factor guided by P .

heading: 4.3 COMPARISON BETWEEN HNPF AND PHN

text:
HNPF (Stage-1) extracts the weak Pareto manifold as an n-dimensional diffusive indicator function as opposed to a (n 1)-dimensional manifold itself. This results in an indicator function M̃ : Rn ! R that with an n-dimensional support in the variable space. There are two explicit advantages in describing this indicator function: (1) The regressed Pareto manifold is not only guided by the weak Pareto points (indicator value 1) but also the sub-optimal points (indicator value 0) for a more robust and accurate extraction. This is comparable to classification under a balanced class density wherein if the class densities were increasingly imbalanced, the extracted classification boundary becomes more inaccurate. (2) The Pareto optimal set can be extracted and represented by a neural network (function approximator) even when the manifold is an implicit surface. For eg., consider representing a circle (equation as opposed to a function) as network extracted manifold in a 2D variable domain.
In comparison, PHN (Navon et al., 2021) uses a neural network to regress over the (n 1)-dimensional manifold in the variable domain directly, given solution points obtained from EPO or LS. However, i) neither EPO nor LS are guaranteed to work for non-convex scenarios (see Section 5); ii) if the nondominated Pareto set is discontinuous, then PHN will wrongly identify non-Pareto points as Pareto optimal; iii) even when the Pareto optimal set is a continuous manifold, PHN inherently assumes that the point spread obtained from EPO or LS is uniformly distributed over the true manifold for the neural regressor to not overfit; and iv) if the manifold is an implicit surface then a direct regression using a neural network is not feasible. Further numerical justifications can be found in Appendix D.

heading: 5 RESULTS

text:
In this section, we present four numerical experiments for benchmarking and analysis, with five more in Appendix I. These address analytical forms, with increasing complexity and scale in the number
of functions (k), constraints (m) and variable dimension (n). The benchmark cases are chosen to test various aspects desired from a general Pareto solver. We compare HNPF vs. two OR methods: mCHIM and PK2; and four recent MTL methods: MOOMTL, PMTL, EPO, and PHN. For HNPF’s setup and error tolerance specification, refer to Appendix J. Additional experiments are presented as an ablation study (Appendix K) with loss profiles (Appendix L) and runtimes (Appendix M). Data Sampling: Since the location of the Pareto front is not know a priori, samples are drawn uniformly at random from a feasible variable domain. Objective functions evaluated at these points generate a quantized, topographic map of the function domain that is then used to identify optimal points. For each benchmark test case below, we generate 11k points from a random uniform distribution in the feasible variable domain, to serve as training data. The training-validation split is 90-10%. Once the manifold M̃(X̃) is approximated by the network, we use 90k points in the feasible domain as test data to visualize the Pareto set.
Remark. Although MTL methods do not explicitly handle constraints, the Pareto solution set is only reported for the benchmark cases (I, II, IV) where the linear constraints form a bounding box.

heading: 5.1 CHALLENGES IN PARETO OPTIMALITY & THE NEED FOR A PARETO FILTER

text:
(n = 2, k = 1, m = 0)
We now consider a two objective, unconstrained, MOO problem in a single variable domain where the two, non-convex objective functions are given by:
f1(x) = (x 1)(x 2)(x 3)(x 4)
f2(x) = (x 1.5)(x 2.5)(x 3.6)(x 4.5)
We design this benchmark MOO problem to specifically test out HNPF’s robustness when multiple optimal manifolds exist for the corresponding SOO problem. The LS single-objective function is:
S(↵, x) = ↵f1(x) + (1 ↵)f2(x)
Fig. 3 (a) shows the surface of S(↵, x) in (↵, x) (blue mesh) with the three optimal manifolds (red curves). Note that for any point x there exists at least one ↵ on one of the three manifolds. Therefore, the final Pareto set can be formed by a union of the subset of each of these three manifolds. Fig. 3 (b) shows that the optimal manifolds (red curve) overlaps the entire f1 vs. f2 plot (cyan curve) in the functional domain, as expected. The Pareto optimal set is then obtained by applying a filter based upon the global criterion of non-dominated points in the functional domain as shown in Fig. 3 (c). Note that for this particular benchmark case, the Pareto optimal set is a union of the subset of two optimal manifolds in Fig. 3 (a). All MTL solvers fail in this case when the Pareto front in the functional domain is an implicit, self intersecting manifold. This numerical experiment also enunciates the importance of a Pareto filter to satisfy the global criterion of non-dominated points inherent to the definition of Pareto optimality. A fixed point iteration converges locally by satisfying a local optimality/termination criterion. The global criterion of non-dominated points can only be satisfied once the optimal set is gathered completely from the previous step.
2Since the codes for mCHIM and PK are not publicly available, we snip results from their works to avoid any artifacts in reproducibility. Hence the color discrepancy in plots.
5.2 CASE I: CONVEX OBJECTIVES, LINEAR CONSTRAINTS (n = 2, k = 2, m = 2) This problem was originally proposed in Fonseca & Fleming (1998). Jointly minimize:
f1(x1, x2) = 1 exp( [(x1 1/ p (2))2 + (x2 1/ p (2))2]) f2(x1, x2) = 1 exp( [(x1 + 1/ p (2))2 + (x2 + 1/ p (2))2]) s.t. g1, g2 : 1/ p 2  x1, x2  1/ p 2
This is a common benchmarking problem in both OR and machine learning literature. Since the objectives and constraints are convex, HNPF (see Appendix I.1), OR and MTL methods all retrieve the Pareto optimal set accurately. The reader is referred to Appendix B for a detailed discussion and visual explanation of Pareto optimality using a variant of this problem.
5.3 CASE II: NON-CONVEX OBJECTIVES, LINEAR CONSTRAINTS (n = 2, k = 2, m = 2) This problem was proposed in Ghane-Kanafi & Khorram (2015). Jointly minimize:
HNPF, LS, NBI, mCHIM, and MOOMTL methods identify Pareto points accurately while PMTL, EPO, and PHN also generate sub-optimal points (marked blue circle). Fig. 4 shows the results from our model with high point density. It also satisfies closely the true Pareto manifold M(X⇤) given by 0  x1  1, x2 = 0 in Fig. 4(b).
5.4 CASE III: CONVEX OBJECTIVES, NON-CONVEX CONSTRAINTS (n = 2, k = 2, m = 4) This problem was proposed in Tanaka et al. (1995). Jointly minimize:
f1(x1, x2) = x1, f2(x1, x2) = x2
s.t. g1(x1, x2) : (x1 0.5)2 + (x2 0.5)2  0.5
g2(x1, x2) : x 2 1 + x 2 2 1 0.1 cos(16 arctan(
x1 x2 )) 0
g3, g4 : 0  x1, x2  ⇡
Here, f1, f2 are convex but the constraints g1, g2 are non-convex in the variable domain and consequently the resulting Pareto front is non-convex. An important point to note here is that the Pareto front is dominated solely by the two constraints g1 and g2, while linear functions f1 and f2 do not contribute. Since MTL solvers do not account for constraints, a comparison is only shown against OR methods that report results for this benchmark. Here, mCHIM extracts a sparse set of non-dominated Pareto points (⇠ 40) while HNPF Stage-1 extracts a dense set of weak Pareto points (Fig. 5). To arrive at the non-dominated Pareto set, we post-process this result using our Stage-2 Pareto filter (see Section 4.2). The updated discontinuous set of non-dominated Pareto points is shown in Fig. 5 (b).
5.5 CASE IV: NON-CONVEX OBJECTIVES, LINEAR CONSTRAINTS (n = 30, k = 2, m = 30) This problem was proposed in Zhang et al. (2008). Jointly minimize:
f1(x) = x1 + 2
|J1|
X
j2J1
y 2 j , f2(x) = 1 p x1 +
2
|J2|
X
j2J2
y 2 j
s.t. g1, . . . , g30 : 0  x1  1, 1  xj  1, j = 2, . . . , m J1 = {j|j is odd, 2  j  m}, J2 = {j|j is even, 2  j  m}
yj =
⇢ xj [0.3x21 cos(24⇡x1 + 4j⇡ m ) + 0.6x1]cos(6⇡x1 + j⇡ m ) j 2 J1
xj [0.3x21 cos(24⇡x1 + 4j⇡ m ) + 0.6x1]cos(6⇡x1 + j⇡ m ) j 2 J2
This form is non-convex in both f1, f2. The dimension of the design variable space is m = 30. The Pareto front in the variable domain is non-convex (sinusoidal spiral), implicit surface. HNPF results (Fig. 6) are in good agreement with mCHIM and PK methods, but with significantly higher density. See Appendix M for compute time scalability of HNPF vs. OR methods. All MTL methods fail with the inherent min-norm solver returning NaN.

heading: 5.6 DISCUSSION OF METHODS

text:
OR methods: NBI works for cases where the detected weak Pareto front consists of non-dominated points only. Therefore, NBI generates correct solution in Cases I, II, IV and V with even density of points on the Pareto front. In essence, applying the Pareto filter on the NBI generated solution set would resolve the discontinuous cases too. NBI, mCHIM, PK and HNPF produce only Pareto points. Additionally, HNPF generates Pareto points uniformly with high density, while OR methods including mCHIM and PK, although accurate, are limited to low point density (⇠ 40) with large compute overhead as the variable dimension scales. Table 2 shows a comparison of the extracted Pareto point density (#extracted Pareto points/#function evaluations) for HNPF, OR, and MTL methods.
MTL Approaches: One of the main concerns regarding MTL solvers is that a termination criterion associated with the optimization approach is not described. For any method to be called a solver a termination criterion is necessary since the number of iterations to reach a solution point is not known a-priori. We suggest the reader avail themselves of the open-source codes provided by the respective MTL authors wherein the solver uses a for loop with a pre-specified number of iterations. Furthermore, it is not clear as to how MTL solvers, relying upon a linear scalarized form (SOO) of the original MOO problem, arrive at the non-stationary points of this SOO. Even if the domain is partitioned into sub-domains (cones, rays etc.) the stationary points of the SOO remain unchanged. The reader is referred to Appendix B for a detailed discussion using a benchmark problem. HNPF: Since HNPF Stage-1 approximates the true Pareto manifold M(X⇤), it has the advantage of learning M̃(X̃) from only 10k training points. No other enhanced scalarization method (mCHIM, PK or MTL) can approximate the Pareto manifold itself, rather detect Pareto points only: hypothetically having just the Fritz-John discriminator in Stage-1 without the neural network preceding it. Additionally, HNPF explicitly uses the Stage-2 filter necessary to remove dominated point from the Stage-1 weak Pareto solution set that cannot be addressed by an optimization problem.
Results on Multi-MNIST: If HNPF or OR machineries are correct, then why not show results on practical datasets? This is a valid question since the core objective of any approach is to resolve a practical problem. We are currently faced with the following obstacles: i) HNPF and OR methods are not yet neural scalable, hence cannot handle variable dimensions at orders ⇠ 103 or higher; ii) MTL methods do not have a termination criteria and during numerical evaluation either produce non-Pareto points or fail completely for some of the benchmark non-convex cases. This raises serious concerns regarding their veracity on practical datasets where the ground truth Pareto solution is unknown; and iii) Framing Multi-MNIST as a Pareto optimality problem raises serious concerns from first principles, as to how the altered network architecture still abides by the Pareto framing. Readers are referred to Appendix E for a detailed discussion on this argument.
5.7 NUMERICAL CORRECTNESS OF HNPF Table 3: Mean Squared Error of HNPF extracted manifold M̃ w.r.t. the true analytical manifold M .
Cases I II III IV V VI VII
Error ( ⇥10 4) 1.2 1.2 3.7 4.1 2.5 3.2 1.7 We numerically verify the proposed error bounds for the benchmark cases cosnidered. Since the analytical form of the Pareto solution manifold M(X⇤) is known, we can numerically verify the network approximated manifold M̃(X̃). Table 3 shows the L2 error between true and approximated manifolds. Note that for all the experiments since ✏ = 5⇥ 10 4, the errors are bounded above by ✏, even when the functions and constraints are non-convex.
Since we are considering non-convex problems in an effort to edge closer to practical applications, providing theoretical guarantees is not possible without making debilitating convexity assumptions on the functions and constraints. MTL methods are supported by several theorems that hold true only for convex functions wherein numerical results are supplied for non-convex problems. Further, no numerical experiments are presented in MTL works that confirm their theoretical results. On the other hand, HNPF is a practical Pareto optimal set extractor with numerically verifiable error bounds without introducing impractical assumptions.

heading: 6 CONCLUSION AND FUTURE WORK

text: A two-stage, neural-filter (HNPF) based optimization framework is presented for extracting the Pareto optimal solution set for MOO constrained optimization problems. HNPF is computationally efficient and scales well with increasing dimensionality of design variable space, objective functions, and constraints w.r.t. OR methods. Results on verifiable benchmark problems show that our Pareto solution set accuracy compares well against known solutions for a wide variety of benchmark problems. The benchmark problems are chosen to test out different aspects of our HNPF framework that might arise in practical MOO problems. The proposed neural architecture is interpretable with an FJC guided discriminator for weak Pareto manifold classification with an efficient Pareto filter to extract an optimal Pareto set. We also show that the approximation error between the true and extracted Pareto manifold can be easily verified for analytical solutions. In our future work, we will develop an FJC guided solver for a scalable HNPF algorithm to address high-dimensional neural MOO problems.


summaries: The paper introduces a representation consolidation method to properly aggregate the pre-trained knowledge from multiple teachers for transfer learning. It claims that a generalist teacher is necessary for preserving the transferability of distilled representations, and task-specific teachers contribute to better performance in the same-domain downstream tasks. The introduced method performs on par with multi-task training while neglecting the need for teacher datasets.

summaries: - The authors propose a method of learning a consolidated image feature representation from a collection of related task-specific teachers that transfer well on novel recognition tasks.  - To achieve it, the authors utilize multi-teacher multi-task model distillation framework that jointly distills one or several task-specific teachers with a generalist one (trained with Imagenet dataset).   - Each teacher is set to operate on a different set of classes, and a multi-head student is trained to emulate all teachers.  - Experimental results show that the proposed method doesn’t need to revisit original training data of each teacher, but gains better performance.

summaries: The paper proposes a task-agnostic Knowledge Distillation method that includes a generalist model as an additional teacher, to limit student forgetting and representation collapse. The authors call this method representation consolidation and claim it can benefit both related and unrelated downstream tasks. The authors experiment with ImageNet and iFood image classification tasks, the accuracy of the proposed method is generally higher than using Knowledge Distillation only.

summaries: This paper studies the problem of multi-model consolidation, which aims to merge (consolidate) the knowledge from multiple models into one model so that we can better transfer this merged model for downstream tasks. Different from the traditional knowledge distillation or multi-teacher distillation, the proposed method aims to transferability of merged models for the downstream tasks, whilst achieving good performance on source tasks. The key technique (technical contribution) is to add a generalized model during the knowledge distillation, which can be regarded as s "special" teacher model.    The authors conduct extensive experiments on multiple datasets and most of the experiments are under a few-shot linear probe transfer learning setting.

abstractText: A library of diverse expert models transfers better to a novel task than a single generalist model. However, growing such a library indefinitely is impractical. Hence, we explore the problem of learning a consolidated image feature representation from a collection of related task-specific teachers that transfer well on novel recognition tasks. This differs from traditional knowledge distillation in which a student model is trained to emulate the input/output functionality of a teacher. Indeed, we observe experimentally that standard distillation of task-specific teachers, or using these teacher representations directly, reduces downstream transferability compared to a task-agnostic generalist model. We show that a simple multihead, multi-task distillation method using an unlabeled proxy dataset and adding a generalist teacher is sufficient to consolidate representations from task-specific teacher(s). We improve downstream performance, outperforming the teacher (or best of all teachers) as well as the strong baseline of ImageNet pre-trained features. Our method almost reaches the performance of a multi-task joint training oracle, reaping the benefit of the teachers without replaying their training data.

authors:
Alessandro Achille
Michael Lam
Rahul Tewari
Avinash Ravichandran
Subhransu Maji
Charless C Fowlkes
Stefano Soatto
Pietro Perona

title: Task2vec: Task embedding for meta-learning

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

authors:
Sravanti Addepalli
Gaurav Kumar Nayak
Anirban Chakraborty
Venkatesh Babu Radhakrishnan

title: Degan: Data-enriching gan for retrieving representative samples from a trained classifier

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Gustavo Aguilar
Yuan Ling
Y. Zhang
Benjamin Yao
Xing Fan
Edward Guo

title: Knowledge distillation from internal representations

venue: In AAAI,

authors:
Victor Besnier
Himalaya Jain
Andrei Bursuc
M. Cord

title: P’erez. This dataset does not exist: Training models from generated images

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Kartikeya Bhardwaj
Naveen Suda
R. Marculescu

title: Dream distillation: A data-independent model compression framework

venue: In Joint Workshop on On-Device Machine Learning & Compact Deep Neural Network Representations, NeurIPS Workshop,

authors:
Lukas Bossard
Matthieu Guillaumin
Luc Van Gool

title: Food-101 – mining discriminative components with random forests

venue: In European Conference on Computer Vision,

title: Multitask learning

venue: In Encyclopedia of Machine Learning and Data Mining,

authors:
Rudrasis Chakraborty
Chun-Hao Yang
B. Vemuri

title: A mixture model for aggregation of multiple pre-trained weak classifiers

venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW),

authors:
Akshay Chawla
Hongxu Yin
Pavlo Molchanov
J. Álvarez

title: Data-free knowledge distillation for object detection

venue: In WACV,

authors:
Hanting Chen
Yunhe Wang
Chang Xu
Zhaohui Yang
Chuanjian Liu
Boxin Shi
Chunjing Xu
Chao Xu
Qi Tian

title: Dafl: Data-free learning of student

authors:
Gong Cheng
Junwei Han
Xiaoqiang Lu

title: Remote sensing image scene classification: Benchmark and state of the art

venue: Proceedings of the IEEE,

authors:
Yi-Min Chou
Yi-Ming Chan
Jia-Hong Lee
Chih-Yi Chiu
Chu-Song Chen

title: Unifying and merging well-trained deep neural networks for inference stage

venue: In Proceedings of the 27th International Joint Conference on Artificial Intelligence,

authors:
M. Cimpoi
Subhransu Maji
I. Kokkinos
S. Mohamed
A. Vedaldi

title: Describing textures in the wild

venue: IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Aditya Deshpande
Alessandro Achille
Avinash Ravichandran
Hao Li
Luca Zancato
Charless Fowlkes
Rahul Bhotika
Stefano Soatto
Pietro Perona

title: A linearized framework and a new benchmark for model selection for fine-tuning

venue: arXiv preprint arXiv:2102.00084,

authors:
Gongfan Fang
J. Song
Chengchao Shen
X. Wang
D. Chen
Mingli Song

title: Data-free adversarial distillation

authors:
Zhiyuan Fang
Jianfeng Wang
Lijuan Wang
L. Zhang
Yezhou Yang
Zicheng Liu

title: Seed: Self-supervised distillation for visual representation

authors:
J. Gao
Zijian Guo
Z. Li
R. Nevatia

title: Knowledge concentration: Learning 100k object classifiers in a single cnn

authors:
Robin Geyer
Luca Corinzia
Viktor Wegmayr

title: Transfer learning by adaptive merging of multiple models

venue: In MIDL,

authors:
G. Griffin
Alex Holub
P. Perona

title: Caltech-256 object category dataset

venue: Technical report, California Institute of Technology,

authors:
Neel Guha
Ameet S. Talwalkar
Virginia Smith

title: One-shot federated learning

venue: Machine Learning on Devices Workshop at NeurIPS,

authors:
S. Guo
Weilin Huang
X. Zhang
Prasanna Srikhanta
Yin Cui
Y. Li
M. Scott
Hartwig Adam
Serge J. Belongie

title: The imaterialist fashion attribute dataset

venue: IEEE/CVF International Conference on Computer Vision Workshop (ICCVW),

authors:
Matan Haroush
Itay Hubara
E. Hoffer
Daniel Soudry

title: The knowledge within: Methods for data-free model compression

venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Kaiming He
X. Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Geoffrey E. Hinton
Oriol Vinyals
J. Dean

title: Distilling the knowledge in a neural network

authors:
Grant Van Horn
Oisin Mac Aodha
Yang Song
Yin Cui
C. Sun
Alexander Shepard
Hartwig Adam
P. Perona
Serge J. Belongie

title: The inaturalist species classification and detection dataset

venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Wenpeng Hu
Z. Lin
B. Liu
Chongyang Tao
Zhengwei Tao
Jinwen Ma
Dongyan Zhao
Rui Yan

title: Overcoming catastrophic forgetting for continual learning via model adaptation

venue: In ICLR,

authors:
Parneet Kaur
Karan Sikka
Weijun Wang
serge Belongie
Ajay Divakaran

title: Foodx-251: A dataset for fine-grained food classification

authors:
J. Kirkpatrick
Razvan Pascanu
Neil C. Rabinowitz
J. Veness
G. Desjardins
Andrei A. Rusu
K. Milan
John Quan
Tiago Ramalho
Agnieszka Grabska-Barwinska
D. Hassabis
C. Clopath
D. Kumaran
R. Hadsell

title: Overcoming catastrophic forgetting in neural networks

venue: Proceedings of the National Academy of Sciences,

authors:
Animesh Koratana
Daniel Kang
Peter Bailis
M. Zaharia

title: Lit: Learned intermediate representation training for model compression

venue: In ICML,

authors:
Jonathan Krause
Michael Stark
Jia Deng
Li Fei-Fei

title: 3d object representations for fine-grained categorization

venue: In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia,

authors:
Sang-Woo Lee Lee
Jin-Hwa Kim
Jaehyun Jun
Jung-Woo Ha
Byoung-Tak Zhang

title: Overcoming Catastrophic Forgetting by Incremental Moment Matching (IMM)

venue: In Advances In Neural Information Processing Systems

authors:
Zhizhong Li
Derek Hoiem

title: Learning without forgetting

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Iou-Jen Liu
J. Peng
A. Schwing

title: Knowledge flow: Improve upon your teachers

venue: 7th International Conference on Learning Representations,

authors:
Raphael Gontijo Lopes
Stefano Fenu
T. Starner

title: Data-free knowledge distillation for deep neural networks. ArXiv

authors:
Liangchen Luo
M. Sandler
Zi Lin
A. Zhmoginov
Andrew G. Howard

title: Large-scale generative data-free distillation

authors:
Subhransu Maji
Esa Rahtu
Juho Kannala
Matthew B. Blaschko
A. Vedaldi

title: Fine-grained visual classification of aircraft

authors:
P. Micaelli
A. Storkey

title: Zero-shot knowledge transfer via adversarial belief matching

venue: In NeurIPS,

authors:
G.K. Nayak
K.R. Mopuri
V. Shaj
R.V. Babu
A. Chakraborty

title: Zero-shot knowledge distillation in deep networks

venue: In International Conference on Machine Learning,

authors:
Maria-Elena Nilsback
Andrew Zisserman

title: Automated flower classification over a large number of classes

venue: Sixth Indian Conference on Computer Vision, Graphics & Image Processing,

authors:
Tribhuvanesh Orekondy
B. Schiele
Mario Fritz

title: Knockoff nets: Stealing functionality of blackbox models

authors:
Seonguk Park
Nojun Kwak

title: Feature-level ensemble knowledge distillation for aggregating knowledge from multiple networks

venue: In ECAI,

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga
Alban Desmaison
Andreas Kopf
Edward Yang
Zachary DeVito
Martin Raison
Alykhan Tejani
Sasank Chilamkurthy
Benoit Steiner
Lu Fang
Junjie Bai
Soumith Chintala

title: Pytorch: An imperative style, high-performance deep learning library

venue: In Advances in Neural Information Processing Systems

authors:
F. Pedregosa
G. Varoquaux
A. Gramfort
V. Michel
B. Thirion
O. Grisel
M. Blondel
P. Prettenhofer
R. Weiss
V. Dubourg
J. Vanderplas
A. Passos
D. Cournapeau
M. Brucher
M. Perrot
E. Duchesnay

title: Scikit-learn: Machine learning in Python

venue: Journal of Machine Learning Research,

authors:
Ameya Prabhu
P. Torr
P. Dokania

title: Gdumb: A simple approach that questions our progress in continual learning

venue: In ECCV,

authors:
Joan Puigcerver
Carlos Riquelme
Basil Mustafa
Cedric Renggli
André Susano Pinto
Sylvain Gelly
Daniel Keysers
Neil Houlsby

title: Scalable transfer learning with expert models

authors:
Sylvestre-Alvise Rebuffi
Alexander Kolesnikov
G. Sperl
Christoph H. Lampert

title: icarl: Incremental classifier and representation learning

authors:
Yonglong Tian
Dilip Krishnan
Phillip Isola

title: Contrastive representation distillation

venue: In International Conference on Learning Representations,

authors:
J. Vongkulbhisal
Phongtharin Vinayavekhin
M.V. Scarzanella

title: Unifying heterogeneous classifiers with distillation

authors:
C. Wah
S. Branson
P. Welinder
P. Perona
S. Belongie

title: The Caltech-UCSD Birds-200-2011 Dataset

venue: Technical Report CNS-TR-2011-001, California Institute of Technology,

authors:
Guodong Xu
Ziwei Liu
Xiaoxiao Li
Chen Change Loy

title: Knowledge distillation meets selfsupervision

authors:
Shoukai Xu
Haokun Li
Bohan Zhuang
Jing Liu
Jiezhang Cao
Chuangrun Liang
Mingkui Tan

title: Generative low-bitwidth data free quantization

authors:
I.Z. Yalniz
H. Jégou
K. Chen
Manohar Paluri
D. Mahajan

title: Billion-scale semi-supervised learning for image classification

authors:
Jingwen Ye
Yixin Ji
X. Wang
Xin Gao
Mingli Song

title: Data-free knowledge amalgamation via group-stack dual-gan

authors:
Hongxu Yin
Pavlo Molchanov
Zhizhong Li
Jose M. Alvarez
Arun Mallya
Derek Hoiem
N. Jha
J. Kautz

title: Dreaming to distill: Data-free knowledge transfer via deepinversion

authors:
Jaemin Yoo
Minyong Cho
T. Kim
U. Kang

title: Knowledge extraction with no observable data

authors:
Friedemann Zenke
Ben Poole
S. Ganguli

title: Continual learning through synaptic intelligence

venue: Proceedings of machine learning research,

authors:
X. Zhao
S. Schulter
G. Sharma
Yi-Hsuan Tsai
Manmohan Chandraker
Ying Wu

title: Object detection with a unified label space from multiple datasets

authors:
B. Zhou
À. Lapedriza
A. Khosla
A. Oliva
A. Torralba

title: Places: A 10 million image database for scene recognition

heading: 1 INTRODUCTION

text:
A promising approach to scale transfer learning to diverse downstream vision tasks is to maintain a library of diverse experts pre-trained on different tasks. (Deshpande et al., 2021) When presented with a novel downstream task, one can select an appropriate expert and fine-tune the representation with small amounts of task-specific data. This strategy has many practical benefits: fine-tuning a task-relevant pre-trained representation is fast, there is no need to store or revisit expert pre-training data. How is such a library of expert representations populated and maintained? Previous work (e.g., Puigcerver et al. (2020); Achille et al. (2019); Deshpande et al. (2021)) has assumed a static collection of experts created by training on domain-specific datasets or domain-specific subsets of general dataset. Instead, we would like to automatically enrich the diversity of the expert library by accumulating knowledge from transferred downstream tasks, so that the overall system performance continually increases over time (life-long meta-learning).
One approach to growing the library would be to add fine-tuned downstream task-specific models back into the library as a candidate expert for transfer on future tasks. Such a naı̈ve approach clearly does not scale and, at a minimum, requires developing techniques for selecting experts that is sub-linear in the size of the library (Puigcerver et al., 2020; Achille et al., 2019; Deshpande et al., 2021). More importantly, as our experiments show in Sec. 4, task-specific models fine-tuned on small amounts of data do not provide transferable representations. Task-specific models tend to overspecialize and degrade under-utilized features in their representations, and thus under-perform on new tasks compared to generic pre-trained models.
To address this, we introduce representation consolidation where the goal is to consolidate knowledge from multiple task-specific teacher models into a single expert student representation which transfers to downstream tasks better than any of the individual teacher representations. There are three factors that make the problem of representation consolidation unique relative to the existing literature: (1) we assume a representation is to be consolidated from a collection of multiple taskspecific models, (2) we avoid the need to revisit task-specific training data of these models, and (3) we focus on improving downstream transfer instead of simply replicating performance on the upstream teacher’s task(s). To the best of our knowledge, no prior work has analyzed the downstream transfer aspect of distilled domain-expert representations.
To carry out representation consolidation, we utilize multi-teacher multi-task model distillation (see Fig. 1). We use a simple yet effective method to jointly distill one or several task-specific teacher with a generalist one. Each teacher may operate on a different set of classes, and a multi-head student is trained to emulate all teachers. Previous work on knowledge distillation has focused on the student model’s performance on the teacher’s task. Instead, we evaluate how well the student representation generalizes to new downstream tasks (whether related or unrelated to the teachers’ tasks). In this new setting we demonstrate several surprising results:

heading: 2 REPRESENTATION CONSOLIDATION

text:
Problem statement. We start with a collection of one or more task-specific image classification models {Mit}Ni=1 as teachers, trained on corresponding datasets {Dit } belonging to some domain (satellite images, images of flowers, etc.). We assume models consist of a feature extractor or backbone φit (·), composed with a classifier head hit (·) so thatMit = hit (φit (·)). We consolidate the knowledge of these task-specific teachers into a single student representation φs(·) using a proxy dataset Dproxy (e.g., ImageNet), and then evaluate φs(·) by training a SVM on top of it (or fine-tuning it) on a given downstream Djd chosen from some set {D j d}. Our goal is that the resulting downstream model hjd(φ j d(·)) achieves good performance, where φ j d denotes the student φs as-is or potentially after tuning on Djd . Fig. 1 highlights how this differs from standard distillation in which the student model hs(φs(·)) is simply evaluated on the same task its teachers were once trained to perform.
Forgetting and representation collapse during distillation. Though our problem statement is novel, can existing approaches solve our problem statement? We observe (Sec. 4) that neither the task-specific teachers nor students distilled with standard or state-of-the-art methods provide sufficiently transferable representations. They under-perform general pre-training representations (e.g. with ImageNet) when evaluated on downstream tasks from the teachers’ domain, and drastically under-perform on tasks outside the teachers domain. Surprisingly, this holds true even though we use ImageNet-pretrained weights to initialize the teacher and student networks. Thus, we argue that training task-specific teachers or distilling from them suffers from catastrophic forgetting of general knowledge that is crucial for transfer learning.
Intuitively, transfer performance depends on how distinguishable different classes are represented in the penultimate layer feature space. In standard distillation, the student only learns from taskspecific teachers trained on smaller datasets and is only required to discriminate the classes in those datasets. This is well suited for the traditional distillation problem that evaluates the student on these same tasks. But for representation consolidation, distinguishing unknown downstream classes requires preserving general features that may not be relevant to the teachers’ specific task. Our strategy is thus to ensure that the student maintains general features as it learns task-specific ones.
Method formulation. A key challenge to our approach is avoiding replay of the images used to train the task-specific teachers. To overcome this, we propose using a proxy dataset and a generalist model that transfers well (e.g. one pre-trained on ImageNet). This model is referred to as a generalist as it is not task specific and contains a wide variety of classes, and use it to help avoid forgetting. More specifically, we use multi-head multi-task distillation. We put N + 1 heads, h0s , . . . , h N s , on top of the student backbone φs(·). In addition to the task-specific teachers hit (φit (·)), i ∈ {1, . . . , N} used in traditional distillation, we also include the generalist teacher trained on ImageNet (denoted as h0t (φ 0 t (·))). The rationale behind this construction is that we learn task knowledge from the task-specific model on the proxy dataset, but also force the student to retain knowledge from the generalist teacher to prevent over-specialization. We learn the student by optimizing the loss:
L = ∑
x∈Dproxy
( λ0 Ldistill ( h0t (φ 0 t (x)), h 0 s (φs(x)) )︸ ︷︷ ︸ generalist teacher term + N∑ i=1 λi Ldistill ( hit (φ i t (x)), h i s(φs(x)) )︸ ︷︷ ︸ traditional distillation ) (1)
with a distillation loss Ldistill. Our key difference from traditional distillation is the generalist teacher term. Setting λ0 = 0 while keeping all other conditions (proxy dataset, network initialization, etc.) yields a standard multi-teacher distillation baseline. Note that this is also distinct from ensemble knowledge distillation, where one averages the homogeneous output of every model and distills the mean onto a single student head, since our teacher outputs are heterogeneous (different tasks).
Our method is agnostic to the underlying distillation method. We can use the standard knowledge distillation (KD) loss (Hinton et al., 2015) which is cross-entropy with temperature T = 2, i.e. Ldistill(pt, ps) = − ∑C c=1 p (c) t log(p (c) s ) where c indexes the C classes, and pt =
softmax ( hit (φ i t (x))/T ) , ps = softmax ( his(φs(x))/T ) . Alternatively, we can use a state-of-theart distillation method, e.g., CRD (Tian et al., 2020). CRD uses the KD loss and adds a contrastive embedding loss between hiCRD,t(φ i t (x)) and h i CRD,s(φs(x)) with a pair of trainable linear embedding layers hiCRD,t and h i CRD,s that are used only in training and discarded afterwards. The teacher φ i t ’s
are trained on completely different tasks, so for CRD we use different embedding layers for each teacher-student-head pair.
We initialize the student backbone φs and its 0-th head h0s using the generalist model’s weights φ0t , h 0 t , whereas other heads are randomly initialized. Since it is important to maintain pre-trained model’s representational power, we simply set the loss weights λ0 = 1 and λi = 1N for 1 ≤ i ≤ N . This forces the learned representation φs to balance between learning general and taskspecific features, which benefits future downstream transfer. After training the student we evaluate the resulting representation φs(·) on multiple downstream tasks {Did}. For each task j we can either fine-tune the whole model hjd(φ j d(·)) or keep the student representation fixed and only learn the classifier head hjd(φs(·)) which is often referred to as a linear probe.

heading: 3 EXPERIMENTAL SETUP

text:
Datasets and downstream tasks. We utilize datasets from a variety of domains to generate teachers and downstream tasks: Cars196 (Krause et al., 2013), Resisc45 (Cheng et al., 2017) (remote sensing images), iFood (Kaur et al., 2019) and Food101 (Bossard et al., 2014), iFashion (Guo et al., 2019), DTD (Cimpoi et al., 2014) (describable textures), iNaturalist (Horn et al., 2018) (species classification, 2019 challenge version), CUB Birds (Wah et al., 2011), Flowers (Nilsback & Zisserman, 2008), Caltech256 (Griffin et al., 2007), and Aircrafts (Maji et al., 2013). Among these, iFood and Food101 are the same domain, and Birds and Flowers are subdomains of iNaturalist. We checked for near-duplicates between these datasets using perceptual hash (Buchner, 2021), and found negligible duplication: 1 out of 134k iNaturalist (50% classes we used) is a duplicate of CUB, and 8 out of 130k/100k images are duplicates between iFood and Food101.
To evaluate if a consolidated student representation has learned features relevant to a specific domain, we require downstream tasks that are related (in the same domain) to that of each task-specific teacher. Except for Food101 and iFashion, we split each dataset at random into two disjoint sets which each contain only 50% of the classes. We take the first half of the dataset, named “t-split”, and use as Dit , to train a task-specific teacher. We use the second half of each dataset, named “dsplit”, as one of the downstream tasks Djd . We primarily evaluate on few-shot downstream transfer, where we randomly sample 5 training images from each available class in Djd , but always use the entire test set for evaluation. We use all samples for those classes with < 5 images. For iFashion, which is a multi-task multi-label dataset, we use all images but with a subset of labels (e.g., those related to clothing category) as Dit for teacher training. We then use all images with a disjoint set of labels (e.g., those related to sleeve style) as a downstream task Djd for evaluating transfer. For iFashion’s few-shot scenario, we randomly subsample 1000 images for downstream training, and evaluate on all test data. We do not train any teacher on Food101 so the complete set of classes are used as a downstream task.
During distillation and consolidation, we use ImageNet (Russakovsky et al., 2015) (ILSVRC12) as the proxy dataset Dproxy for most experiments. However, we also show results with Places365standard (Zhou et al., 2018) or iNaturalist (Horn et al., 2018) as the proxy.
Compared methods and Criteria. As a reminder, our goal is not to duplicate upstream teacher predictions, but rather to improve transfer performance of distilled representations. Therefore, we evaluate each student’s φs using its performance when transferred to various downstream datasets {Djd} and compare multiple methods for initializing the downstream representation φ j d:
(1) ImageNet-pretrained φ0t , a strong baseline for transfer learning. (2) ImageNet-pretrained φ 0 t fine-tuned on the soft-labels produced by the ImageNet-pretrained model with batchnorms in test mode. This baseline aims to isolate the effect of soft-labels / self-distillation on model performance. (3) The task-specific teacher φ1t (or one of the teachers when N > 1) without further distillation. (4) Distillation with N teachers using either standard knowledge distillation (KD) or contrastive representation distillation (CRD). This distills only task-specific teacher(s) on Dproxy, without the ImageNet teacher h0t (φ 0 t (·)). (5) Our consolidated representation φs which includes h0t (φ0t (·)) as a teacher, using either KD or CRD as the underlying distillation loss. (6) A multi-task learning oracle (MTL) that jointly trains on ImageNet and teacher data Dt rather than on unlabeled proxy data.
For comparison fairness, (2-6) are all ImageNet pre-trained (i.e. initialized with (1)) before further training. Both (4) and (5) are using the same proxy dataset (i.e. ImageNet for most experiments).
We use the transfer accuracy on downstream tasks Djd to measure each representation’s power. We primarily use the few-shot linear probe (train a 5-shot linear SVM as hjd over fixed φ) on D j d’s training set (single training run for full dataset, few-shot performance averaged over 50 random subsampling trials). We also verify our results hold when fine-tuning the student representation hjd(φd(·)) on few-shot or full dataset splits.
Implementation details. We will release our code to reproduce this paper upon acceptance. For experimental fairness, we use the same network (ResNet50 (He et al., 2016)) / hyperparameters / ImageNet initialization for the baselines and ours. For more details, please see appendix A.

heading: 4 RESULTS

text:
Exp. 0: Motivational analysis – traditional distill vs. representation consolidation To highlight the difference between representation consolidation and traditional distillation, we use either iFood or Resisc45 (t-split) as D1t to train teachers, and run distillation/consolidation with N = 1. We test on iFood (d-split), Food101 (full), and Resisc45 (d-split) as downstream tasks (t-split and d-split are disjoint 50% classes of each dataset; see Section 3)
On downstream tasks Djd , we follow representation learning’s evaluation protocol (few-shot linear probe): train linear SVM hd on φ(x), x ∈ Djd , evaluate on D j d’s test set. We also evaluate each network following the traditional distillation evaluation protocol, i.e. directly evaluate h1t (φ 1 t (·)) or h1s (φs(·)) on the upstream task D1t ’s test set. The results are shown in Table 2. If we only focus on the upstream task D1t , then traditional (proxy) distillation almost matches the teacher’s performance, and representation consolidation performs worse than both. However, if we instead focus on the downstream transfer performance onDjd , we see the opposite trend. In Table 2a with iFood representations, for the teacher-related downstream tasks Food101 and iFood d-split (the disjoint classes split from the same dataset as D1t ), we see a benefit of consolidation over both taskspecific teacher and generic pre-trained features. On the unrelated downstream task (Resisc45), where the generalist teacher excels, we notice that our method almost matches the generalist performance even though we improve over it for the food tasks. We see a similar trend in Table 2b: the consolidated student outperforms the teacher and the generalist on the teacher-related task (Resisc45 d-split) but still retains the generalist teacher’s performance on unrelated food tasks. This clearly demonstrates the significant difference between upstream and downstream transfer.
We perform nearly the same as the MTL oracle which trains also on the teacher datasets Dt, indicating that while we avoid replayingDt, our simple consolidation method withDproxy can extract nearly all of its benefit. Note that although we have carefully optimized the MTL oracle for downstream performance, adding state-of-the-art MTL techniques could outperform our MTL implementation.
We only demonstrate that it is not trivial to outperform our method using original teacher training data.
Exp. 1: Improving student representation when N = 1. We show that this advantage of consolidation over baselines holds for a wide range of upstream and downstream datasets. We also compare to CRD to show state-of-the-art distillation often underperforms even when specifically targeting representation learning. Fig. 2 summarizes few-shot SVM accuracy using different representations relative to ImageNet-pretrained (See appendix for the full Fig. 6 and raw numbers).
The conclusions are similar regardless of using KD or CRD – consolidation outperforms (Cars196, Resisc45, iFood, CUB, Aircrafts, iNaturalist teachers) or matches (iFashion, DTD, Flowers, Caltech256 teachers) ImageNet pre-trained model performance on related downstream tasks, and matches its performance on unrelated ones. In contrast, both traditional distillation methods (1) underperform consolidation on related downstream tasks for all teachers except Cars196, iFashion, and Aircraft, and (2) drastically underperform both consolidated and ImageNet features on unrelated downstream tasks – although CRD is better than KD, it still degrades performance. Notably, traditional distillation methods underperform ImageNet even on related downstream tasks for Resisc45, DTD, Caltech256, and (for KD only) Flowers teachers. We match the MTL oracle performance across the board, reaping the teacher dataset’s benefit without replaying it.
Exp. 2: Consolidating representations withN > 1. We can merge multiple task-specific teacher representations that are related to get a better one. In addition, our method is not constrained to merging models with the same architecture or φit ’s feature space dimensions like Geyer et al. (2019). To illustrate this, we split the “t-split” of Cars196 and iFood into five random splits containing 10% of the original classes. We train a ResNet18 teacher on each of the five splits. Then, we use either traditional distillation or representation consolidation to merge these models with a ResNet50 generalist teacher. We compare the resulting representations with the teacher model’s and ImageNet. Fig. 3a shows this result. For downstream tasks, we obtain better performance than using only one of the five teachers on both related and unrelated tasks, especially for iFood and Food101 where the teachers themselves underperform on related downstream task. This shows that our method can benefit from even teachers whose representation is weaker, as long as they have domain knowledge.
Full comparison with traditional distillation and representation consolidation from only one of the five teachers are in the appendix – we gain performance on similar Djd by using five task-specific teachers instead of one, and we outperform both traditional distill methods on related downstream

heading: Accuracy (rel. to ImageNet-pretrained model)

text:
tasks for iFood/Food101. We also show results when using all ResNet50 teachers for the five splits in the appendix. The gaps are closer but the results are similar and the conclusions are identical.
Finally, we explore merging models from different domains to form a multi-domain consolidated student. See Fig. 3b. We observe that we can outperform ImageNet on all related downstream tasks, but the performance gain is smaller than representation consolidation in just one domain. We show in the appendix that ours+KD outperforms KD on most downstream tasks except Cars196.
Exp. 3: Fine-tuning downstream. We also verify that our conclusions generalize to fine-tuning as well, especially without few-shot sampling. Fig. 4 shows an excerpt of our results that include full dataset transfer. See appendix for fine-tuning using few-shot and full d-splits, whose conclusions are the same as this section’s. Fine-tuning allows the representation to change into one that better suits the downstream task, so the benefit of teachers’ domain knowledge shrinks compared to fixed φs with linear SVM h j d. Despite this, the conclusions are the same as the fixed φs scenario.
Exp. 4: Influence of the Loss weights. Fig. 5a shows the effect of the choice of loss weight on our method – when we use λ0 = 12 , λi = 3 2N (i > 0), we gain performance on the related downstream task but lose performance on unrelated ones, whereas using λ0 = 32 , λi = 1
2N (i > 0) gives us the opposite. We can potentially use this trade-off to suppress unreliable teachers’ influence.
Exp. 5: Proxy data choice. Fig. 5b shows results replacing ImageNet with Places365 or iNaturalist as the Dproxy dataset. Places365 yields similar performances as using ImageNet, while a narrower-scope iNaturalist is weaker on unrelated downstream tasks. Jointly training on Places365 labels is not helpful, meaning that the improvement comes from the generalist teacher, not the data.

heading: 5 RELATED WORK

text:
Our goal is to maximize the transferability of a representation consolidated from multiple teachers. This downstream transfer aspect has received little attention in the literature (Liu et al., 2019; Geyer et al., 2019; Tian et al., 2020), but this problem formulation is closely related to prior work on multi-model merging and distillation with proxy data.
Multi-model merging. It is useful to combine separate models that perform different tasks into a single model for efficiency and performance benefits. Knowledge Concentration (Gao et al., 2017) combine teachers trained on subsets of 100k classes in the EFT dataset (Gao et al., 2017) into one improved single model, using handcrafted sparse connections for the final student layers. Chou et al. (2018) merge CNNs by combining the kernel weights of different models by clustering and lookup, followed by fine-tuning. Zhao et al. (2020) merge object detection models when some classes may be the background to other models. Vongkulbhisal et al. (2019) combine models with overlapping classes using the combined train set, by deriving soft labels from intra-model class correspondence. Ye et al. (2020) progressively train a GAN to regenerate proxy data for all teachers, and combine teachers layer by layer. Chakraborty et al. (2018) and Park & Kwak (2020) aggregate ensembles of members trained on the same task. In one-shot federated learning (Guha et al., 2019), multiple clients with their own private data train a model each, and finally merge the models using an ensemble or distillation in a way that protects their data privacy.
Unlike our approach, these methods merge models in order to perform exactly the same task as the teacher models, are not concerned with the performance of the student when transferred to a downstream task, and often (Gao et al., 2017; Chou et al., 2018; Zhao et al., 2020; Vongkulbhisal et al., 2019; Chakraborty et al., 2018; Park & Kwak, 2020) require revisiting the original training images. We have shown that these teachers themselves have poor transferability compared to a simple pre-trained baseline, and learning only from them (even using state-of-the-art CRD (Tian et al., 2020)) yields suboptimal transfer learning performance.
Multi-model merging for transfer. Knowledge Flow (Liu et al., 2019) connects the student to multiple teachers’ intermediate layers to kick-start target task training, and gradually penalizes its reliance on teacher models over time. Geyer et al. (2019) uses IMM (Lee et al., 2017) to merge multiple models using their diagonal approximated Fisher information matrix (FIM) to balance different
teacher’s weight importance, merge, and fine-tune. Computing the FIM requires reprocessing the original teacher training data (unlike our approach that only needs generic proxy data). Furthermore, this method requires homogeneous architectures for students and teachers, while ours works on any architecture combinations. These methods directly optimize performance on the downstream task and thus require re-running for each different target dataset. Our approach is more efficient, as it only requires consolidating teachers once to improve the pre-trained representation independently of the downstream task. Finally, we note that the representations learned were not compared to a strong baseline (i.e., pre-training on ImageNet), which we argue is a prerequisite for being useful in realworld applications. One of our main contributions is observing the need for including a generalist teacher, an insight which is orthogonal to, and could be combined with these previous approaches.
Distillation with representation losses (“representation distillation”) tries to capture additional structure of feature representations by aligning student and teacher feature activations during distillation. Koratana et al. (2019) compress models by adding L2 losses between intermediate representations of the teacher and students. Aguilar et al. (2020) use KL divergence and cosine similarity to make the attention and representation of intermediate features of the student and teacher similar. Tian et al. (2020) adds contrastive representation learning loss to the penultimate layer to preserve feature structures, by maximizing each image’s student and teacher features’ mutual information. Despite their focus on yielding better representations, these methods fully rely on the teacher’s transferability. We show that this yields downstream performance worse than generalist baselines.
Proxy data (“data-free”) distillation transfers the input-output function of a teacher network to a student network without using the teacher’s training data. (Yalniz et al., 2019; Orekondy et al., 2019) opt to use a large general proxy dataset to query the teacher, and their teacher outputs to on this data to train the student. Other methods (Nayak et al., 2019; Chen et al., 2019; Haroush et al., 2020; Chawla et al., 2021) generate proxy data directly from the trained models, and use this data to train the students. Further, Micaelli & Storkey (2019); Yin et al. (2020) also encourage generating samples the student and teacher disagree on. Lastly, other methods require the original dataset to compute meta-data information such as feature cluster mean. (Lopes et al., 2017; Bhardwaj et al., 2019) Some train a GAN from the teachers to maximize chosen class predictions (Fang et al., 2019; Yoo et al., 2019; Ye et al., 2020), sometimes also batchnorm statistics (Luo et al., 2020; Xu et al., 2020b), and sometimes on proxy data instead. (Addepalli et al., 2020; Besnier et al., 2020) Some work combines self-supervised learning with distillation (Tian et al., 2020; Xu et al., 2020a; Fang et al., 2021) to improve the representation performance on the original task.
These works try to emulate the original data for standard distillation, and do not concern the student’s transferability or merging multiple teachers. Our simple method already performs comparably to a multi-task oracle that uses the original teacher dataset, indicating that these more complex methods that improve upstream task distillation may be unnecessary when the goal is downstream transfer.
Incremental learning is also related to our overall goal of growing a library of expert representations. These methods continually learn tasks or classes but often with limited access to past training data (Li & Hoiem, 2018; Rebuffi et al., 2017; Kirkpatrick et al., 2017; Zenke et al., 2017; Hu et al., 2019; Yin et al., 2020; Prabhu et al., 2020). Our approach addresses many of the same challenges by consolidating knowledge in the form of feature representations without revisiting old data used to train teachers, but our “increments” are whole tasks given in the form of trained models instead of data, whose labels may overlap. Multi-task learning (Caruana, 1998) assumes all datasets are available and trains jointly on them whereas we assume task-specific datasets absent. We perform comparably to this oracle.

heading: 6 SUMMARY

text: In this paper, we show that traditional distillation can result in a representation suboptimal for downstream task transfer learning, because it only focuses on preserving the end-to-end input-output mapping of the old task. We propose representation consolidation with the generalist model as an additional teacher. Our method preserves the wide-range transferability of the strong ImageNet baseline and improve the performance for both related and unrelated downstream tasks over traditionally distilled networks. We show that we can merge multiple models in the same domain to get a better representation than any single model.

heading: B COMPLETE FIGURES FOR SECTION 4 EXPERIMENTS

text:
We now provide the full graphs and table for our experiments. Note that we have summarized these results and all conclusions in the main paper.
Exp. 1: Improving student representation when N = 1. Fig. 6 and Table 3 show full results for Fig. 2. Please see the main paper for analysis and conclusions.
Exp. 2: Consolidating representations with N > 1. Figs. 7, 8, 9 shows full results for Fig. 3.
For Figs. 7 and 8 same-domain model merging, in addition to the main paper results merging ResNet18 φit and ResNet50 φ 0 t , we show similar results for all teachers being ResNet50 in Fig. 8. We also show comparison to traditional distill with 5 teachers, representation consolidation with only 1 teacher, and the teacher itself (best out of 5 according to related Djd performance). The conclusions are the same, and merging five teachers using representation consolidation outperforms all baselines on both related and unrelated downstream tasks (except Cars196 with related Djd against traditional distill).
For multi-domain model merging in Fig. 9, we compare to traditional distill with multi-task learning. We outperform or match it on downstream Djd except for Cars196. We also explore using a concatenation of multiple large unlabeled datasets as Dproxy. With a more diverse proxy, the performance

heading: Teacher: iNaturalist (t-split) model

heading: Teacher: Cars196+Resisc45+iFood (t-split) model

text: drops a little for related Djd and stays similar for unrelated D j d , suggesting we are somewhat insensitive to choice of datasets, but a more diverse proxy data may not provide better model merging performance.

text:
Exp. 3: Fine-tuning downstream. Figs. 10, 11 show full results for Fig. 4. These extensive results show that in both few-shot and full-shot scenarios, and both single-domain and multi-domain scenarios, we have the same conclusions as the fixed representation experiments.
Exp. 5: Proxy data choice. Fig. 12 shows full results for Fig. 5b. This shows similar results with different teacher Dt, i.e. Places365 performs similarly to ImageNet as the proxy, but the narrowerscope iNaturalist underperforms.

heading: C RAW NUMBER ACCURACY TABLES FOR ALL EXPERIMENTS

text: • Table 4: fixed representation few-shot results for Figs. 2, 3 (Figs. 6,7,8,9 in the appendix). • Table 5: fixed representation few-shot results for Fig. 5. • Table 6: fine-tuning few-shot results for Fig. 10 in the appendix. • Table 7: fine-tuning full-shot results for Fig. 4 (Figs. 11 in the appendix.)

heading: D LIMITATIONS

text: We assume we have perfect knowledge of which tasks form the same domain and which tasks do not belong to a domain. Our performance drops when teachers from different domains are consolidated. In future work, we plan to automatically determine how to cluster a large amount of teachers into domains. In addition, one of our contributions assume the existence of a strong representation baseline such as the ImageNet pre-trained model, which is true for fields like images and language, but not for others (e.g., 3D reconstruction). Our method also takes the teachers as is and learn from them, and any mistakes made by the teachers can be propagated during the consolidation. Possible mitigations include using a better teacher that makes less such mistakes, or using regularization on both teacher and student training, such as making similar inputs map to similar outputs.