summaries: The authors consider the problem of designing accurate and computationally efficient deep sparse networks (DSNs), which are an important problem in applications with sparse features such as click-through rate or movie recommendation. The authors propose a new approach based on neural architecture search (NAS) using two techniques: distillation of the search space (of cross-features), and a progressive search algorithm. They show that their approach leads to improvements in both accuracy and inference time, on three datasets.

summaries: This paper targets applying NAS to Deep Sparse Network (DSN) domain. To search for a good DSN, previous methods that apply DARTS based method to search an encoding vector, where each element represents the probability to select feature-interaction. Due to the interaction grows in a factorial way, the search parameters will soon become intractable. This work proposes to decompose this vector into multiple low-rank ones, to reduce the search space dramatically.  Experiments on various datasets show a clear improvement comparing to the previous baseline AutoFIS. I would recommend accepting this one but since I am not really familiar with the DSN domain, I might change my mind based on other reviews.

summaries: This paper proposes a method for searching for the interaction of the feature columns in a CTR prediction application. The paper title suggests an application for deep sparse networks but it doesn’t seem to be explicitly tied with the requirement of the input being sparse. To this end, the paper proposes an iterative algorithm that incrementally learns the interaction tensor from lower rank to higher rank, and from lower order to higher order. Empirically it observes some benefits compared to current competitive methods on Criteo, Avazu and ML1M datasets.

summaries: This paper proposes a neural architecture search approach, PROFIT,  for deep sparse networks.  PROFIT introduces a distilled low-rank search space and a progressive search algorithm from the lower orders. The proposed method is evaluated on three benchmark datasets and the authors conduct extensive ablation study and discussion on the method. The paper is well written and very easy to follow even for people without a NAS background.

summaries: In this paper, the authors study the problem of deep sparse prediction based on NAS. In order to reduce the search cost, they propose a distilled search space, which is a low-rank approximation of the full space. They also propose a progressive differentiable search to capture the order-priority in sparse prediction. Numerous experiments are conducted to demonstrate the effectiveness of the proposed method.

abstractText: Deep sparse networks (DSNs), of which the crux is exploring the high-order feature interactions, have become the state-of-the-art on the prediction task with highsparsity features. However, these models suffer from low computation efficiency, including large model size and slow model inference, which largely limits these models’ application value. In this work, we approach this problem with neural architecture search by automatically searching the critical component in DSNs, the feature-interaction layer. We propose a distilled search space to cover the desired architectures with fewer parameters. We then develop a progressive search algorithm for efficient search on the space and well capture the order-priority property in sparse prediction tasks. Experiments on three real-world benchmark datasets show promising results of PROFIT in both accuracy and efficiency. Further studies validate the feasibility of our designed search space and search algorithm.

authors:
Bowen Baker
Otkrist Gupta
Nikhil Naik
Ramesh Raskar

title: Designing neural network architectures using reinforcement learning

venue: In International Conference on Learning Representations,

authors:
Han Cai
Chuang Gan
Tianzhe Wang
Zhekai Zhang
Song Han

title: Once-for-all: Train one network and specialize it for efficient deployment

authors:
Han Cai
Ligeng Zhu
Song Han

title: Proxylessnas: Direct neural architecture search on target task and hardware

authors:
Heng-Tze Cheng
Levent Koc
Jeremiah Harmsen
Tal Shaked
Tushar Chandra
Hrishi Aradhye
Glen Anderson
Greg Corrado
Wei Chai
Mustafa Ispir

title: Wide & deep learning for recommender systems

venue: In 1st workshop on deep learning for recommender systems,

authors:
Weiyu Cheng
Yanyan Shen
Linpeng Huang

title: Adaptive factorization network: Learning adaptive-order feature interactions

venue: In AAAI Conference on Artificial Intelligence,

authors:
Minsu Cho
Mohammadreza Soltani
Chinmay Hegde

title: One-shot neural architecture search via compressive sensing

venue: arXiv preprint arXiv:1906.02869,

authors:
Corinna Cortes
Mehryar Mohri

title: Auc optimization vs. error rate minimization

venue: In Advances in Neural Information Processing Systems,

authors:
Thomas Elsken
Jan Hendrik Metzen
Frank Hutter

title: Neural architecture search: A survey

authors:
Huifeng Guo
Ruiming Tang
Yunming Ye
Zhenguo Li
Xiuqiang He

title: Deepfm: a factorization-machine based neural network for ctr prediction

venue: In International Joint Conference on Artificial Intelligence,

authors:
Zichao Guo
Xiangyu Zhang
Haoyuan Mu
Wen Heng
Zechun Liu
Yichen Wei
Jian Sun

title: Single path one-shot neural architecture search with uniform sampling

venue: In European Conference on Computer Vision,

authors:
Song Han
Huizi Mao
William J Dally

title: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding

authors:
Elad Hazan
Adam Klivans
Yang Yuan

title: Hyperparameter optimization: a spectral approach

authors:
Xiangnan He
Tat-Seng Chua

title: Neural factorization machines for sparse predictive analytics

venue: In International ACM SIGIR conference on Research and Development in Information Retrieval,

authors:
David W Hosmer Jr.
Stanley Lemeshow
Rodney X Sturdivant

title: Applied logistic regression, volume 398

authors:
Yuchin Juan
Yong Zhuang
Wei-Sheng Chin
Chih-Jen Lin

title: Field-aware factorization machines for CTR prediction

venue: In ACM Conference on Recommender Systems,

authors:
Farhan Khawar
Xu Hang
Ruiming Tang
Bin Liu
Zhenguo Li
Xiuqiang He

title: Autofeature: Searching for feature interactions and their architectures for click-through rate prediction

venue: In ACM International Conference on Information & Knowledge Management,

authors:
Tamara G Kolda
Brett W Bader

title: Tensor decompositions and applications

venue: SIAM review,

authors:
Jianxun Lian
Xiaohuan Zhou
Fuzheng Zhang
Zhongxia Chen
Xing Xie
Guangzhong Sun

title: xdeepfm: Combining explicit and implicit feature interactions for recommender systems

venue: In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,

authors:
Bin Liu
Niannan Xue
Huifeng Guo
Ruiming Tang
Stefanos Zafeiriou
Xiuqiang He
Zhenguo Li

title: Autogroup: Automatic feature grouping for modelling explicit high-order feature interactions in CTR prediction

venue: In International ACM SIGIR Conference on Research and Development in Information Retrieval,

authors:
Bin Liu
Chenxu Zhu
Guilin Li
Weinan Zhang
Jincai Lai
Ruiming Tang
Xiuqiang He
Zhenguo Li
Yong Yu

title: AutoFIS: Automatic feature interaction selection in factorization models for click-through rate prediction

venue: In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,

authors:
Hanxiao Liu
Karen Simonyan
Yiming Yang

title: DARTS: Differentiable architecture

authors:
Siyi Liu
Chen Gao
Yihong Chen
Depeng Jin
Yong Li

title: Learnable embedding sizes for recommender systems

authors:
Yuanfei Luo
Mengshuo Wang
Hao Zhou
Quanming Yao
Wei-Wei Tu
Yuqiang Chen
Wenyuan Dai
Qiang Yang

title: Autocross: Automatic feature crossing for tabular data in real-world applications

title: Machine learning: a probabilistic perspective

venue: MIT press,

authors:
Yanru Qu
Han Cai
Kan Ren
Weinan Zhang
Yong Yu
Ying Wen
Jun Wang

title: Product-based neural networks for user response prediction

venue: In International Conference on Data Mining,

title: Pruning algorithms-a survey

venue: IEEE Transactions on Neural Networks,

title: Factorization machines

venue: In IEEE International Conference on Data Mining,

authors:
Mark Sandler
Andrew Howard
Menglong Zhu
Andrey Zhmoginov
Liang-Chieh Chen

title: Mobilenetv2: Inverted residuals and linear bottlenecks

venue: In IEEE Conference on Computer Vision and Pattern Recognition,

authors:
David So
Quoc Le
Chen Liang

title: The evolved transformer

authors:
Qingquan Song
Dehua Cheng
Hanning Zhou
Jiyan Yang
Yuandong Tian
Xia Hu

title: Towards automated neural interaction discovery for click-through rate prediction

authors:
Weiping Song
Chence Shi
Zhiping Xiao
Zhijian Duan
Yewen Xu
Ming Zhang
Jian Tang

title: Autoint: Automatic feature interaction learning via self-attentive neural networks

venue: In ACM International Conference on Information and Knowledge Management,

authors:
Xiaoyuan Su
Taghi M Khoshgoftaar

title: A survey of collaborative filtering techniques

venue: Advances in Artificial Intelligence,

authors:
Mingxing Tan
Bo Chen
Ruoming Pang
Vijay Vasudevan
Mark Sandler
Andrew Howard
Quoc V Le

title: Mnasnet: Platform-aware neural architecture search for mobile

authors:
Wei Wen
Chunpeng Wu
Yandan Wang
Yiran Chen
Hai Li

title: Learning structured sparsity in deep neural networks

authors:
Jun Xiao
Hao Ye
Xiangnan He
Hanwang Zhang
Fei Wu
Tat-Seng Chua

title: Attentional factorization machines: Learning the weight of feature interactions via attention networks

authors:
Sirui Xie
Hehui Zheng
Chunxiao Liu
Liang Lin

title: Snas: stochastic neural architecture search

authors:
Niannan Xue
Bin Liu
Huifeng Guo
Ruiming Tang
Fengwei Zhou
Stefanos P Zafeiriou
Yuzhou Zhang
Jun Wang
Zhenguo Li

title: AutoHash: Learning higher-order feature interactions for deep CTR prediction

venue: IEEE Transactions on Knowledge and Data Engineering,

authors:
Yibo Yang
Hongyang Li
Shan You
Fei Wang
Chen Qian
Zhouchen Lin

title: Ista-nas: Efficient and consistent neural architecture search by sparse coding

venue: Advances in Neural Information Processing Systems,

authors:
Quanming Yao
Mengshuo Wang
Yuqiang Chen
Wenyuan Dai
Yu-Feng Li
Wei-Wei Tu
Qiang Yang
Yang Yu

title: Taking human out of learning applications: A survey on automated machine learning

venue: Technical report,

authors:
Quanming Yao
Ju Xu
Wei-Wei Tu
Zhanxing Zhu

title: Efficient neural architecture search via proximal iterations

authors:
Xiangyu Zhao
Haochen Liu
Hui Liu
Jiliang Tang
Weiwei Guo
Jun Shi
Sida Wang
Huiji Gao
Bo Long

title: Autodim: Field-aware embedding dimension searchin recommender systems

venue: In Web Conference,

authors:
Barret Zoph
Quoc V Le

title: Neural architecture search with reinforcement learning

heading: 1 Introduction

text:
The deep sparse network (DSN) [9, 4] is a special type of deep networks, which targets learning from sparse and categorical features. Such a learning problem frequently appears in many industrial applications. Take the click-through rate (CTR) prediction as an example, features such as user id, user age, item id, and item category, are usually sparse and high-multidimensional [4]. In advertisement recommendation [27], there are sparse features such as category, brand, etc. In fraud detection [23], there are sparse features such as income level, region, etc.
A general DSN framework (see Figure 1) consists of three layers. First, the embedding layer transforms the raw sparse features into dense embeddings. Then the feature-interaction layer can construct cross-features of raw features’ embeddings via feature-interaction operation. The cross-features are further fed into the output layer to calculate the prediction score. To achieve high prediction accuracy, researchers found that the crux of the deep sparse networks is to design complex featureinteraction layers [9, 31, 5, 18, 25].
For example, in Figure 1, the second-order cross-feature, AB, is constructed by two raw features, A and B, via feature interaction. DeepFM [9] feed the two-order cross-features into multi-layer perceptions for implicitly generating high-order cross-features. Further advances replace the MLP in DeepFM with more complex neural networks, including deep cross network in xDeepFM [18], outer product network in PNN [25], self-attention network in AutoInt [31], and logarithmic transformation
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
network in AFN [5]. Since these deep neural networks work as a black box, high-order cross-features are learned in an implicit manner.
Neural architecture search (NAS) [8, 39], has become a powerful approach in replacing human experts in proposing both effective and lightweight models for various areas such as computer vision [? ], natural language processing [29], etc. More recently, there are some works that adopt neural architecture search for designing the feature-interaction layer in DSN. For example, SIF [40] searches simple first order interaction function between users and items; AutoFIS [20] and AutoGroup [19] explicitly search for more powerful higher order cross-features. Although the searched featureinteraction layer is more lightweight to some extent, these methods are suffering from high search costs, resulting in limited prediction performance and application value. However, these works require extremely large search costs, limiting their application values on large datasets.
In general, the human-designed DSNs are faced with the efficiency issue since they mainly pursue prediction accuracy. In fact, efficiency has a very high impact on the application value of machine learning models in real-world scenarios [3, 33, 28]. Considering that today’s machine learning systems are always deployed at diverse hardware platforms, inference time and model size are the two most significant factors. Specifically, the inference time can largely affect the time delay of the user services, which largely affects the application value. The model size can determine whether the model can be deployed at specific devices, which sets the constraint to the model’s application scenarios, especially for edge devices. Therefore, these complicated deep sparse networks are still limited due to their inferior efficiency.
In this work, to address the above-mentioned efficiency-accuracy dilemma of state-of-the-art deep sparse networks, we propose to utilize NAS to design lightweight and effective models. However, this problem suffers from two challenges. exponentially with the order of cross-features and searching in this space is very hard and time-consuming. Second, in sparse prediction tasks, there exists the order-priority property, which is hard to handle. Specifically, the lower-order cross-features should be assigned higher priority compared with high-order ones. To tackle them, we propose a PROFIT method (short for PROgressive Feature InTeraction Search). We first design a distilled search space, a low-rank approximation of the full search space, which can vastly reduce the difficulty of finding desired models. More precisely, the full search space, which uses a tensor to represent all possible architectures, is approximated by the composition of low-rank tensors. We then develop a progressive search algorithm that searches cross-features from lower orders to higher orders sequentially. When searching for the higher orders, the corresponding architecture parameters of lower orders are fixed, helping distinguish the various importance of different orders. We conduct experiments on three real-world benchmark datasets and compare both accuracy and efficiency with state-of-the-art models. Empirical results demonstrate that the model searched by our PROFIT can achieve high accuracy while keeping low computation cost and model size. Further studies verify the feasibility of each design, consisting of the search space and search algorithm.

heading: 2 Related Work

heading: 2.1 Neural architecture search (NAS)

text: Neural architecture search (NAS) [42, 1, 39] replaces human experts in designing neural networks architectures, which has been demonstrated a promising approach in many research areas, e.g., vision and natural language processing. There are two significant perspectives that determines its utility: search space and search algorithm. The search space is defined as the set of all possible architectures. It is task-specific and should be general enough to cover existing models but not too general to incur high search costs. The search algorithm attempts to find the desired architecture in a given space. A good search algorithm should well explore domain properties to find good architectures and achieve high search efficiency. More recently, some pioneer works [10, 2] propose NAS methods for automatically designing lightweight models for computer vision tasks, which can reduce the inference time on edge devices. Compared with the existing works, our work approach the problem of searching for deep sparse networks, of which there exist critical challenges of large search space and high search cost.

heading: 2.2 Deep Sparse Networks (DSNs)

text:
As mentioned in Section 1, deep sparse networks (DSNs) serve as the state-of-the-art solution for the sparse prediction task. Its crux is to extract cross-features, which can have strong abilities in prediction, from raw input features.
Here, for the sparse prediction task, we recap the evolution from earlier shallow models to DSNs. Factorization machine [27] is a pioneer work that assigned a linear matrix factorization operation to all possible second-order cross-features (built with two raw features), which was extended by NFM [13] and AFM [35]. To overcome the shortcomings of these shallow feature-interaction operations, deep sparse networks [9, 18, 5, 31, 25] are proposed. In general, deep sparse networks adopt neural feature-interaction layers to extract cross-features, as illustrated in Figure 1. Wide&Deep [4] and DeepFM [9] feed the low-order features into multi-layer perceptrons (MLPs) to implicitly reveal any order cross-features. More complex neural networks are proposed to serve as the feature-interaction layer, replacing MLPs, such as the compressed interaction network (CIN) in xDeepFM [18] and outer product network in PNN [25].
There are some recent works [31, 5] approaching the cross-feature search of DSNs. Specifically, in AutoInt [31], the feature-interaction layer is designed to stack multi-layer self-attention architectures; AFN [5] used a logarithmic transformation layer to convert the order to a learnable parameter and then adopted a feature summation as the feature-interaction layer, which fuzzes the order of cross-features.
However, there is no model that can achieve both promising efficiency and high accuracy. Thus, it is important to solve the efficiency-accuracy dilemma in DSNs. It is worth mentioning that there are some recent works [22, 41] that adopt AutoML in feature embeddings, which is the input of DSNs, for reducing the parameter number of embeddings.

heading: 2.3 Deep Network Sparsification

text: Besides, it is worth mentioning there is another research direction closely related to this work, network sparsification [26, 34, 11, 37], which aims to prune the network components or parameters to reduce the network size while keeping a little performance decrease. Different from them, our work searches for the architecture in an interpretable and explicit manner: finding architectures that reserve important cross-features and remove useless ones; for network pruning models, embedding parameters are dropped in an implicit manner — the pruning masks are always implicitly learned from data, does not provide any interpretability. In deep sparse prediction tasks, it is more significant to explicitly lighten the model at the level of cross-feature, while existing works of network sparsification or network pruning is at the level of latent dimension.

heading: 3 The Proposed Method

heading: 3.1 The Search Problem

text:
Existing works of DSNs [18, 31, 5] have shown that the feature-interaction layer is an important component. It can be defined as a network module of which the input is embeddings of raw features, and the output is interactions of embeddings. The interactions have extracted features’ co-relations and can then be fed to the output layer for obtaining the final prediction results. A feature-interaction layer that extracts o-th order cross-features can be formulated as
Layero= { fc1 fc2 · · · fco |co ∈ Co } , s.t. |Co|≤No, (1)
of which c denotes a l-size vector of involved features’ indexes of a o-th cross-feature (co, c2, ..., co are indexes of raw features), f denotes the embeddings of raw features (i.e., the output of embedding layer), and Co denotes the set of o-th order cross-features on which there is a sparsity constraint of No, ensuring both the quality of cross-features and the model efficiency. The feature interaction is perturbation invariant, i.e., f1 f2 = f2 f1. The element-wise product is the most used operation for feature interaction [18, 31, 5]. Let F = {f1, f2, · · · , fM}. Then, in short, the general objective of DSN can be formulated as follows,
minF L ( φ(F, {Co});D ) , (2)
where L denotes the loss function and φ is the prediction function, of which a linear predictor or MLP can be used. The objective of the AutoDSN problem should be finding good sets of feature-interaction
cells, {Co}, with o from 1 to O, formulated as follows.
{Co}∗ = min {Co} L ( φ(F∗, {Co});Dval ) , s.t.
{ F∗ = argminF L ( φ(F, {Co});Dtra ) ,
|Co|≤ No, ∀o = 1, 2, ..., O, (3)
where Dtra and Dval denote the training and validation datasets, respectively. For the AutoDSN problem, it is a challenging problem due to multiple aspects. First, the search space for C is relatively large, which burdens the search cost. Second, the desired sets are discrete, and there are also sparse constraints, making it an NP-hard problem. Third, the desired cross-features are closely co-related with each other, from lower orders to higher orders, which should be carefully considered.

heading: 3.1.1 Revisiting Existing Approaches

text:
For the search problem in (3), an intuitive NAS-based solution is to use a parameterized vector to represent all cross-features’ importance. Such solution was recently proposed by AutoFIS [20], assigning different weights to cross-features and using the continuous relaxation, similar as DARTS [21]. For o-order cross-feature, a o-th order tensor A can be introduced, where Ac1,c2,··· ,co represents the weight of a cross-feature involved with raw features {fc1 , fc2 , · · · , fco}. Then, a feature-interaction layer in (1) is expressed as ∑
c∈Cfull Ac1c2···co(fc1 fc2 · · · fco), (4)
where Cfull denotes the set of all possible cross-features. Then the NAS problem with a continuous space can be re-formulated as follows,
A∗ = min A L ( ψ(F∗,A);Dval ) , s.t. F∗ = argmin F L ( ψ(F,A);Dtra ) , (5)
where ψ is the prediction function, which first reserves No cross-features with the top-No weights to meet the sparsity constraint and then feed reserved features to output layer. That is, it is similar with φ in Definition 1. Such a continuous space can overcome the issue of discrete search space and the learned architecture parameters provide explicit meaning, i.e. the importance of cross-features.
It is worth mentioning that AutoInt [31] learned the importance of cross-features implicitly by selfattention layers, which thus is stilled limited. Therefore we try to explicitly search for cross-features at the feature-interaction layer. Faced with the challenge of large search space, we approach the problem with the idea of encoding the space into low-rank vectors for reducing the cost.

heading: 3.2 Proposed Search Method

text:
First let us recap some recent works [20, 23, 19] adopting NAS to help explicitly search powerful cross-features as the feature-interaction layer. Although the searched feature-interaction layer is more lightweight, these methods suffer from high search costs. Furthermore, it ignores the different orders of cross-features, from 1 to O, and consider them as the same, which results in a very large-size weight tensor A, with length ∑O o=1 C o M , where M denotes the number of raw features. Such a NAS problem with a continuous space still has two critical issues.
• First, representation of the search space takes too many architecture parameters, which slows down the optimization procedure. With M fields of raw features and the desired feature order O, the size of architecture parameter for the search space of cross-feature is close to COM , which approximately equals to MO considering M O. Obviously, MO is a too large number in the gradient-based NAS. For example, the widely-used Criteo benchmark dataset has M = 39 of fields of raw features. If we choose the highest order as 4, then C439 = 82,251. Such a parameter number is much larger compared with existing works of gradient-based NAS [21, 36], of which the number would always be fewer than one hundred. Therefore, the challenge of large search space is still unresolved. • Second, such design still ignores the order-priority property in deep sparse prediction tasks. Specifically, the property consists of two characteristics revealed by human practice: higher-order features’ quality can be relevant to their de-generated low-order ones, and lower-order cross-features are likely to be more vital compared with higher-order ones.
To address the above-mentioned two limitations in Section 3.1, we present our solution PROFIT here with 1) a distilled low-rank approximated search space to address the space issue and 2) a progressive search algorithm to take order-priority property into consideration.

heading: 3.2.1 Distilling the Search Space

text:
As above, the architecture parameter, i.e., tensor A, can be too large and ignore priorities on feature interactions. This can lead to high search costs and not being capable of finding desired architectures. Here, we study the property of A and attempt to approximate it with much fewer parameters. We conducted an experiment on the Criteo dataset and set the max order as two due to the computation cost. This dataset has 39 fields, and thus weightsA for all second-order interactions can be represented by a matrix of size 39×39. Singular values fromA and a randomly generated matrix are compared in Figure 2 (a). We can observe that the parameter matrix has several dominant singular values, exhibits a low-rank property. In other words, skew singular values in Figure 2 (a) shows there is redundancy in the architecture matrix for the second-order cross-feature. Besides, note that feature interaction is perturbation invariant, i.e., f1 f2 = f2 f1, which means the tensor should be symmetric. Thus, we propose to distilled tensor A as
A ≈ g({βo}) ≡ ∑R
r=1 βr ◦ · · · ◦ βr︸ ︷︷ ︸ βr repeats O times. , (6)
where each βr ∈ R1×M , and R M is the positive integer. Equation (6) can also be seen as a symmetric CP decomposition [17] for the tensor A. In other words, βr and βo refer to the approximated tensor and decomposed tensor, respectively. Subsequently, the search problem in (3) is transformed as
{βo}∗ = min β L ( ψ(F∗, g({βo}));Dval ) , s.t. F∗ = argmin F L ( ψ(F, g({βo}));Dtra ) . (7)
Comparing with the original A in (5), of which size is about MO and grows exponentially with O, {βr} is of size M × O and can be much smaller. Thus, the number of architecture parameters is reduced significantly.
Remark 1 Here we discuss a research topic related to our solution, search space compression [6, 38, 12]. Different from these works such as [6] which is based on compressed sensing, our method is based on low-rank approximation. Therefore, although both these works and our work conduct the space distillation, the fundamental assumptions (vector space v.s. matrix space) are different.

heading: 3.2.2 Progressive Search Algorithm

text:
As mentioned above, different orders of cross-features are highly correlated. To confirm that higherorder cross-features that are constructed by important lower-order cross-features tend to be important, since the ML1M dataset has fewer fields of features and then a full space of high-order features is easier to optimize, we conduct the experiment on the ML1M dataset. We still use A in (4) and set the maximum order to two and three. The relative ranking is computed according to the importance weights of all same-order cross-features. For example, a second-order cross-feature with the relative ranking as one means it is the most important second-order cross-feature. For a given 2nd-order cross-feature, fc1 fc2 , we can also compute the average relative ranking of third-order cross-features that consists of c1 and c2. We study whether an important second-order cross-feature can generate
important third-order cross-features. The relative ranking is shown at Figure 2 (b), and experimental details are in Appendix A.5. We can easily observe that a good 2nd-order cross-feature is more likely to generate a good 3rd-order cross-feature. Note that these points are not monotonically increasing, with some special points such as the 5-th one. It shows the necessity of utilizing auxiliary parameters to model the high-order cross-features.
The above motivates us to adopt a progressive strategy to search for the architecture parameters from lower orders to higher orders, to tackle the problem of modeling the order-priority property. The complete steps are in Algorithm 1. Specifically, when searching for o-th order cross-feature, at step 2, we propose to fix the architecture parameters with order lower than o, i.e., {β1, . . . ,βo−1}. Then, following existing works [21, 20, 40], at step 4 and 5, we also use gradient descent to update both architecture and embedding parameters.
Algorithm 1 Progressive gradient descent. 1: for o = 1, . . . , O do 2: fix β1, β2, ..., βo−1; 3: for epoch = 1, . . . , Eo do 4: Update architecture parameters βo using gradients∇βoL ( ψ(F, g({βo}));Dval ) ;
5: Update embedding parameters F using gradients∇FL ( ψ(F, g({βo}));Dtra ) ; 6: end for 7: end for

heading: 3.2.3 Retraining the Searched Lightweight Model

text: Finally, as the existing NAS works [21, 42] reveal, it is better to re-train the model with the obtained embedding parameters. In our task, to ensure both the prediction accuracy and model efficiency, we only reserve the cross-features with top-No importance weights to meet the sparsity constraints. The reason is in two folds. First, some features do not benefit the task, and removing them can benefit the training procedure to achieve better prediction accuracy. Second, more importantly, only reserving No cross-features can largely reduce the model size and inference times, ensuring the real-world deployment on various hardware platforms.

heading: 4 Experiments

heading: 4.1 Settings

text:
Datasets. To validate the effectiveness of our proposed PROFIT, we conduct experiments on three benchmark datasets (Criteo, Avazu and ML1M) widely used in existing works of deep sparse networks [5, 31, 18] to evaluate the performance, of which the details are provided in the Appendix.
Baselines. We compare our PROFIT with state-of-the-art methods for sparse prediction. We choose methods that are widely used and open-source:
• First-order method: logistic regression [14] (denoted as “LR”); • Second-order methods: we compared with the most widely-used second-order method factorization
machine [27] (denoted as “FM”) and the state-of-the-art second-order method attentive factorization machine [35]1 (denoted as “AFM”). NFM [13] is a de-generated version of AFM, and thus we do not compare with it. • Higher-order methods with deep feature-interaction layers: deep factorization machine DeepFM [9] extreme deep factorization machine [18]2 (denoted as “DeepFM”) , extreme deep factorization machine [18]3 (denoted as “xDeepFM”), product neural network [25]4 (denoted as “PNN”), automatic feature Interaction learning [31] (denoted as “AutoInt”) 5 and adaptive factorization network [5]6 (denoted as “AFN”).
1https://github.com/hexiangnan/attentional_factorization_machine 2https://github.com/ChenglongChen/tensorflow-DeepFM 3https://github.com/Leavingseason/xDeepFM 4https://github.com/shenweichen/DeepCTR 5https://github.com/DeepGraphLearning/RecommenderSystems. 6https://github.com/WeiyuCheng/AFN-AAAI-20
These human-designed baselines models are also selected as the representative DSN models for comparison in the AutoFIS paper. Besides, there is another recent work, i.e., AutoGroup [19], which is similar to AutoFIS. Considering it has close prediction performance and search cost compared with AutoFIS, we only compare AutoFIS in our experiments. There are some recent works [30, 16] approaching the problem via automatically fusing human-designed models, which is still restrained by human-designed feature-interaction layers.
As mentioned above, there are some pioneer works that propose NAS methods for lightweight CNN models for computer vision tasks. Here we do not compare with them since it is hard to adapt them to the deep sparse networks.
Evaluation Metrics. We evalua te models from two perspectives, prediction performance and inference efficiency. For accuracy, we use two standard metrics, Logloss (denoted as “Loss”) [24] and AUC [7], following [5, 9, 18, 27, 25]. For efficiency, we also use two standard metrics, inference time and model size, which are also widely used in existing works of deep sparse networks [31, 15].
Hyper-parameter Settings. To ensure the reproducibility of experimental results, here we introduce the implementation setting, of which more details are in Appendix. We implement our methods using PyTorch. We apply Adam with a learning rate of 0.001 and a mini-batch size of 4096, a widely-used setting in existing works [5, 31]. We set the embedding sizes to 16 in all the models. We use the same neural network structure ({400, 400, 400}) for all methods that adopt MLP for a fair comparison, following [5, 31]. All the other hyper-parameters are tuned on the validation set. To make the results more reliable, for each instance, we run the repetitive experiments three times with different random seeds and report the average value.

heading: 4.2 Benchmark Comparison

text:
Prediction accuracy. In Table 1, we present the prediction performance of all these methods, from which we have the following observations. First, models with higher-order cross-features can achieve better prediction performance than lower-order ones, which verifies the rationality of exploring highorder feature interactions. Second, with the carefully designed search space and search algorithm, PROFIT can achieve the best prediction performance on three benchmark datasets (or very close to the best baseline). Third, the deep sparse networks achieve better performance compared with shallow models, which demonstrates the necessity of designing powerful deep sparse networks. Note that due to the limitations in search space and algorithm, which make the searching procedure very slow, AutoFIS does not achieve promising results compared with human-designed ones7.
Efficiency. We further compare the efficiency and present the results in Figure 3. We can observe that PROFIT is lightweight and fast-to-infer among all compared methods. In real-world applications, both the accuracy and the inference efficiency should be considered. Since our method not only achieves promising accuracy but also reduces the inference time, it has high application value in practice.
7AutoFIS achieves better performance compared with human-crafted baselines when setting the max order of cross-features as three, as shown by the original paper. However, during the experiments in our normal hardware platform, for the Criteo dataset and Avazu dataset, it cannot be well optimized due to too large search space and computation costs. Thus, we only report the results setting max order to two for these two datasets.

heading: 4.3 Understanding the Search Strategy

text:
Progressive search. In order to validate the effectiveness of the progressive manner in our search algorithm, we compared with the performance that utilizes a one-step manner. It can be regarded as the de-generated version of our PROFIT that replaces our progressive search with one step, and the details of its searching procedure are shown in Appendix A.4. We also compare the random search with PS (parameter sharing), of which architecture parameters are randomly assigned, and the model parameters are shared during the search procedures. The final performance of the searched architectures is shown in Table 2. The curve of validation performance during the searching procedure is shown in Figure 5.
From these results, we can observe that random search achieves the worst performance since it does not explore any domain properties, as we have analyzed in Section 3.1. Also, with the simple one-step search algorithm, the performance is poorer compared with the progressive search algorithm. This validates the rationality of our design of progressive search, which can well capture the order-priority property of cross-features in the deep sparse prediction task. In short, the effectiveness of our design of progressive search is well validated by the results.
Distilled search space. First, we conduct experiments to study the searched results if different search space is used. We compared our distilled space, i.e., Eqn (7), with a full space, i.e. Eqn (5). We choose the order of three. The curve of validation performance during the search procedure of two large-scale datasets, Avazu and Criteo, is shown in Figure 6, and the final results are shown in Table 3. We can observe that searching in the distilled space can easily reach a promising validation performance. However, when searching in the full space, it requires a higher computation cost, and as a result, it is hard to obtain good architecture parameters. As a result, the distilled search space
obtains better prediction performance. These results show that with a reduced budget for searching, using the distilled space is a much better choice compared with the original full search space.

heading: 5 Conclusion

text: In this work, we approached the problem of deep sparse prediction with neural architecture search. We propose a distilled search space which is a low-rank approximation of the full space, significantly reducing the search cost. We then propose a progressive differentiable search algorithm that can efficiently search for desired models in the space while being consistent with the order-priority property. Extensive experiments on three real-world benchmark datasets demonstrate that our proposed PROFIT can achieve both high accuracy and high efficiency. Further studies illustrate that the proposed search space and search algorithm of PROFIT is feasible. As for future work, we will explore more on how dataset characteristics affect the searching results.

heading: Acknowledgment

text: This work was supported by National Natural Science Foundation of China under U1936217, 61971267, 61972223, 61941117, 61861136003.

title: Progressive Feature Interaction Search for Deep Sparse Network


summaries: This paper studies the generalization and robustness of regularized fine-tuning. They present a PAC-Bayes generalization bound that depends on the distance traveled in each layer during fine-tuning and the noise stability of the pre-trained model. They empirically measure these quantities to analyze the behavior of fine-tuning. They present an algorithm that interpolates between two components: (i) regularization over the amount of distance traveled in each layer; (ii) iteratively correcting mislabeled data points that the model is highly confident and down-weighting less confident data points.

summaries: This paper includes a PAC-Bayesian analysis of distance-based regularisation during fine-tuning. Using this analysis and supplementary experiments, the authors propose a new means for tuning hyperparameters for distance-based regularisation with constraints and heuristics for dealing with label noise during fine-tuning. The experiments show that the method works slightly better than previous approaches designed for both fine-tuning and training with label noise.

summaries: The paper proposes a new regularization method for training and fine-tuning large neural networks. The paper proposes to constrain the maximum change in network parameters during fine-tuning. This constraint is different in each layer. Experiments demonstrate that the proposed method outperforms previous approaches.

summaries: The paper analyses the fine-tuning of neural networks. It provides a PAC-Bayes bound for the generalization of a fine-tuned network. The paper then provides three tricks to improve finetuning: layerwise regularization, label correction, and label removal. The resulting fine-tuning algorithm improves performance on various image classification tasks, including those with artificially-added label noise.

summaries: This paper studies the aspects of fine-tuning a model (pre-trained on a separate dataset) in order to learn a new task. It has the following contribution:  (i) Generalization properties of fine-tuning explained by the PAC-Bayes generalization bound that depends on two terms: (a) distance between each layer from the pre-trained model and (b) noise stability of the pre-trained model.  (ii) Inspired by the generalization analysis, the paper presents an algorithm to perform fine-tuning. This algorithm includes (a) regularization term that computes the distance between the current parameters and the pre-trained model, and (b) iteratively corrects mislabeled examples where model has high confidence and down-weighting less confident examples.  (iii) Proposed algorithm is evaluated on a suite of benchmark tasks including transfer learning and few shot classification tasks.

abstractText: A widely used algorithm for transfer learning is fine-tuning, where a pre-trained model is fine-tuned on a target task with a small amount of labeled data. When the capacity of the pre-trained model is much larger than the size of the target data set, fine-tuning is prone to overfitting and “memorizing” the training labels. Hence, an important question is to regularize fine-tuning and ensure its robustness to noise. To address this question, we begin by analyzing the generalization properties of fine-tuning. We present a PAC-Bayes generalization bound that depends on the distance traveled in each layer during fine-tuning and the noise stability of the fine-tuned model. We empirically measure these quantities. Based on the analysis, we propose regularized self-labeling—the interpolation between regularization and self-labeling methods, including (i) layer-wise regularization to constrain the distance traveled in each layer; (ii) self label-correction and label-reweighting to correct mislabeled data points (that the model is confident) and reweight less confident data points. We validate our approach on an extensive collection of image and text data sets using multiple pre-trained model architectures. Our approach improves baseline methods by 1.76% (on average) for seven image classification tasks and 0.75% for a few-shot classification task. When the target data set includes noisy labels, our approach outperforms baseline methods by 3.56% on average in two noisy settings.

authors:
Takuya Akiba
Shotaro Sano
Toshihiko Yanase
Takeru Ohta
Masanori Koyama

title: Optuna: A nextgeneration hyperparameter optimization framework

venue: In International Conference on Knowledge Discovery and Data Mining,

authors:
Sanjeev Arora
Simon Du
Wei Hu
Zhiyuan Li
Ruosong Wang

title: Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks

venue: In International Conference on Machine Learning

authors:
Peter Bartlett
Dylan J Foster
Matus Telgarsky

title: Spectrally-normalized margin bounds for neural networks

venue: arXiv preprint arXiv:1706.08498,

authors:
Peter L Bartlett
Nick Harvey
Christopher Liaw
Abbas Mehrabian

title: Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks

venue: The Journal of Machine Learning Research,

authors:
Shai Ben-David
John Blitzer
Koby Crammer
Alex Kulesza
Fernando Pereira
Jennifer Wortman Vaughan

title: A theory of learning from different domains

venue: Machine learning,

authors:
Xinyang Chen
Sinan Wang
Bo Fu
Mingsheng Long
Jianmin Wang

title: Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer

authors:
Guneet S Dhillon
Pratik Chaudhari
Avinash Ravichandran
Stefano Soatto

title: A baseline for few-shot image classification

venue: arXiv preprint arXiv:1909.02729,

authors:
Alexey Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
Mostafa Dehghani
Matthias Minderer
Georg Heigold
Sylvain Gelly

title: An image is worth 16x16 words: Transformers for image recognition at scale

venue: arXiv preprint arXiv:2010.11929,

authors:
Gintare Karolina Dziugaite
Daniel M Roy

title: Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data

venue: arXiv preprint arXiv:1703.11008,

authors:
Pierre Foret
Ariel Kleiner
Hossein Mobahi
Behnam Neyshabur

title: Sharpness-aware minimization for efficiently improving generalization

authors:
Henry Gouk
Timothy M Hospedales
Massimiliano Pontil

title: Distance-based regularisation of deep networks for fine-tuning

authors:
Gregory Griffin
Alex Holub
Pietro Perona

title: Caltech-256 object category dataset

authors:
Yunhui Guo
Honghui Shi
Abhishek Kumar
Kristen Grauman
Tajana Rosing
Rogerio Feris

title: Spottune: transfer learning through adaptive fine-tuning

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Conference on computer vision and pattern recognition,

authors:
Kaiming He
Ross Girshick
Piotr Dollár

title: Rethinking imagenet pre-training

venue: In International Conference on Computer Vision,

authors:
Minqing Hu
Bing Liu

title: Mining and summarizing customer reviews

venue: In ACM SIGKDD international conference on Knowledge discovery and data mining,

authors:
Lang Huang
Chao Zhang
Hongyang Zhang

title: Self-adaptive training: beyond empirical risk minimization

authors:
Yiding Jiang
Behnam Neyshabur
Hossein Mobahi
Dilip Krishnan
Samy Bengio

title: Fantastic generalization measures and where to find them

authors:
Aditya Khosla
Nityananda Jayadevaprakash
Bangpeng Yao
Li Fei-Fei

title: Novel dataset for fine-grained image categorization

venue: In Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Jonathan Krause
Michael Stark
Jia Deng
Li Fei-Fei

title: 3d object representations for fine-grained categorization

venue: In Workshop on 3D Representation and Recognition,

authors:
Hao Li
Pratik Chaudhari
Hao Yang
Michael Lam
Avinash Ravichandran
Rahul Bhotika
Stefano Soatto

title: Rethinking the hyperparameters for fine-tuning

venue: arXiv preprint arXiv:2002.11770,

authors:
Xin Li
Dan Roth

title: Learning question classifiers

venue: In International conference on Computational linguistics,

authors:
Xingjian Li
Haoyi Xiong
Hanchao Wang
Yuxuan Rao
Liping Liu
Jun Huan

title: Delta: Deep learning transfer using feature map with attention for convolutional networks

venue: In International Conference on Learning Representations,

authors:
Xuhong Li
Yves Grandvalet
Franck Davoine

title: Explicit inductive bias for transfer learning with convolutional networks

venue: arXiv preprint arXiv:1802.01483,

authors:
Yuanzhi Li
Tengyu Ma
Hongyang Zhang

title: Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations

venue: In Conference On Learning Theory,

authors:
Sheng Liu
Jonathan Niles-Weed
Narges Razavian
Carlos Fernandez-Granda

title: Early-learning regularization prevents memorization of noisy labels

venue: arXiv preprint arXiv:2007.00151,

authors:
Philip M Long
Hanie Sedghi

title: Generalization bounds for deep convolutional neural networks

venue: ICLR,

authors:
Xingjun Ma
Hanxun Huang
Yisen Wang
Simone Romano
Sarah Erfani
James Bailey

title: Normalized loss functions for deep learning with noisy labels

venue: In International Conference on Machine Learning,

authors:
Subhransu Maji
Esa Rahtu
Juho Kannala
Matthew Blaschko
Andrea Vedaldi

title: Fine-grained visual classification of aircraft

venue: arXiv preprint arXiv:1306.5151,

title: Pac-bayesian model averaging

venue: In Proceedings of the twelfth annual conference on Computational learning theory,

title: Some pac-bayesian theorems

venue: Machine Learning,

authors:
Rafael Müller
Simon Kornblith
Geoffrey Hinton

title: When does label smoothing help

venue: arXiv preprint arXiv:1906.02629,

authors:
Vaishnavh Nagarajan
Zico Kolter

title: Deterministic pac-bayesian generalization bounds for deep networks via generalizing noise-resilience

authors:
Nagarajan Natarajan
Inderjit S Dhillon
Pradeep Ravikumar
Ambuj Tewari

title: Learning with noisy labels

venue: In Conference on Neural Information Processing Systems,

authors:
Behnam Neyshabur
Srinadh Bhojanapalli
Nathan Srebro

title: A pac-bayesian approach to spectrallynormalized margin bounds for neural networks

authors:
Maria-Elena Nilsback
Andrew Zisserman

title: Automated flower classification over a large number of classes

venue: In Indian Conference on Computer Vision, Graphics & Image Processing,

authors:
Bo Pang
Lillian Lee

title: A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts

venue: In Annual meeting on Association for Computational Linguistics,

title: Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

venue: In Annual meeting on association for computational linguistics,

authors:
Pranav Rajpurkar
Jeremy Irvin
Kaylie Zhu
Brandon Yang
Hershel Mehta
Tony Duan
Daisy Ding
Aarti Bagul
Curtis Langlotz
Katie Shpanskaya

title: Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning

venue: arXiv preprint arXiv:1711.05225,

authors:
Alexander Ratner
Christopher De Sa
Sen Wu
Daniel Selsam
Christopher Ré

title: Data programming: Creating large training sets, quickly

venue: Advances in neural information processing systems,

authors:
Alexander Ratner
Stephen H Bach
Henry Ehrenberg
Jason Fries
Sen Wu
Christopher Ré

title: Snorkel: Rapid training data creation with weak supervision

venue: In International Conference on Very Large Data Bases. NIH Public Access,

authors:
Alexander Ratner
Braden Hancock
Jared Dunnmon
Frederic Sala
Shreyash Pandey
Christopher Ré

title: Training complex models with multi-task weak supervision

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Olga Russakovsky
Jia Deng
Hao Su
Jonathan Krause
Sanjeev Satheesh
Sean Ma
Zhiheng Huang
Andrej Karpathy
Aditya Khosla
Michael Bernstein

title: Imagenet large scale visual recognition challenge

venue: International journal of computer vision,

authors:
Khaled Saab
Sarah M Hooper
Nimit S Sohoni
Jupinder Parmar
Brian Pogatchnik
Sen Wu
Jared A Dunnmon
Hongyang R Zhang
Daniel Rubin
Christopher Ré

title: Observational supervision for medical image classification using gaze data

venue: In International Conference on Medical Image Computing and Computer-Assisted Intervention,

authors:
Hadi Salman
Andrew Ilyas
Logan Engstrom
Ashish Kapoor
Aleksander Madry

title: Do adversarially robust imagenet models transfer better

venue: arXiv preprint arXiv:2007.08489,

authors:
Ali Sharif Razavian
Hossein Azizpour
Josephine Sullivan
Stefan Carlsson

title: Cnn features off-the-shelf: an astounding baseline for recognition

venue: In Conference on computer vision and pattern recognition workshops,

authors:
Richard Socher
Alex Perelygin
Jean Wu
Jason Chuang
Christopher D Manning
Andrew Ng
Christopher Potts

title: Recursive deep models for semantic compositionality over a sentiment treebank

venue: In Conference on empirical methods in natural language processing,

authors:
Kai Sheng Tai
Peter Bailis
Gregory Valiant

title: Sinkhorn label allocation: Semi-supervised classification via annealed self-training

venue: arXiv preprint arXiv:2102.08622,

authors:
Chuanqi Tan
Fuchun Sun
Tao Kong
Wenchang Zhang
Chao Yang
Chunfang Liu

title: A survey on deep transfer learning

venue: In International conference on artificial neural networks,

authors:
Sunil Thulasidasan
Tanmoy Bhattacharya
Jeff Bilmes
Gopinath Chennupati
Jamal MohdYusof

title: Combating label noise in deep learning using abstention

authors:
Yonglong Tian
Yue Wang
Dilip Krishnan
Joshua B Tenenbaum
Phillip Isola

title: Rethinking few-shot image classification: a good embedding is all you need

venue: arXiv preprint arXiv:2003.11539,

title: An introduction to matrix concentration inequalities

venue: arXiv preprint arXiv:1501.01571,

authors:
Oriol Vinyals
Charles Blundell
Timothy Lillicrap
Koray Kavukcuoglu
Daan Wierstra

title: Matching networks for one shot learning

venue: arXiv preprint arXiv:1606.04080,

authors:
C. Wah
S. Branson
P. Welinder
P. Perona
S. Belongie

title: The Caltech-UCSD Birds-200-2011 Dataset

venue: Technical Report CNS-TR-2011-001, California Institute of Technology,

authors:
Xiaosong Wang
Yifan Peng
Le Lu
Zhiyong Lu
Mohammadhadi Bagheri
Ronald M Summers

title: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases

authors:
Yisen Wang
Xingjun Ma
Zaiyi Chen
Yuan Luo
Jinfeng Yi
James Bailey

title: Symmetric cross entropy for robust learning with noisy labels

authors:
Colin Wei
Tengyu Ma

title: Data-dependent sample complexity of deep neural networks via lipschitz augmentation

venue: arXiv preprint arXiv:1905.03684,

title: Improved sample complexities for deep networks and robust classification via an all-layer margin

venue: arXiv preprint arXiv:1910.04284,

authors:
Janyce Wiebe
Theresa Wilson
Claire Cardie

title: Annotating expressions of opinions and emotions in language

venue: Language resources and evaluation,

authors:
Sen Wu
Hongyang R Zhang
Christopher Ré

title: Understanding and improving information transfer in multi-task learning

authors:
Yi Xu
Lei Shang
Jinxing Ye
Qi Qian
Yu-Feng Li
Baigui Sun
Hao Li
Rong Jin

title: Dash: Semi-supervised learning with dynamic thresholding

authors:
Fan Yang
Hongyang R Zhang
Sen Wu
Weijie J Su
Christopher Ré

title: Analysis of information transfer from heterogeneous sources via precise high-dimensional asymptotics

authors:
Chiyuan Zhang
Samy Bengio
Moritz Hardt
Benjamin Recht
Oriol Vinyals

title: Understanding deep learning requires rethinking generalization

venue: arXiv preprint arXiv:1611.03530,

authors:
Zhilu Zhang
Mert R Sabuncu

title: Generalized cross entropy loss for training deep neural networks with noisy labels

venue: arXiv preprint arXiv:1805.07836,

authors:
Wenda Zhou
Victor Veitch
Morgane Austern
Ryan P Adams
Peter Orbanz

title: Non-vacuous generalization bounds at the imagenet scale: a pac-bayesian compression approach

venue: arXiv preprint arXiv:1804.05862,

authors:
Yang Zou
Zhiding Yu
Xiaofeng Liu
BVK Kumar
Jinsong Wang

title: Confidence regularized self-training

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

heading: 1 Introduction

text:
Learning from limited labeled data is a fundamental problem in many real-world applications (Ratner et al., 2016, 2017). A common approach to address this problem is fine-tuning a large model that has been pre-trained on publicly available labeled data (He et al., 2019). Since fine-tuning is typically applied to a target task with limited labels, this algorithm is prone to overfitting or “memorization” issues (Tan et al., 2018). These issues worsen when the target task contains noisy labels (Zhang et al., 2016). In this paper, we analyze regularization methods for fine-tuning from both theoretical and empirical perspectives. Based on the analysis, we propose a regularized self-labeling approach that improves the generalization and robustness properties of fine-tuning.
Previous works (Li et al., 2018a,b) have proposed regularization methods to constrain the distance between a fine-tuned model and the pre-trained model in the Euclidean norm. Li et al. (2020) provides extensive study to show that the performance of fine-tuning and the benefit of adding regularization depend on the hyperparameter choices. Salman et al. (2020) empirically find that performing adversarial training during the pre-training phase helps learn pre-trained models that transfer better to downstream tasks. The work of Gouk et al. (2021) generalizes the above ideas to various norm choices and finds that projected gradient descent methods perform well for implementing distance-based regularization. Additionally, they derive generalization bounds for fine-tuning using Rademacher complexity. These works focus on settings where there is no label noise in the target data set. When
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
label noise is present, for example, due to applying weak supervision techniques (Ratner et al., 2016), an important question is to design methods that are robust to such noise. The problem of learning from noisy labels has a rich history of study in supervised learning (Natarajan et al., 2013). In contrast, little is known in the transfer learning setting. These considerations motivate us to analyze the generalization and robustness properties of fine-tuning.
In Section 4.1, we begin by conducting a PAC-Bayesian analysis of regularized fine-tuning. This is inspired by recent works that have found PAC-Bayesian analysis correlates with empirical performance better than Rademacher complexity (Jiang et al., 2020). We identify two critical measures for analyzing the generalization performance of fine-tuning. The first measure is the `2 norm of the distance between the pre-trained model (initialization) and the fine-tuned model. The second measure is the perturbed loss of the fine-tuned model, i.e. its loss after the model weights get perturbed by random noise. First, we observe that the fine-tuned weights remain closed to the pre-trained model. Moreover, the top layers travel much further away from the pre-trained model than the bottom layers. Second, we find that fine-tuning from a pre-trained model implies better noise stability than training from a randomly initialized model. In Section 4.2, we evaluate regularized fine-tuning for target tasks with noisy labels. We find that fine-tuning is prone to “memorizing the noisy labels”, and regularization helps alleviate such memorization behavior. Moreover, we observe that the neural network has not yet overfitted to the noisy labels during the early phase of fine-tuning. Thus, its prediction could be used to relabel the noisy labels.
We propose an algorithm that incorporates layerwise regularization and self-labeling for improved regularization and robustness based on our results. Figure 1 illustrates the two components. First, we encode layer-wise distance constraints to regularize the model weights at different levels. Compared to (vanilla) fine-tuning, our algorithm reduces the gap between the training and test accuracy, thus alleviating overfitting. Second, we add a self-labeling mechanism that corrects and reweights “noisy labels” based on the neural network’s predictions. Figure 1 shows that our algorithm effectively hinders the model from learning the incorrect labels by relabeling them to correct ones.
In Section 5, we evaluate our proposed algorithm for both transfer learning and few-shot classification tasks with image and text data sets. First, using ResNet-101 (He et al., 2016) as the pre-trained model, our algorithm outperforms previous fine-tuning methods on seven image classification tasks by 1.76% on average and 3.56% when their labels are noisy. Second, we find qualitatively similar results for applying our approach to medical image classification tasks (ChestX-ray14 (Wang et al., 2017; Rajpurkar et al., 2017)) and vision transformers (Dosovitskiy et al., 2020). Finally, we extend our approach to few-shot learning and sentence classification. For these related but different tasks and data modalities, we find an improvement of 0.75% and 0.46% over previous methods, respectively.
In summary, our contributions are threefold. First, we provide a PAC-Bayesian analysis of regularized fine-tuning. Our result implies empirical measures that explain the generalization performance of regularized fine-tuning. Second, we present a regularized self-labeling approach to enhance the generalization and robustness properties of fine-tuning. Third, we validate our approach on an extensive collection of classification tasks and pre-trained model architectures.

heading: 2 Related work

text:
Fine-tuning is widely used in multi-task and transfer learning, meta-learning, and few-shot learning. Previous works (Li et al., 2018b,a) find that injecting `2 regularization helps improve the performance of fine-tuning. Li et al. (2018b) propose a `2 distance regularization method that penalizes the `2 distance between the fine-tuned weights and the pre-trained weights. Li et al. (2018a) penalize the distance between the feature maps as opposed to the layer weights. Chen et al. (2019) design a regularization method that suppresses the spectral components (of the feature maps) with small
singular values to avoid negative transfer. Gouk et al. (2021) instead encode distance constraints in constrained minimization and use projected gradient descent to ensure the weights are close to the pre-trained model. Salman et al. (2020) show that fine-tuning from adversarially robust pre-trained models outperforms fine-tuning from (standard) pre-trained models.
The robustness of learning algorithms in the presence of label noise has been extensively studied in supervised learning (Natarajan et al., 2013). Three broad ideas for designing robust algorithms include defining novel losses, identifying noisy labels, and regularization methods. Zhang and Sabuncu (2018) design the robust Generalized Cross Entropy (GCE) loss which is a mixture of Cross Entropy (CE) and mean absolute error. The Symmetric Cross Entropy (SCE) (Wang et al., 2019) loss combines reverse cross-entropy with the CE loss. Ma et al. (2020) proposes APL, which normalizes the loss to be robust to noisy labels and combines active and passive loss functions. Thulasidasan et al. (2019) propose DAC, which identifies and suppresses the signals of noisy samples by abstention-based training. Liu et al. (2020) introduce ELR, an early learning regularization approach to mitigate label “memorization”. Huang et al. (2020) propose a self-adaptive training method that corrects noisy labels and reweights training data to suppress erroneous signals. These works primarily concern the supervised learning setting. To the best of our knowledge, fine-tuning algorithms under label noise are relatively under-explored in transfer learning. Our approach draws inspiration from the semi-supervised learning literature, which has evaluated pseudo-labeling and self-training approaches given a limited amount of labeled data, and a large amount of unlabeled data (Zou et al., 2019; Xu et al., 2021; Tai et al., 2021). A very recent work considered a sharpness-aware approach and evaluated its performance for fine-tuning from noisy labels (Foret et al., 2021). It would be interesting to see if combining their approach with our ideas could lead to better results.
From a theoretical perspective, the seminal work of Ben-David et al. (2010) considers a setting where labeled data from many source tasks and a target task is available. They show that minimizing a weighted combination of the source and target empirical risks leads to the best result. In the supervised setting, Arora et al. (2019) provide data-dependent generalization bounds for neural networks. Nagarajan and Kolter (2018) and Wei and Ma (2019a,b) provide improved generalization bounds that depend only polynomially on the depth of the neural network characterized by a margin condition. Recent work has found that the PAC-Bayes theory provides generalization measures that correlate with empirical generalization performance better than other alternatives (Jiang et al., 2020). We defer a more extensive review of PAC-Bayesian generalization theory to Section A.

heading: 3 Preliminaries

text:
Problem setup. We begin by formally introducing the setup. Suppose we would like to solve a target task. We have a training data set of size n(t). Let (x(t)1 , y (t) 1 ), . . . , (x (t) n(t) , y (t) n(t)
) be the feature vectors and the labels of the training data set. We assume that every x(t)
i lies in a d-dimensional space
denoted by X ⇢ Rd. Following standard terminologies in statistical learning, we assume all data are drawn from some unknown distribution supported on X ⇥ Y , where Y ✓ R is the label space. Denote the underlying data distribution as P(t).
For our result in Section 4.1, we consider feedforward neural networks, and our results can also be extended to convolutional neural networks (e.g., using ideas from Long and Sedghi (2020)). Consider an L layer neural network. For each layer i from 1 to L, let i be the activation function and let Wi be the weight matrix at layer i. Given an input z to the i-th layer, the output is then denoted as i(z) = i(Wiz). Thus, the final output of the network is given by
fW (x) = L L 1 · · · 1(x), for any input x 2 X , where we use W = [W1, . . . ,WL] to include all the parameters of the network for ease of notation. The prediction error of fW is measured by a loss function `(·) that is both convex and 1-Lipschitz.
L (t)(fW ) = E (x,y)⇠P(t) [`(fW (x), y)] . (1)
Suppose we have access to a pre-trained source model Ŵ (s). Using Ŵ (s) as an initialization, we then fine-tune the layers Ŵ (s)1 , . . . , Ŵ (s) L to predict the target labels.
Applications. We consider seven image classification data sets described in Table 1. We use ResNets pre-trained on ImageNet as the initialization Ŵ (s) (Russakovsky et al., 2015; He et al., 2016). We
perform fine-tuning using the pre-trained network on the above data sets. See Section 5.1 for further description of the training procedure.
Modeling label noise. In many settings, fine-tuning is applied to a target task whose labels may contain noise; for example, if the target labels are created using weak supervision techniques (Ratner et al., 2019; Saab et al., 2021). To capture such settings, we denote a noisy data set as (x(t)1 , ỹ (t) 1 ), . . . , (x (t) n(t) , ỹ (t) n(t) ), where ỹ(t) i is a noisy version of y(t) i
. We consider two types of label noise: independent noise and correlated noise. We say that the label noise is independent if it is independent of the input feature vector
Pr(ỹ(t) i = k|y(t) i = j, x(t) i ) = Pr(ỹ(t) i = k|y(t) i = j) = ⌘j,k
for some fixed ⌘j,k between 0 and 1. On the other hand, we say that the label noise is correlated if it depends on the input feature vector (i.e. the above equation does not hold).

heading: 4 Our proposed approaches

text: Given the problem setup described above, next, we study the generalization and robustness properties of regularized fine-tuning methods. First, we present a PAC-Bayes generalization bound for regularized fine-tuning. This result motivates us to evaluate two empirical measures: the fine-tuned distance in each layer and the perturbed loss of the fine-tuned model. Second, we consider fine-tuning from noisy labels. We show that layer-wise regularization prevents the model from memorizing the noisy labels. We then suggest injecting predictions of the model during training, similar to self-training and pseudo-labeling. Finally, we incorporate both components into our regularized self-labeling approach, blending the strengths of both to improve the generalization performance and the robustness of fine-tuning.

heading: 4.1 Fine-tuning and regularization

text:
We consider the following regularized fine-tuning problem, which constrains the network from traveling too far from the pre-trained initialization Ŵ (s) (Li et al., 2018a,b; Gouk et al., 2021).
Ŵ argmin L̂(t)(fW ) (2)
s.t. kWi Ŵ (s) i kF  Di, 8 i = 1, . . . , L. (3)
Above, Di is a hyperparameter that constrains how far the i-th layer Wi can travel from the pre-trained initialization Ŵ (s)
i . Previous works have observed
that stronger regularization reduces the generalization gap during fine-tuning (cf. Figure 2). Next, we analyze the generalization error of Ŵ , that is, L (t)(f
Ŵ ) L̂(t)(f Ŵ ), where L(t)(f Ŵ ) is the test
loss of f Ŵ according to equation (1) and L̂(t)(f Ŵ ) is the empirical loss of f Ŵ on the training data set.
PAC-Bayesian analysis. We begin by analyzing the generalization error of f
Ŵ using PAC-Bayesian
tools (McAllester, 1999a,b). This departs from the previous work of Gouk et al. (2021), which bases on their analysis using Rademacher complexity. This change of perspective is inspired by several recent works that have found PAC-Bayesian bounds to better correlate with empirical performance than Rademacher complexity bounds (Jiang et al., 2020). We refer the interested reader to Section A for further references from this line of work. After presenting our result, we provide empirical measures of our result and discuss the practical implication.
Theorem 4.1 (PAC-Bayes generalization bound for fine-tuning). Suppose for every i = 1, . . . , L, kŴ
(s) i k2  Bi, for a fixed Bi > 1. Suppose the feature vectors in the domain X are all bounded: kxk2  C1 for every x 2 X , for some C1 1. Finally, suppose the loss function `(·) is 1-Lipschitz and bounded from above by a fixed constant C2. Under these conditions, let fŴ be the minimizer of regularized fine-tuning, solved from problem (3). Let " > 0 be an arbitrary small value. Let H be the maximum over the width over all the L layers and the input dimension d. Then, with probability at least 1 2 for some small > 0, the expected loss L(t)(f Ŵ ) is upper bounded by
L (t)(f Ŵ )  L̂(t)(f Ŵ ) + "+ C2
s 36 "2 ·C21H log(4LHC2) ⇣ PL i=1 QL j=1(Bj+Dj) Bi+Di ⌘2⇣PL i=1 D 2 i ⌘ +3 ln n (t) +8
n(t) . (4)
Relation to prior works. Compared to the result of Gouk et al. (2021), we instead proceed by factoring the generalization error into a noise error (i.e. the " term in equation (4)) and a KL-divergence between Ŵ (s) and Ŵ (i.e. the final term in equation (4)). The proof of Theorem 4.1 is based on Neyshabur et al. (2018) and is presented in Appendix A. The difference between Theorem 4.1 and the result of Neyshabur et al. (2018) is that our result is stated for the (e.g. cross-entropy) loss function `(·) whereas Neyshabur et al. (2018) states the result using the soft margin loss.
Our result suggests that the fine-tuned distances {Di}Li=1 and the perturbed loss (more precisely EU ⇥ `(f Ŵ+U (x), y) ⇤
where every entry of U is drawn from N (0, 2)) are two important measures for fine-tuning. Next, we empirically evaluate these two measures on real-world data sets.
Fine-tuned distances {Di}Li=1. First, we present our empirical finding for the fine-tuned distances in each layer. We fine-tune a ResNet-18 model (pre-trained on ImageNet) on seven data sets and calculate the Frobenius distance between Ŵ (s)
i and Ŵi for every layer i.
Figure 3a shows the ratio of the fine-tuned distances to the pre-trained weights in each layer. First, we observe that the fine-tuned distances are relatively small compared to the pre-trained network. This means that fine-tuning stays within a small local region near the pre-trained model. Second, we find that Di varies across layers. Di is smaller at lower layers and is larger at higher layers. This observation aligns with the folklore intuition that different layers in Convolutional neural networks play a different role (Guo et al., 2019). Bottom layers extract higher-level representations of features
and stay close to the pre-trained model. Top layers extract class-specific features and vary among different tasks. Thus, top layers travel further away from the pre-trained model.
Based on the above empirical finding, we propose layer-wise constraints as regularization. In particular, the constraints for the bottom layers should be small, and the constraints for the top layers should be large. In practice, we find that an exponentially increasing scheme involving a base distance parameter D and a scale factor works well. This is summarized in the following regularized fine-tuning problem:
Ŵ argmin L̂(t)(fW ) (5)
s.t. kWi Ŵ (s) i kF  D · i 1 , 8 i = 1, . . . , L. (6)
where the scale factor > 1 according to Figure 3a. The previous work of Gouk et al. (2021) uses the same distance parameter in equation (6), i.e., Di = D, 8i = 1, . . . , L. In our experiments, we have observed that such constant regularization constraints are only active at the top layers. With layer-wise regularization, we instead extend the constraints to the bottom layers as well. In Table 8 (cf. Section B.3), we compare the value of P L
i=1 D 2 i between constant regularization and layer-wise regularization. We find that layer-wise regularization leads to a smaller value of P L
i=1 D 2 i , thus
indicating a tighter generalization bound according to Theorem 4.1. Perturbed loss EU h `(fŴ+U(x), y) i . Second, we compare the perturbed loss of models finetuned from random, pre-trained, and adversarially robust pre-trained model initializations. The perturbed loss is measured by the average loss on the training set, after perturbing each entry of the layer weights by a Gaussian random variable N(0, 2).
In Table 3b, we report the perturbed losses for both the CUB-200-2011 and the Indoor data set. The results show that models fine-tuned from pre-trained initializations are more stable than models fine-tuned from random initializations.
Explaining why adversarial robustness helps fine-tuning. Recent work (Salman et al., 2020) found that performing adversarial training in the pre-training phase leads to models that transfer better to downstream tasks. More precisely, the adversarial training objective is defined as
L̃ (s) adv(✓) = min
W
1
n(s)
n (s)X
i=1
max k k2"
`(fW (x (s) i + ), y(s) i ). (7)
Let Ŵ (s)adv be a fine-tuned neural network using a pre-trained model from adversarial training. We measure the perturbed loss of Ŵ (s)adv using the pre-trained model provided by Salman et al. (2020) on the CUB-200-2011 and Indoor data set. Table 3b shows that f
Ŵ (s) adv often incurs lower perturbed losses than models fine-tuned from random and pre-trained initializations. In Section B.3, we additionally observe that the layers of Ŵ (s)adv generally have lower Frobenious norms compared to Ŵ
(s). Thus, the improved noise stability and the smaller layer-wise norms together imply a tighter generalization bound for adversarial training. We leave a thorough theoretical analysis of adversarial training in the context of fine-tuning to future works.

heading: 4.2 Fine-tuning and robustness

text:
Next, we extend our approach to fine-tuning from noisy labels. We compare fine-tuning with and without regularization on a target task with independent label noise to motivate our approach. In Figure 4, we plot the training and test accuracy curves on the Indoor dataset with two noise rates. We observe that the test accuracy increases at first, meaning the network learns from the correct labels during the first few epochs. As the learning process progresses, the test accuracy of fine-tuning decreases, implying the network is “overfitting” to noisy labels. Furthermore, the gap between the training and test accuracy curves is more than 20%. Similar observations have been presented in prior works (Zhang et al., 2016; Li et al., 2018c; Liu et al., 2020), where over-parametrized neural networks can memorize the entire training set. One way to mitigate memorization is to explicitly reduce model capacity (Wu et al., 2020; Yang et al., 2021). Another way is via explicit regularization since the distance constraint significantly reduces the training-test accuracy gaps and improves test accuracy by a significant margin. Based on our empirical observation, the neural network has some discriminating power during the early fine-tuning phase. Thus, we can leverage the model predictions to relabel incorrect labels and reweight data points with incorrect labels. We describe the two components next.
Self label-correction. We propose a label correction step to augment the number of correct labels. We leverage the discriminating power of a network during the early phase and correct data points for which the model has high confidence. Concretely, let pi be the prediction of the i-th data point. Given a confidence threshold pt at epoch t, if max(pi) pt and argmax(pi) 6= ỹ (t) i
, we identify data point i as noisy and relabel it to
ỹ (t) i = argmax(pi).
Figure 5a illustrates the precision of this step. We define precision as the number of correctly relabeled data points divided by the total number of data points whose labels are relabeled. We observe that precision is high at the beginning (around 60%) and gradually decreases in later epochs. Moreover, we find precision increases with regularization. This suggests an intricate interaction between regularization and self-labeling during fine-tuning. We elaborate on the role of regularization in label correction by looking at the number of relabeled data points in Section B.3.
Self label-reweighting. We incorporate a soft data removal step to prevent the model from overfitting to noisy labels. We identify data points as noisy if their loss values are large and thus down-weight them during training. The model will prioritize data points with correct labels in its gradient by assigning smaller weights to large-loss data points. More precisely, given a data point (x(t)
i , ỹ (t) i
), we reweight it by !i = exp( `(fW (x (t) i ), ỹ(t) i )/⌧) where ⌧ is a temperature parameter. In the implementation, we normalize the weights of every data point in every mini-batch B:
L̂B(fW ) = 1P
i2B !i
X i2B !i · `(fW (x (t) i ), ỹ(t) i ).
Algorithm 1 Regularized self-labeling (REGSL) Input: Input nosiy data (x(t)1 , ỹ (t) 1 ), . . . , (x (t) n(t) , ỹ(t) n(t) ), network model fW , constraint distance D, distance scale factor , re-weight start step Er , re-weight temperature ⌧ , label correction start step Ec, label correction threhold pt, and fine-tuning steps T 1: Initialize model parameters with pre-trained weights W = Ŵ s and the start step t = 0 2: while t < T do 3: Fetch a mini-batch of data {(x(t)i , ỹ (t) i )} m i=1
4: for each data point i in the batch do 5: Calculate the predicted probability pi = softmax(fW (x(t)i )) 6: if t > Ec and max(pi) > pt and argmax(pi) 6= ỹ(t)i then 7: Correct the label ỹ(t)i = argmax(pi) 8: end if 9: Calculate loss `i = `(fW (x(t)i ), ỹ (t) i )
10: if t > Er then Calculate the weight !i = exp( `i/⌧) else Set the weight !i = 1 11: end for 12: Update fW by SGD on L = 1P
i !i
Pm i !i`i and t = t+ 1
13: Project the weights Wi of each layer inside the layer-wise constraint region as in Equation 6 14: end while
Figure 5b compares the average weight of data points with the correct label to data points with an incorrect label. We find that with our reweighting scheme, the average weight of incorrectly labeled data points is much lower than the average weight of correctly labeled data points. Thus, the reweighting scheme expands the gradient of correctly labeled data points, thus reducing the fraction of noisy data points in the training set.
The pseudo-code for our final approach, which combines both layer-wise regularization and selflabeling, is presented in Algorithm 1.

heading: 5 Experiments

text: We evaluate our proposed algorithm in a wide range of tasks and pre-trained networks. First, we show that our algorithm outperforms the baselines by 1.75% on average over seven transfer learning tasks and can generalize to a more distant task of medical images. Second, our algorithm improves the robustness of fine-tuning in the presence of noisy labels compared to previous methods. Our algorithm improves over numerous baselines by 3.56% on average under both independent and correlated label noise. Moreover, our approach can achieve a similar performance boost for finetuning vision transformer models on noisy labeled data. Finally, we show our algorithms improve over fine-tuning baselines by 0.75% on average for the related task of few-shot classification. Our extensive evaluation confirms that our approach applies to a broad range of settings, validating our algorithmic insights. Due to space constraints, we defer the ablation studies to Section B.3. Our code is available at https://github.com/NEU-StatsML-Research/Regularized-Self-Labeling.

heading: 5.1 Experimental setup

text:
Data sets. First, we evaluate fine-tuning on seven image classification data sets and one medical image data set. The statistics of the seven image data sets are described in Table 1. For the medical imaging task, we consider the ChestX-ray14 data set contains 112120 frontal-view chest X-ray images labeled with 14 different diseases (Wang et al., 2017; Rajpurkar et al., 2017). Second, we evaluate Algorithm 1 under two different kinds of label noise to test the robustness of fine-tuning. We selected the MIT-Indoor data set (Sharif Razavian et al., 2014) as the benchmark data set and randomly flipped the labels of the training samples. The results for the other data sets are similar, which can be found in Section B.2. We consider both independent random noise and correlated noise in our experiments. We generate the independent random noise by flipping the labels uniformly with a given noise rate. We simulate the correlated noise by using the predictions of an auxiliary network as noisy labels. Section B.1 describes the label flipping process. For few-shot image classification, we conduct experiments on the miniImageNet benchmark (Vinyals et al., 2016) following the setup of Tian et al. (2020).
Architectures. We use the ResNet-101 (He et al., 2016) network for transfer learning tasks and ResNet-18 (He et al., 2016) network for ChestX-ray data set. We use ResNet-18 network for the labelnoise experiments and extend the our algorithm to the vision transformer (ViT) model (Dosovitskiy et al., 2020). The ResNet models are pre-trained on ImageNet (Russakovsky et al., 2015) and the ViT model is pre-trained on ImageNet-21k data set (Dosovitskiy et al., 2020). For the few-shot learning tasks, we use ResNet-12 pre-trained on the meta-training data set as in the previous work (Tian et al., 2020). We set four different values of Di for the four blocks of the ResNet models in our algorithm. We describe the fine-tuning procedure and the hyperparameters in Section B.1.
Baselines. For the transfer learning tasks, we use the Frobenius norm for distance regularization. We present ablation studies to compare the Frobenius norm and other norms in Section B.3. we compare our algorithm with fine-tuning with early stop (Fine-tuning), fine-tuning with weight decay (`2- Norm), label smoothing (LS) method formulated in Müller et al. (2019), `2-SP (Li et al., 2018b), and ` 2-PGM (Gouk et al., 2021). For testing the robustness of our algorithm, in addition to the baselines described above, we adopt several baselines that have been proposed in the supervised learning setting, including GCE (Zhang and Sabuncu, 2018), SCE (Wang et al., 2019), DAC (Thulasidasan et al., 2019), APL (Ma et al., 2020), ELR (Liu et al., 2020) and self-adaptive training (SAT) (Huang et al., 2020). We compare with the same baselines to transfer learning in few-shot classification tasks. We describe the implementation and hyper-parameters of baselines in Section B.1.

heading: 5.2 Experimental results

text:
Improved regularization for transfer learning. We report the test accuracy of fine-tuning ResNet101 on seven data sets in Table 2. We observe that our method performs the best among six regularized fine-tuning methods on average. Our proposed layer-wise regularization method provides an improvement of 1.76% on average over distance-based regularization (Gouk et al., 2021). In particular, our approach outperforms `2-PGM by 2 ⇠ 3% on both Stanford-Dogs and MIT-Indoor data sets. This suggests that adding appropriate distance constraints for all the layers is better than applying only one constraint to the top layers. In Table 10 (cf. Section B.3), we note that using the MARS norm of Gouk et al. (2021) yields comparable results to using the `2 norm with our approach.
Next, we apply our approach to medical image classification, a more distant transfer task (relative to the source data set ImageNet). We report the mean AUROC (averaged over predicting all 14 labels) on the ChestX-ray14 data set in Table 6a (cf. Section B.2). With layer-wise regularization, we see an 0.39% improvement over the baseline methods. Finally, we extend our approach to text classification. The results are consistent with the above and can be found in Table 7 (cf. Section B.2).
Improved robustness under label noise. Next, we report the test accuracy of our approach on the Indoor data set with independent and correlated noise in Table 3. We find that our approach consistently outperforms baseline methods by 1 ⇠ 3% for various settings involving label noise. First, our method (REGSL) improves the performance by over 4% on average compared to distance-based regularization (Gouk et al., 2021). This result implies our method is more robust to label noise in the training labels. Second, our method outperforms previous supervised training methods by 3.56% on average. This result suggests that regularization is critical for fine-tuning. Taken together, our results suggest that regularization and self-labeling complement each other during fine-tuning. This is reinforced by our ablation study in Section B.3; We study the influence of each component of our algorithm and find that removing any component degrades performance.
We further apply our approach to fine-tuning pre-trained ViT (Dosovitskiy et al., 2020) from noisy labels, under the same setting as Table 3. Table 6b shows the result (cf. Section B.2). First, we find that
our approach improves upon the best regularization methods by 1.17% averaged over two settings. Second, we find that our approach also improves upon self-labeling by 13.57% averaged over two settings. These two results again highlight that both regularization and self-labeling contribute to the final result. While regularization prevents the model from overfitting to the random labels, self-labeling injects the belief of the fine-tuned model into the noisy data set.
Table 4: Test accuracy with 95% confidence interval for fine-tuning ResNet-12 over 600 meta-test splits of miniImageNet.
Methods miniImageNet
5-way-1-shot 5-way-5-shot
Fine-tuning 60.56 ± 0.78 76.42 ± 0.36 `2-Norm 60.93 ± 0.81 76.57 ± 0.55 LS 61.31 ± 0.77 76.73 ± 0.62 `2-SP 61.48 ± 0.76 77.02 ± 0.62 `2-PGM 61.35 ± 0.83 77.33 ± 0.59 REGSL (ours) 61.71 ± 0.77 78.03 ± 0.54
Extension to few-shot classification. We extend our approach to a few-shot image classification task. We compare our approach to the baseline regularization methods. In Table 4, we report the average accuracy of 600 sampled tasks from the meta-test split of the miniImageNet benchmark (Vinyals et al., 2016). We find that our layer-wise regularization method achieves 0.75% improvement on average compared to previous regularization methods. Additionally, regularization methods generally improve the performance of fine-tuning by over 1%.
Ablation studies. We study the influence of removing each component from our algorithm. Results shown in Table 11 (cf. Section B.3) suggest that they all degrade performance. Thus, all three components in Algorithm 1 contribute to the final performance. The importance of each component depends on the noise rate. In particular, if the noise rate is higher than 40%, label-correction is the most critical component. Other ablation studies we present include: comparing the `2 norm and the MARS norm in our algorithm and comparing the distance parameters between layer-wise and constant regularization. We leave the details to Section B.3.

heading: 6 Conclusion

text:
This paper studied regularization methods for fine-tuning as well as their robustness properties. We investigated the generalization error of fine-tuning using PAC-Bayesian techniques. This leads to two empirical measures, which we empirically computed. The analysis inspired us to consider layer-wise regularization for fine-tuning. This approach performs well on both pre-trained ResNets and ViTs; we have found the fine-tuned distance using vanilla fine-tuning and encoded the distance patterns in layer-wise regularization. We then evaluated the performance of regularized fine-tuning from noisy labels. We proposed a self label-correction and label-reweighting approach in the noisy setting. We found that regularization and self-labeling complement each other in our experiments, leading to significant improvement over previous methods, which either use regularization methods or use self-labeling methods but not both. We discuss the limitation of our work in Section C.
Negative societal impacts. Our work concerns the theory and the empirical performance of finetuning for transfer learning. Due to its technical nature, there is very little negative societal impact. Since transfer learning is commonly used in practice, one avenue that deserves attention for these applications is to carefully evaluate bias or fairness metrics before applying our proposed approaches.
Acknowledgment. Thanks to the anonymous reviewers and the area chairs for carefully reading through our work and providing constructive feedback. Thanks to the program committee chairs for communicating with us regarding an omitted factor in equation (14), which we have added in the current version. HZ is grateful to Sen Wu and Chris Ré for stimulating discussions. HZ is supported by Northeastern University’s Khoury seed/proof-of-concept grant.

title: Improved Regularization and Robustness for Fine-tuning in Neural Networks