summaries: This paper studies the generalization and robustness of regularized fine-tuning. They present a PAC-Bayes generalization bound that depends on the distance traveled in each layer during fine-tuning and the noise stability of the pre-trained model. They empirically measure these quantities to analyze the behavior of fine-tuning. They present an algorithm that interpolates between two components: (i) regularization over the amount of distance traveled in each layer; (ii) iteratively correcting mislabeled data points that the model is highly confident and down-weighting less confident data points.  This paper includes a PAC-Bayesian analysis of distance-based regularisation during fine-tuning. Using this analysis and supplementary experiments, the authors propose a new means for tuning hyperparameters for distance-based regularisation with constraints and heuristics for dealing with label noise during fine-tuning. The experiments show that the method works slightly better than previous approaches designed for both fine-tuning and training with label noise. The paper proposes a new regularization method for training and fine-tuning large neural networks. The paper proposes to constrain the maximum change in network parameters during fine-tuning. This constraint is different in each layer. Experiments demonstrate that the proposed method outperforms previous approaches. The paper analyses the fine-tuning of neural networks. It provides a PAC-Bayes bound for the generalization of a fine-tuned network. The paper then provides three tricks to improve finetuning: layerwise regularization, label correction, and label removal. The resulting fine-tuning algorithm improves performance on various image classification tasks, including those with artificially-added label noise. This paper studies the aspects of fine-tuning a model (pre-trained on a separate dataset) in order to learn a new task. It has the following contribution:  (i) Generalization properties of fine-tuning explained by the PAC-Bayes generalization bound that depends on two terms: (a) distance between each layer from the pre-trained model and (b) noise stability of the pre-trained model.  (ii) Inspired by the generalization analysis, the paper presents an algorithm to perform fine-tuning. This algorithm includes (a) regularization term that computes the distance between the current parameters and the pre-trained model, and (b) iteratively corrects mislabeled examples where model has high confidence and down-weighting less confident examples.  (iii) Proposed algorithm is evaluated on a suite of benchmark tasks including transfer learning and few shot classification tasks.  
abstractText: A widely used algorithm for transfer learning is fine-tuning, where a pre-trained model is fine-tuned on a target task with a small amount of labeled data. When the capacity of the pre-trained model is much larger than the size of the target data set, fine-tuning is prone to overfitting and “memorizing” the training labels. Hence, an important question is to regularize fine-tuning and ensure its robustness to noise. To address this question, we begin by analyzing the generalization properties of fine-tuning. We present a PAC-Bayes generalization bound that depends on the distance traveled in each layer during fine-tuning and the noise stability of the fine-tuned model. We empirically measure these quantities. Based on the analysis, we propose regularized self-labeling—the interpolation between regularization and self-labeling methods, including (i) layer-wise regularization to constrain the distance traveled in each layer; (ii) self label-correction and label-reweighting to correct mislabeled data points (that the model is confident) and reweight less confident data points. We validate our approach on an extensive collection of image and text data sets using multiple pre-trained model architectures. Our approach improves baseline methods by 1.76% (on average) for seven image classification tasks and 0.75% for a few-shot classification task. When the target data set includes noisy labels, our approach outperforms baseline methods by 3.56% on average in two noisy settings.
authors.name: Dongyue Li
authors.name: Hongyang R. Zhang
references.authors: Takuya Akiba Shotaro Sano Toshihiko Yanase Takeru Ohta Masanori Koyama
references.title: Optuna: A nextgeneration hyperparameter optimization framework
references.venue: In International Conference on Knowledge Discovery and Data Mining,
references.authors: Sanjeev Arora Simon Du Wei Hu Zhiyuan Li Ruosong Wang
references.title: Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks
references.venue: In International Conference on Machine Learning
references.authors: Peter Bartlett Dylan J Foster Matus Telgarsky
references.title: Spectrally-normalized margin bounds for neural networks
references.venue: arXiv preprint arXiv:1706.08498,
references.authors: Peter L Bartlett Nick Harvey Christopher Liaw Abbas Mehrabian
references.title: Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks
references.venue: The Journal of Machine Learning Research,
references.authors: Shai Ben-David John Blitzer Koby Crammer Alex Kulesza Fernando Pereira Jennifer Wortman Vaughan
references.title: A theory of learning from different domains
references.venue: Machine learning,
references.authors: Xinyang Chen Sinan Wang Bo Fu Mingsheng Long Jianmin Wang
references.title: Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer
references.authors: Guneet S Dhillon Pratik Chaudhari Avinash Ravichandran Stefano Soatto
references.title: A baseline for few-shot image classification
references.venue: arXiv preprint arXiv:1909.02729,
references.authors: Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly
references.title: An image is worth 16x16 words: Transformers for image recognition at scale
references.venue: arXiv preprint arXiv:2010.11929,
references.authors: Gintare Karolina Dziugaite Daniel M Roy
references.title: Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data
references.venue: arXiv preprint arXiv:1703.11008,
references.authors: Pierre Foret Ariel Kleiner Hossein Mobahi Behnam Neyshabur
references.title: Sharpness-aware minimization for efficiently improving generalization
references.authors: Henry Gouk Timothy M Hospedales Massimiliano Pontil
references.title: Distance-based regularisation of deep networks for fine-tuning
references.authors: Gregory Griffin Alex Holub Pietro Perona
references.title: Caltech-256 object category dataset
references.authors: Yunhui Guo Honghui Shi Abhishek Kumar Kristen Grauman Tajana Rosing Rogerio Feris
references.title: Spottune: transfer learning through adaptive fine-tuning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Conference on computer vision and pattern recognition,
references.authors: Kaiming He Ross Girshick Piotr Dollár
references.title: Rethinking imagenet pre-training
references.venue: In International Conference on Computer Vision,
references.authors: Minqing Hu Bing Liu
references.title: Mining and summarizing customer reviews
references.venue: In ACM SIGKDD international conference on Knowledge discovery and data mining,
references.authors: Lang Huang Chao Zhang Hongyang Zhang
references.title: Self-adaptive training: beyond empirical risk minimization
references.venue: Advances in Neural Information Processing Systems,
references.authors: Yiding Jiang Behnam Neyshabur Hossein Mobahi Dilip Krishnan Samy Bengio
references.title: Fantastic generalization measures and where to find them
references.authors: Aditya Khosla Nityananda Jayadevaprakash Bangpeng Yao Li Fei-Fei
references.title: Novel dataset for fine-grained image categorization
references.venue: In Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Jonathan Krause Michael Stark Jia Deng Li Fei-Fei
references.title: 3d object representations for fine-grained categorization
references.venue: In Workshop on 3D Representation and Recognition,
references.authors: Hao Li Pratik Chaudhari Hao Yang Michael Lam Avinash Ravichandran Rahul Bhotika Stefano Soatto
references.title: Rethinking the hyperparameters for fine-tuning
references.venue: arXiv preprint arXiv:2002.11770,
references.authors: Xin Li Dan Roth
references.title: Learning question classifiers
references.venue: In International conference on Computational linguistics,
references.authors: Xingjian Li Haoyi Xiong Hanchao Wang Yuxuan Rao Liping Liu Jun Huan
references.title: Delta: Deep learning transfer using feature map with attention for convolutional networks
references.venue: In International Conference on Learning Representations,
references.authors: Xuhong Li Yves Grandvalet Franck Davoine
references.title: Explicit inductive bias for transfer learning with convolutional networks
references.venue: arXiv preprint arXiv:1802.01483,
references.authors: Yuanzhi Li Tengyu Ma Hongyang Zhang
references.title: Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations
references.venue: In Conference On Learning Theory,
references.authors: Sheng Liu Jonathan Niles-Weed Narges Razavian Carlos Fernandez-Granda
references.title: Early-learning regularization prevents memorization of noisy labels
references.venue: arXiv preprint arXiv:2007.00151,
references.authors: Philip M Long Hanie Sedghi
references.title: Generalization bounds for deep convolutional neural networks
references.venue: ICLR,
references.authors: Xingjun Ma Hanxun Huang Yisen Wang Simone Romano Sarah Erfani James Bailey
references.title: Normalized loss functions for deep learning with noisy labels
references.venue: In International Conference on Machine Learning,
references.authors: Subhransu Maji Esa Rahtu Juho Kannala Matthew Blaschko Andrea Vedaldi
references.title: Fine-grained visual classification of aircraft
references.venue: arXiv preprint arXiv:1306.5151,
references.authors: David A McAllester
references.title: Pac-bayesian model averaging
references.venue: In Proceedings of the twelfth annual conference on Computational learning theory,
references.authors: David A McAllester
references.title: Some pac-bayesian theorems
references.venue: Machine Learning,
references.authors: Rafael Müller Simon Kornblith Geoffrey Hinton
references.title: When does label smoothing help
references.venue: arXiv preprint arXiv:1906.02629,
references.authors: Vaishnavh Nagarajan Zico Kolter
references.title: Deterministic pac-bayesian generalization bounds for deep networks via generalizing noise-resilience
references.venue: In International Conference on Learning Representations,
references.authors: Nagarajan Natarajan Inderjit S Dhillon Pradeep Ravikumar Ambuj Tewari
references.title: Learning with noisy labels
references.venue: In Conference on Neural Information Processing Systems,
references.authors: Behnam Neyshabur Srinadh Bhojanapalli Nathan Srebro
references.title: A pac-bayesian approach to spectrallynormalized margin bounds for neural networks
references.authors: Maria-Elena Nilsback Andrew Zisserman
references.title: Automated flower classification over a large number of classes
references.venue: In Indian Conference on Computer Vision, Graphics & Image Processing,
references.authors: Bo Pang Lillian Lee
references.title: A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts
references.venue: In Annual meeting on Association for Computational Linguistics,
references.authors: Bo Pang Lillian Lee
references.title: Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
references.venue: In Annual meeting on association for computational linguistics,
references.authors: Pranav Rajpurkar Jeremy Irvin Kaylie Zhu Brandon Yang Hershel Mehta Tony Duan Daisy Ding Aarti Bagul Curtis Langlotz Katie Shpanskaya
references.title: Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning
references.venue: arXiv preprint arXiv:1711.05225,
references.authors: Alexander Ratner Christopher De Sa Sen Wu Daniel Selsam Christopher Ré
references.title: Data programming: Creating large training sets, quickly
references.venue: Advances in neural information processing systems,
references.authors: Alexander Ratner Stephen H Bach Henry Ehrenberg Jason Fries Sen Wu Christopher Ré
references.title: Snorkel: Rapid training data creation with weak supervision
references.venue: In International Conference on Very Large Data Bases. NIH Public Access,
references.authors: Alexander Ratner Braden Hancock Jared Dunnmon Frederic Sala Shreyash Pandey Christopher Ré
references.title: Training complex models with multi-task weak supervision
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein
references.title: Imagenet large scale visual recognition challenge
references.venue: International journal of computer vision,
references.authors: Khaled Saab Sarah M Hooper Nimit S Sohoni Jupinder Parmar Brian Pogatchnik Sen Wu Jared A Dunnmon Hongyang R Zhang Daniel Rubin Christopher Ré
references.title: Observational supervision for medical image classification using gaze data
references.venue: In International Conference on Medical Image Computing and Computer-Assisted Intervention,
references.authors: Hadi Salman Andrew Ilyas Logan Engstrom Ashish Kapoor Aleksander Madry
references.title: Do adversarially robust imagenet models transfer better
references.venue: arXiv preprint arXiv:2007.08489,
references.authors: Ali Sharif Razavian Hossein Azizpour Josephine Sullivan Stefan Carlsson
references.title: Cnn features off-the-shelf: an astounding baseline for recognition
references.venue: In Conference on computer vision and pattern recognition workshops,
references.authors: Richard Socher Alex Perelygin Jean Wu Jason Chuang Christopher D Manning Andrew Ng Christopher Potts
references.title: Recursive deep models for semantic compositionality over a sentiment treebank
references.venue: In Conference on empirical methods in natural language processing,
references.authors: Kai Sheng Tai Peter Bailis Gregory Valiant
references.title: Sinkhorn label allocation: Semi-supervised classification via annealed self-training
references.venue: arXiv preprint arXiv:2102.08622,
references.authors: Chuanqi Tan Fuchun Sun Tao Kong Wenchang Zhang Chao Yang Chunfang Liu
references.title: A survey on deep transfer learning
references.venue: In International conference on artificial neural networks,
references.authors: Sunil Thulasidasan Tanmoy Bhattacharya Jeff Bilmes Gopinath Chennupati Jamal MohdYusof
references.title: Combating label noise in deep learning using abstention
references.authors: Yonglong Tian Yue Wang Dilip Krishnan Joshua B Tenenbaum Phillip Isola
references.title: Rethinking few-shot image classification: a good embedding is all you need
references.venue: arXiv preprint arXiv:2003.11539,
references.authors: Joel A Tropp
references.title: An introduction to matrix concentration inequalities
references.venue: arXiv preprint arXiv:1501.01571,
references.authors: Oriol Vinyals Charles Blundell Timothy Lillicrap Koray Kavukcuoglu Daan Wierstra
references.title: Matching networks for one shot learning
references.venue: arXiv preprint arXiv:1606.04080,
references.authors: C. Wah S. Branson P. Welinder P. Perona S. Belongie
references.title: The Caltech-UCSD Birds-200-2011 Dataset
references.venue: Technical Report CNS-TR-2011-001, California Institute of Technology,
references.authors: Xiaosong Wang Yifan Peng Le Lu Zhiyong Lu Mohammadhadi Bagheri Ronald M Summers
references.title: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases
references.venue: In Conference on computer vision and pattern recognition,
references.authors: Yisen Wang Xingjun Ma Zaiyi Chen Yuan Luo Jinfeng Yi James Bailey
references.title: Symmetric cross entropy for robust learning with noisy labels
references.venue: In International Conference on Computer Vision,
references.authors: Colin Wei Tengyu Ma
references.title: Data-dependent sample complexity of deep neural networks via lipschitz augmentation
references.venue: arXiv preprint arXiv:1905.03684,
references.authors: Colin Wei Tengyu Ma
references.title: Improved sample complexities for deep networks and robust classification via an all-layer margin
references.venue: arXiv preprint arXiv:1910.04284,
references.authors: Janyce Wiebe Theresa Wilson Claire Cardie
references.title: Annotating expressions of opinions and emotions in language
references.venue: Language resources and evaluation,
references.authors: Sen Wu Hongyang R Zhang Christopher Ré
references.title: Understanding and improving information transfer in multi-task learning
references.authors: Yi Xu Lei Shang Jinxing Ye Qi Qian Yu-Feng Li Baigui Sun Hao Li Rong Jin
references.title: Dash: Semi-supervised learning with dynamic thresholding
references.venue: In International Conference on Machine Learning,
references.authors: Fan Yang Hongyang R Zhang Sen Wu Weijie J Su Christopher Ré
references.title: Analysis of information transfer from heterogeneous sources via precise high-dimensional asymptotics
references.authors: Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals
references.title: Understanding deep learning requires rethinking generalization
references.venue: arXiv preprint arXiv:1611.03530,
references.authors: Zhilu Zhang Mert R Sabuncu
references.title: Generalized cross entropy loss for training deep neural networks with noisy labels
references.venue: arXiv preprint arXiv:1805.07836,
references.authors: Wenda Zhou Victor Veitch Morgane Austern Ryan P Adams Peter Orbanz
references.title: Non-vacuous generalization bounds at the imagenet scale: a pac-bayesian compression approach
references.venue: arXiv preprint arXiv:1804.05862,
references.authors: Yang Zou Zhiding Yu Xiaofeng Liu BVK Kumar Jinsong Wang
references.title: Confidence regularized self-training
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
sections.heading: 1 Introduction
sections.text: Learning from limited labeled data is a fundamental problem in many real-world applications (Ratner et al., 2016, 2017). A common approach to address this problem is fine-tuning a large model that has been pre-trained on publicly available labeled data (He et al., 2019). Since fine-tuning is typically applied to a target task with limited labels, this algorithm is prone to overfitting or “memorization” issues (Tan et al., 2018). These issues worsen when the target task contains noisy labels (Zhang et al., 2016). In this paper, we analyze regularization methods for fine-tuning from both theoretical and empirical perspectives. Based on the analysis, we propose a regularized self-labeling approach that improves the generalization and robustness properties of fine-tuning.
  Previous works (Li et al., 2018a,b) have proposed regularization methods to constrain the distance between a fine-tuned model and the pre-trained model in the Euclidean norm. Li et al. (2020) provides extensive study to show that the performance of fine-tuning and the benefit of adding regularization depend on the hyperparameter choices. Salman et al. (2020) empirically find that performing adversarial training during the pre-training phase helps learn pre-trained models that transfer better to downstream tasks. The work of Gouk et al. (2021) generalizes the above ideas to various norm choices and finds that projected gradient descent methods perform well for implementing distance-based regularization. Additionally, they derive generalization bounds for fine-tuning using Rademacher complexity. These works focus on settings where there is no label noise in the target data set. When
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  label noise is present, for example, due to applying weak supervision techniques (Ratner et al., 2016), an important question is to design methods that are robust to such noise. The problem of learning from noisy labels has a rich history of study in supervised learning (Natarajan et al., 2013). In contrast, little is known in the transfer learning setting. These considerations motivate us to analyze the generalization and robustness properties of fine-tuning.
  In Section 4.1, we begin by conducting a PAC-Bayesian analysis of regularized fine-tuning. This is inspired by recent works that have found PAC-Bayesian analysis correlates with empirical performance better than Rademacher complexity (Jiang et al., 2020). We identify two critical measures for analyzing the generalization performance of fine-tuning. The first measure is the `2 norm of the distance between the pre-trained model (initialization) and the fine-tuned model. The second measure is the perturbed loss of the fine-tuned model, i.e. its loss after the model weights get perturbed by random noise. First, we observe that the fine-tuned weights remain closed to the pre-trained model. Moreover, the top layers travel much further away from the pre-trained model than the bottom layers. Second, we find that fine-tuning from a pre-trained model implies better noise stability than training from a randomly initialized model. In Section 4.2, we evaluate regularized fine-tuning for target tasks with noisy labels. We find that fine-tuning is prone to “memorizing the noisy labels”, and regularization helps alleviate such memorization behavior. Moreover, we observe that the neural network has not yet overfitted to the noisy labels during the early phase of fine-tuning. Thus, its prediction could be used to relabel the noisy labels.
  We propose an algorithm that incorporates layerwise regularization and self-labeling for improved regularization and robustness based on our results. Figure 1 illustrates the two components. First, we encode layer-wise distance constraints to regularize the model weights at different levels. Compared to (vanilla) fine-tuning, our algorithm reduces the gap between the training and test accuracy, thus alleviating overfitting. Second, we add a self-labeling mechanism that corrects and reweights “noisy labels” based on the neural network’s predictions. Figure 1 shows that our algorithm effectively hinders the model from learning the incorrect labels by relabeling them to correct ones.
  In Section 5, we evaluate our proposed algorithm for both transfer learning and few-shot classification tasks with image and text data sets. First, using ResNet-101 (He et al., 2016) as the pre-trained model, our algorithm outperforms previous fine-tuning methods on seven image classification tasks by 1.76% on average and 3.56% when their labels are noisy. Second, we find qualitatively similar results for applying our approach to medical image classification tasks (ChestX-ray14 (Wang et al., 2017; Rajpurkar et al., 2017)) and vision transformers (Dosovitskiy et al., 2020). Finally, we extend our approach to few-shot learning and sentence classification. For these related but different tasks and data modalities, we find an improvement of 0.75% and 0.46% over previous methods, respectively.
  In summary, our contributions are threefold. First, we provide a PAC-Bayesian analysis of regularized fine-tuning. Our result implies empirical measures that explain the generalization performance of regularized fine-tuning. Second, we present a regularized self-labeling approach to enhance the generalization and robustness properties of fine-tuning. Third, we validate our approach on an extensive collection of classification tasks and pre-trained model architectures.
sections.heading: 2 Related work
sections.text: Fine-tuning is widely used in multi-task and transfer learning, meta-learning, and few-shot learning. Previous works (Li et al., 2018b,a) find that injecting `2 regularization helps improve the performance of fine-tuning. Li et al. (2018b) propose a `2 distance regularization method that penalizes the `2 distance between the fine-tuned weights and the pre-trained weights. Li et al. (2018a) penalize the distance between the feature maps as opposed to the layer weights. Chen et al. (2019) design a regularization method that suppresses the spectral components (of the feature maps) with small
  singular values to avoid negative transfer. Gouk et al. (2021) instead encode distance constraints in constrained minimization and use projected gradient descent to ensure the weights are close to the pre-trained model. Salman et al. (2020) show that fine-tuning from adversarially robust pre-trained models outperforms fine-tuning from (standard) pre-trained models.
  The robustness of learning algorithms in the presence of label noise has been extensively studied in supervised learning (Natarajan et al., 2013). Three broad ideas for designing robust algorithms include defining novel losses, identifying noisy labels, and regularization methods. Zhang and Sabuncu (2018) design the robust Generalized Cross Entropy (GCE) loss which is a mixture of Cross Entropy (CE) and mean absolute error. The Symmetric Cross Entropy (SCE) (Wang et al., 2019) loss combines reverse cross-entropy with the CE loss. Ma et al. (2020) proposes APL, which normalizes the loss to be robust to noisy labels and combines active and passive loss functions. Thulasidasan et al. (2019) propose DAC, which identifies and suppresses the signals of noisy samples by abstention-based training. Liu et al. (2020) introduce ELR, an early learning regularization approach to mitigate label “memorization”. Huang et al. (2020) propose a self-adaptive training method that corrects noisy labels and reweights training data to suppress erroneous signals. These works primarily concern the supervised learning setting. To the best of our knowledge, fine-tuning algorithms under label noise are relatively under-explored in transfer learning. Our approach draws inspiration from the semi-supervised learning literature, which has evaluated pseudo-labeling and self-training approaches given a limited amount of labeled data, and a large amount of unlabeled data (Zou et al., 2019; Xu et al., 2021; Tai et al., 2021). A very recent work considered a sharpness-aware approach and evaluated its performance for fine-tuning from noisy labels (Foret et al., 2021). It would be interesting to see if combining their approach with our ideas could lead to better results.
  From a theoretical perspective, the seminal work of Ben-David et al. (2010) considers a setting where labeled data from many source tasks and a target task is available. They show that minimizing a weighted combination of the source and target empirical risks leads to the best result. In the supervised setting, Arora et al. (2019) provide data-dependent generalization bounds for neural networks. Nagarajan and Kolter (2018) and Wei and Ma (2019a,b) provide improved generalization bounds that depend only polynomially on the depth of the neural network characterized by a margin condition. Recent work has found that the PAC-Bayes theory provides generalization measures that correlate with empirical generalization performance better than other alternatives (Jiang et al., 2020). We defer a more extensive review of PAC-Bayesian generalization theory to Section A.
sections.heading: 3 Preliminaries
sections.text: Problem setup. We begin by formally introducing the setup. Suppose we would like to solve a target task. We have a training data set of size n(t). Let (x(t)1 , y (t) 1 ), . . . , (x (t) n(t) , y (t) n(t)
  ) be the feature vectors and the labels of the training data set. We assume that every x(t)
  i lies in a d-dimensional space
  denoted by X ⇢ Rd. Following standard terminologies in statistical learning, we assume all data are drawn from some unknown distribution supported on X ⇥ Y , where Y ✓ R is the label space. Denote the underlying data distribution as P(t).
  For our result in Section 4.1, we consider feedforward neural networks, and our results can also be extended to convolutional neural networks (e.g., using ideas from Long and Sedghi (2020)). Consider an L layer neural network. For each layer i from 1 to L, let i be the activation function and let Wi be the weight matrix at layer i. Given an input z to the i-th layer, the output is then denoted as i(z) = i(Wiz). Thus, the final output of the network is given by
  fW (x) = L L 1 · · · 1(x), for any input x 2 X , where we use W = [W1, . . . ,WL] to include all the parameters of the network for ease of notation. The prediction error of fW is measured by a loss function `(·) that is both convex and 1-Lipschitz.
  L (t)(fW ) = E (x,y)⇠P(t) [`(fW (x), y)] . (1)
  Suppose we have access to a pre-trained source model Ŵ (s). Using Ŵ (s) as an initialization, we then fine-tune the layers Ŵ (s)1 , . . . , Ŵ (s) L to predict the target labels.
  Applications. We consider seven image classification data sets described in Table 1. We use ResNets pre-trained on ImageNet as the initialization Ŵ (s) (Russakovsky et al., 2015; He et al., 2016). We
  perform fine-tuning using the pre-trained network on the above data sets. See Section 5.1 for further description of the training procedure.
  Modeling label noise. In many settings, fine-tuning is applied to a target task whose labels may contain noise; for example, if the target labels are created using weak supervision techniques (Ratner et al., 2019; Saab et al., 2021). To capture such settings, we denote a noisy data set as (x(t)1 , ỹ (t) 1 ), . . . , (x (t) n(t) , ỹ (t) n(t) ), where ỹ(t) i is a noisy version of y(t) i
  . We consider two types of label noise: independent noise and correlated noise. We say that the label noise is independent if it is independent of the input feature vector
  Pr(ỹ(t) i = k|y(t) i = j, x(t) i ) = Pr(ỹ(t) i = k|y(t) i = j) = ⌘j,k
  for some fixed ⌘j,k between 0 and 1. On the other hand, we say that the label noise is correlated if it depends on the input feature vector (i.e. the above equation does not hold).
sections.heading: 4 Our proposed approaches
sections.text: Given the problem setup described above, next, we study the generalization and robustness properties of regularized fine-tuning methods. First, we present a PAC-Bayes generalization bound for regularized fine-tuning. This result motivates us to evaluate two empirical measures: the fine-tuned distance in each layer and the perturbed loss of the fine-tuned model. Second, we consider fine-tuning from noisy labels. We show that layer-wise regularization prevents the model from memorizing the noisy labels. We then suggest injecting predictions of the model during training, similar to self-training and pseudo-labeling. Finally, we incorporate both components into our regularized self-labeling approach, blending the strengths of both to improve the generalization performance and the robustness of fine-tuning.
sections.heading: 4.1 Fine-tuning and regularization
sections.text: We consider the following regularized fine-tuning problem, which constrains the network from traveling too far from the pre-trained initialization Ŵ (s) (Li et al., 2018a,b; Gouk et al., 2021).
  Ŵ argmin L̂(t)(fW ) (2)
  s.t. kWi Ŵ (s) i kF  Di, 8 i = 1, . . . , L. (3)
  Above, Di is a hyperparameter that constrains how far the i-th layer Wi can travel from the pre-trained initialization Ŵ (s)
  i . Previous works have observed
  that stronger regularization reduces the generalization gap during fine-tuning (cf. Figure 2). Next, we analyze the generalization error of Ŵ , that is, L (t)(f
  Ŵ ) L̂(t)(f Ŵ ), where L(t)(f Ŵ ) is the test
  loss of f Ŵ according to equation (1) and L̂(t)(f Ŵ ) is the empirical loss of f Ŵ on the training data set.
  PAC-Bayesian analysis. We begin by analyzing the generalization error of f
  Ŵ using PAC-Bayesian
  tools (McAllester, 1999a,b). This departs from the previous work of Gouk et al. (2021), which bases on their analysis using Rademacher complexity. This change of perspective is inspired by several recent works that have found PAC-Bayesian bounds to better correlate with empirical performance than Rademacher complexity bounds (Jiang et al., 2020). We refer the interested reader to Section A for further references from this line of work. After presenting our result, we provide empirical measures of our result and discuss the practical implication.
  Theorem 4.1 (PAC-Bayes generalization bound for fine-tuning). Suppose for every i = 1, . . . , L, kŴ
  (s) i k2  Bi, for a fixed Bi > 1. Suppose the feature vectors in the domain X are all bounded: kxk2  C1 for every x 2 X , for some C1 1. Finally, suppose the loss function `(·) is 1-Lipschitz and bounded from above by a fixed constant C2. Under these conditions, let fŴ be the minimizer of regularized fine-tuning, solved from problem (3). Let " > 0 be an arbitrary small value. Let H be the maximum over the width over all the L layers and the input dimension d. Then, with probability at least 1 2 for some small > 0, the expected loss L(t)(f Ŵ ) is upper bounded by
  L (t)(f Ŵ )  L̂(t)(f Ŵ ) + "+ C2
  s 36 "2 ·C21H log(4LHC2) ⇣ PL i=1 QL j=1(Bj+Dj) Bi+Di ⌘2⇣PL i=1 D 2 i ⌘ +3 ln n (t) +8
  n(t) . (4)
  Relation to prior works. Compared to the result of Gouk et al. (2021), we instead proceed by factoring the generalization error into a noise error (i.e. the " term in equation (4)) and a KL-divergence between Ŵ (s) and Ŵ (i.e. the final term in equation (4)). The proof of Theorem 4.1 is based on Neyshabur et al. (2018) and is presented in Appendix A. The difference between Theorem 4.1 and the result of Neyshabur et al. (2018) is that our result is stated for the (e.g. cross-entropy) loss function `(·) whereas Neyshabur et al. (2018) states the result using the soft margin loss.
  Our result suggests that the fine-tuned distances {Di}Li=1 and the perturbed loss (more precisely EU ⇥ `(f Ŵ+U (x), y) ⇤
  where every entry of U is drawn from N (0, 2)) are two important measures for fine-tuning. Next, we empirically evaluate these two measures on real-world data sets.
  Fine-tuned distances {Di}Li=1. First, we present our empirical finding for the fine-tuned distances in each layer. We fine-tune a ResNet-18 model (pre-trained on ImageNet) on seven data sets and calculate the Frobenius distance between Ŵ (s)
  i and Ŵi for every layer i.
  Figure 3a shows the ratio of the fine-tuned distances to the pre-trained weights in each layer. First, we observe that the fine-tuned distances are relatively small compared to the pre-trained network. This means that fine-tuning stays within a small local region near the pre-trained model. Second, we find that Di varies across layers. Di is smaller at lower layers and is larger at higher layers. This observation aligns with the folklore intuition that different layers in Convolutional neural networks play a different role (Guo et al., 2019). Bottom layers extract higher-level representations of features
  and stay close to the pre-trained model. Top layers extract class-specific features and vary among different tasks. Thus, top layers travel further away from the pre-trained model.
  Based on the above empirical finding, we propose layer-wise constraints as regularization. In particular, the constraints for the bottom layers should be small, and the constraints for the top layers should be large. In practice, we find that an exponentially increasing scheme involving a base distance parameter D and a scale factor works well. This is summarized in the following regularized fine-tuning problem:
  Ŵ argmin L̂(t)(fW ) (5)
  s.t. kWi Ŵ (s) i kF  D · i 1 , 8 i = 1, . . . , L. (6)
  where the scale factor > 1 according to Figure 3a. The previous work of Gouk et al. (2021) uses the same distance parameter in equation (6), i.e., Di = D, 8i = 1, . . . , L. In our experiments, we have observed that such constant regularization constraints are only active at the top layers. With layer-wise regularization, we instead extend the constraints to the bottom layers as well. In Table 8 (cf. Section B.3), we compare the value of P L
  i=1 D 2 i between constant regularization and layer-wise regularization. We find that layer-wise regularization leads to a smaller value of P L
  i=1 D 2 i , thus
  indicating a tighter generalization bound according to Theorem 4.1. Perturbed loss EU h `(fŴ+U(x), y) i . Second, we compare the perturbed loss of models finetuned from random, pre-trained, and adversarially robust pre-trained model initializations. The perturbed loss is measured by the average loss on the training set, after perturbing each entry of the layer weights by a Gaussian random variable N(0, 2).
  In Table 3b, we report the perturbed losses for both the CUB-200-2011 and the Indoor data set. The results show that models fine-tuned from pre-trained initializations are more stable than models fine-tuned from random initializations.
  Explaining why adversarial robustness helps fine-tuning. Recent work (Salman et al., 2020) found that performing adversarial training in the pre-training phase leads to models that transfer better to downstream tasks. More precisely, the adversarial training objective is defined as
  L̃ (s) adv(✓) = min
  W
  1
  n(s)
  n (s)X
  i=1
  max k k2"
  `(fW (x (s) i + ), y(s) i ). (7)
  Let Ŵ (s)adv be a fine-tuned neural network using a pre-trained model from adversarial training. We measure the perturbed loss of Ŵ (s)adv using the pre-trained model provided by Salman et al. (2020) on the CUB-200-2011 and Indoor data set. Table 3b shows that f
  Ŵ (s) adv often incurs lower perturbed losses than models fine-tuned from random and pre-trained initializations. In Section B.3, we additionally observe that the layers of Ŵ (s)adv generally have lower Frobenious norms compared to Ŵ
  (s). Thus, the improved noise stability and the smaller layer-wise norms together imply a tighter generalization bound for adversarial training. We leave a thorough theoretical analysis of adversarial training in the context of fine-tuning to future works.
sections.heading: 4.2 Fine-tuning and robustness
sections.text: Next, we extend our approach to fine-tuning from noisy labels. We compare fine-tuning with and without regularization on a target task with independent label noise to motivate our approach. In Figure 4, we plot the training and test accuracy curves on the Indoor dataset with two noise rates. We observe that the test accuracy increases at first, meaning the network learns from the correct labels during the first few epochs. As the learning process progresses, the test accuracy of fine-tuning decreases, implying the network is “overfitting” to noisy labels. Furthermore, the gap between the training and test accuracy curves is more than 20%. Similar observations have been presented in prior works (Zhang et al., 2016; Li et al., 2018c; Liu et al., 2020), where over-parametrized neural networks can memorize the entire training set. One way to mitigate memorization is to explicitly reduce model capacity (Wu et al., 2020; Yang et al., 2021). Another way is via explicit regularization since the distance constraint significantly reduces the training-test accuracy gaps and improves test accuracy by a significant margin. Based on our empirical observation, the neural network has some discriminating power during the early fine-tuning phase. Thus, we can leverage the model predictions to relabel incorrect labels and reweight data points with incorrect labels. We describe the two components next.
  Self label-correction. We propose a label correction step to augment the number of correct labels. We leverage the discriminating power of a network during the early phase and correct data points for which the model has high confidence. Concretely, let pi be the prediction of the i-th data point. Given a confidence threshold pt at epoch t, if max(pi) pt and argmax(pi) 6= ỹ (t) i
  , we identify data point i as noisy and relabel it to
  ỹ (t) i = argmax(pi).
  Figure 5a illustrates the precision of this step. We define precision as the number of correctly relabeled data points divided by the total number of data points whose labels are relabeled. We observe that precision is high at the beginning (around 60%) and gradually decreases in later epochs. Moreover, we find precision increases with regularization. This suggests an intricate interaction between regularization and self-labeling during fine-tuning. We elaborate on the role of regularization in label correction by looking at the number of relabeled data points in Section B.3.
  Self label-reweighting. We incorporate a soft data removal step to prevent the model from overfitting to noisy labels. We identify data points as noisy if their loss values are large and thus down-weight them during training. The model will prioritize data points with correct labels in its gradient by assigning smaller weights to large-loss data points. More precisely, given a data point (x(t)
  i , ỹ (t) i
  ), we reweight it by !i = exp( `(fW (x (t) i ), ỹ(t) i )/⌧) where ⌧ is a temperature parameter. In the implementation, we normalize the weights of every data point in every mini-batch B:
  L̂B(fW ) = 1P
  i2B !i
  X i2B !i · `(fW (x (t) i ), ỹ(t) i ).
  Algorithm 1 Regularized self-labeling (REGSL) Input: Input nosiy data (x(t)1 , ỹ (t) 1 ), . . . , (x (t) n(t) , ỹ(t) n(t) ), network model fW , constraint distance D, distance scale factor , re-weight start step Er , re-weight temperature ⌧ , label correction start step Ec, label correction threhold pt, and fine-tuning steps T 1: Initialize model parameters with pre-trained weights W = Ŵ s and the start step t = 0 2: while t < T do 3: Fetch a mini-batch of data {(x(t)i , ỹ (t) i )} m i=1
  4: for each data point i in the batch do 5: Calculate the predicted probability pi = softmax(fW (x(t)i )) 6: if t > Ec and max(pi) > pt and argmax(pi) 6= ỹ(t)i then 7: Correct the label ỹ(t)i = argmax(pi) 8: end if 9: Calculate loss `i = `(fW (x(t)i ), ỹ (t) i )
  10: if t > Er then Calculate the weight !i = exp( `i/⌧) else Set the weight !i = 1 11: end for 12: Update fW by SGD on L = 1P
  i !i
  Pm i !i`i and t = t+ 1
  13: Project the weights Wi of each layer inside the layer-wise constraint region as in Equation 6 14: end while
  Figure 5b compares the average weight of data points with the correct label to data points with an incorrect label. We find that with our reweighting scheme, the average weight of incorrectly labeled data points is much lower than the average weight of correctly labeled data points. Thus, the reweighting scheme expands the gradient of correctly labeled data points, thus reducing the fraction of noisy data points in the training set.
  The pseudo-code for our final approach, which combines both layer-wise regularization and selflabeling, is presented in Algorithm 1.
sections.heading: 5 Experiments
sections.text: We evaluate our proposed algorithm in a wide range of tasks and pre-trained networks. First, we show that our algorithm outperforms the baselines by 1.75% on average over seven transfer learning tasks and can generalize to a more distant task of medical images. Second, our algorithm improves the robustness of fine-tuning in the presence of noisy labels compared to previous methods. Our algorithm improves over numerous baselines by 3.56% on average under both independent and correlated label noise. Moreover, our approach can achieve a similar performance boost for finetuning vision transformer models on noisy labeled data. Finally, we show our algorithms improve over fine-tuning baselines by 0.75% on average for the related task of few-shot classification. Our extensive evaluation confirms that our approach applies to a broad range of settings, validating our algorithmic insights. Due to space constraints, we defer the ablation studies to Section B.3. Our code is available at https://github.com/NEU-StatsML-Research/Regularized-Self-Labeling.
sections.heading: 5.1 Experimental setup
sections.text: Data sets. First, we evaluate fine-tuning on seven image classification data sets and one medical image data set. The statistics of the seven image data sets are described in Table 1. For the medical imaging task, we consider the ChestX-ray14 data set contains 112120 frontal-view chest X-ray images labeled with 14 different diseases (Wang et al., 2017; Rajpurkar et al., 2017). Second, we evaluate Algorithm 1 under two different kinds of label noise to test the robustness of fine-tuning. We selected the MIT-Indoor data set (Sharif Razavian et al., 2014) as the benchmark data set and randomly flipped the labels of the training samples. The results for the other data sets are similar, which can be found in Section B.2. We consider both independent random noise and correlated noise in our experiments. We generate the independent random noise by flipping the labels uniformly with a given noise rate. We simulate the correlated noise by using the predictions of an auxiliary network as noisy labels. Section B.1 describes the label flipping process. For few-shot image classification, we conduct experiments on the miniImageNet benchmark (Vinyals et al., 2016) following the setup of Tian et al. (2020).
  Architectures. We use the ResNet-101 (He et al., 2016) network for transfer learning tasks and ResNet-18 (He et al., 2016) network for ChestX-ray data set. We use ResNet-18 network for the labelnoise experiments and extend the our algorithm to the vision transformer (ViT) model (Dosovitskiy et al., 2020). The ResNet models are pre-trained on ImageNet (Russakovsky et al., 2015) and the ViT model is pre-trained on ImageNet-21k data set (Dosovitskiy et al., 2020). For the few-shot learning tasks, we use ResNet-12 pre-trained on the meta-training data set as in the previous work (Tian et al., 2020). We set four different values of Di for the four blocks of the ResNet models in our algorithm. We describe the fine-tuning procedure and the hyperparameters in Section B.1.
  Baselines. For the transfer learning tasks, we use the Frobenius norm for distance regularization. We present ablation studies to compare the Frobenius norm and other norms in Section B.3. we compare our algorithm with fine-tuning with early stop (Fine-tuning), fine-tuning with weight decay (`2- Norm), label smoothing (LS) method formulated in Müller et al. (2019), `2-SP (Li et al., 2018b), and ` 2-PGM (Gouk et al., 2021). For testing the robustness of our algorithm, in addition to the baselines described above, we adopt several baselines that have been proposed in the supervised learning setting, including GCE (Zhang and Sabuncu, 2018), SCE (Wang et al., 2019), DAC (Thulasidasan et al., 2019), APL (Ma et al., 2020), ELR (Liu et al., 2020) and self-adaptive training (SAT) (Huang et al., 2020). We compare with the same baselines to transfer learning in few-shot classification tasks. We describe the implementation and hyper-parameters of baselines in Section B.1.
sections.heading: 5.2 Experimental results
sections.text: Improved regularization for transfer learning. We report the test accuracy of fine-tuning ResNet101 on seven data sets in Table 2. We observe that our method performs the best among six regularized fine-tuning methods on average. Our proposed layer-wise regularization method provides an improvement of 1.76% on average over distance-based regularization (Gouk et al., 2021). In particular, our approach outperforms `2-PGM by 2 ⇠ 3% on both Stanford-Dogs and MIT-Indoor data sets. This suggests that adding appropriate distance constraints for all the layers is better than applying only one constraint to the top layers. In Table 10 (cf. Section B.3), we note that using the MARS norm of Gouk et al. (2021) yields comparable results to using the `2 norm with our approach.
  Next, we apply our approach to medical image classification, a more distant transfer task (relative to the source data set ImageNet). We report the mean AUROC (averaged over predicting all 14 labels) on the ChestX-ray14 data set in Table 6a (cf. Section B.2). With layer-wise regularization, we see an 0.39% improvement over the baseline methods. Finally, we extend our approach to text classification. The results are consistent with the above and can be found in Table 7 (cf. Section B.2).
  Improved robustness under label noise. Next, we report the test accuracy of our approach on the Indoor data set with independent and correlated noise in Table 3. We find that our approach consistently outperforms baseline methods by 1 ⇠ 3% for various settings involving label noise. First, our method (REGSL) improves the performance by over 4% on average compared to distance-based regularization (Gouk et al., 2021). This result implies our method is more robust to label noise in the training labels. Second, our method outperforms previous supervised training methods by 3.56% on average. This result suggests that regularization is critical for fine-tuning. Taken together, our results suggest that regularization and self-labeling complement each other during fine-tuning. This is reinforced by our ablation study in Section B.3; We study the influence of each component of our algorithm and find that removing any component degrades performance.
  We further apply our approach to fine-tuning pre-trained ViT (Dosovitskiy et al., 2020) from noisy labels, under the same setting as Table 3. Table 6b shows the result (cf. Section B.2). First, we find that
  our approach improves upon the best regularization methods by 1.17% averaged over two settings. Second, we find that our approach also improves upon self-labeling by 13.57% averaged over two settings. These two results again highlight that both regularization and self-labeling contribute to the final result. While regularization prevents the model from overfitting to the random labels, self-labeling injects the belief of the fine-tuned model into the noisy data set.
  Table 4: Test accuracy with 95% confidence interval for fine-tuning ResNet-12 over 600 meta-test splits of miniImageNet.
  Methods miniImageNet
  5-way-1-shot 5-way-5-shot
  Fine-tuning 60.56 ± 0.78 76.42 ± 0.36 `2-Norm 60.93 ± 0.81 76.57 ± 0.55 LS 61.31 ± 0.77 76.73 ± 0.62 `2-SP 61.48 ± 0.76 77.02 ± 0.62 `2-PGM 61.35 ± 0.83 77.33 ± 0.59 REGSL (ours) 61.71 ± 0.77 78.03 ± 0.54
  Extension to few-shot classification. We extend our approach to a few-shot image classification task. We compare our approach to the baseline regularization methods. In Table 4, we report the average accuracy of 600 sampled tasks from the meta-test split of the miniImageNet benchmark (Vinyals et al., 2016). We find that our layer-wise regularization method achieves 0.75% improvement on average compared to previous regularization methods. Additionally, regularization methods generally improve the performance of fine-tuning by over 1%.
  Ablation studies. We study the influence of removing each component from our algorithm. Results shown in Table 11 (cf. Section B.3) suggest that they all degrade performance. Thus, all three components in Algorithm 1 contribute to the final performance. The importance of each component depends on the noise rate. In particular, if the noise rate is higher than 40%, label-correction is the most critical component. Other ablation studies we present include: comparing the `2 norm and the MARS norm in our algorithm and comparing the distance parameters between layer-wise and constant regularization. We leave the details to Section B.3.
sections.heading: 6 Conclusion
sections.text: This paper studied regularization methods for fine-tuning as well as their robustness properties. We investigated the generalization error of fine-tuning using PAC-Bayesian techniques. This leads to two empirical measures, which we empirically computed. The analysis inspired us to consider layer-wise regularization for fine-tuning. This approach performs well on both pre-trained ResNets and ViTs; we have found the fine-tuned distance using vanilla fine-tuning and encoded the distance patterns in layer-wise regularization. We then evaluated the performance of regularized fine-tuning from noisy labels. We proposed a self label-correction and label-reweighting approach in the noisy setting. We found that regularization and self-labeling complement each other in our experiments, leading to significant improvement over previous methods, which either use regularization methods or use self-labeling methods but not both. We discuss the limitation of our work in Section C.
  Negative societal impacts. Our work concerns the theory and the empirical performance of finetuning for transfer learning. Due to its technical nature, there is very little negative societal impact. Since transfer learning is commonly used in practice, one avenue that deserves attention for these applications is to carefully evaluate bias or fairness metrics before applying our proposed approaches.
  Acknowledgment. Thanks to the anonymous reviewers and the area chairs for carefully reading through our work and providing constructive feedback. Thanks to the program committee chairs for communicating with us regarding an omitted factor in equation (14), which we have added in the current version. HZ is grateful to Sen Wu and Chris Ré for stimulating discussions. HZ is supported by Northeastern University’s Khoury seed/proof-of-concept grant.
title: Improved Regularization and Robustness for Fine-tuning in Neural Networks