  Pre-training on large-scale datasets and then fine-tuning on downstream tasks
have become a standard practice in deep learning. However, pre-training data
often contain label noise that may adversely affect the generalization of the
model. This paper aims to understand the nature of noise in pre-training
datasets and to mitigate its impact on downstream tasks. More specifically,
through extensive experiments of supervised pre-training models on synthetic
noisy ImageNet-1K and YFCC15M datasets, we demonstrate that while slight noise
in pre-training can benefit in-domain (ID) transfer performance, where the
training and testing data share the same distribution, it always deteriorates
out-of-domain (OOD) performance, where training and testing data distribution
are different. We empirically verify that the reason behind is noise in
pre-training shapes the feature space differently. We then propose a
lightweight black-box tuning method (NMTune) to affine the feature space to
mitigate the malignant effect of noise and improve generalization on both ID
and OOD tasks, considering one may not be able to fully fine-tune or even
access the pre-trained models. We conduct practical experiments on popular
vision and language models that are pre-trained on noisy data for evaluation of
our approach. Our analysis and results show the importance of this interesting
and novel research direction, which we term Noisy Model Learning.

Machine Learning;Artificial Intelligence;Computer Vision and Pattern Recognition