  Foundation models, including Vision Language Models (VLMs) and Large Language
Models (LLMs), possess the $generality$ to handle diverse distributions and
tasks, which stems from their extensive pre-training datasets. The fine-tuning
of foundation models is a common practice to enhance task performance or align
the model's behavior with human expectations, allowing them to gain
$speciality$. However, the small datasets used for fine-tuning may not
adequately cover the diverse distributions and tasks encountered during
pre-training. Consequently, the pursuit of speciality during fine-tuning can
lead to a loss of {generality} in the model, which is related to catastrophic
forgetting (CF) in deep learning. In this study, we demonstrate this phenomenon
in both VLMs and LLMs. For instance, fine-tuning VLMs like CLIP on ImageNet
results in a loss of generality in handling diverse distributions, and
fine-tuning LLMs like Galactica in the medical domain leads to a loss in
following instructions and common sense.
  To address the trade-off between the speciality and generality, we
investigate multiple regularization methods from continual learning, the weight
averaging method (Wise-FT) from out-of-distributional (OOD) generalization,
which interpolates parameters between pre-trained and fine-tuned models, and
parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA). Our
findings show that both continual learning and Wise-ft methods effectively
mitigate the loss of generality, with Wise-FT exhibiting the strongest
performance in balancing speciality and generality.

Machine Learning