TSVD: Bridging Theory and Practice in Continual Learning with Pre-trained Models
Liangzu Peng Juan Elenter Joshua Agterberg Alejandro Ribeiro Rene Vidal
The goal of continual learning (CL) is to train a model that can solve multiple
tasks presented sequentially. Recent CL approaches have achieved strong performance
by leveraging large pre-trained models that generalize well to downstream
tasks. However, such methods lack theoretical guarantees, making them prone to
unexpected failures. Conversely, principled CL approaches often fail to achieve
competitive performance. In this work, we aim to bridge this gap between theory
and practice by designing a simple CL method that is theoretically sound and
highly performant. Specifically, we lift pre-trained features into a higher dimensional
space and formulate an over-parametrized minimum-norm least-squares
problem. We find that the lifted features are highly ill-conditioned, potentially
leading to large training errors (numerical instability) and increased generalization
errors. We address these challenges by continually truncating the singular value
decomposition (SVD) of the lifted features. Our approach, termed TSVD, is stable
with respect to the choice of hyperparameters, can handle hundreds of tasks, and
outperforms state-of-the-art CL methods on multiple datasets. Importantly, our
method satisfies a recurrence relation throughout its continual learning process,
which allows us to prove it maintains small training and generalization errors by
appropriately truncating a fraction of SVD factors. This results in a stable continual
learning method with strong empirical performance and theoretical guarantees.
Code available: https://github.com/liangzu/tsvd.
Continual Learning; Pretrained Models; Overparameterization; Generalization; Random Feature Models
The paper proposes a simple method that delivers stable and strong performance with theoretical guarantees for continual learning with pre-trained models.