Hyperparameter Selection in Continual Learning
<|reference_start|>Hyperparameter Selection in Continual Learning: In continual learning (CL) -- where a learner trains on a stream of data -- standard hyperparameter optimisation (HPO) cannot be applied, as a learner does not have access to all of the data at the same time. This has prompted the development of CL-specific HPO frameworks. The most popular way to tune hyperparameters in CL is to repeatedly train over the whole data stream with different hyperparameter settings. However, this end-of-training HPO is unrealistic as in practice a learner can only see the stream once. Hence, there is an open question: what HPO framework should a practitioner use for a CL problem in reality? This paper answers this question by evaluating several realistic HPO frameworks. We find that all the HPO frameworks considered, including end-of-training HPO, perform similarly. We therefore advocate using the realistic and most computationally efficient method: fitting the hyperparameters on the first task and then fixing them throughout training.<|reference_end|>
@article{lee2024hyperparameter,
    title={Hyperparameter Selection in Continual Learning},
    author={Thomas L. Lee, Sigrid Passano Hellan, Linus Ericsson, Elliot J.
  Crowley and Amos Storkey},
    journal={arXiv preprint arXiv:2404.06466},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.06466},
    primaryClass={cs.LG stat.ML}
}