A Second-Order Perspective on Model Compositionality and Incremental Learning
<|reference_start|>A Second-Order Perspective on Model Compositionality and Incremental Learning: The fine-tuning of deep pre-trained models has revealed compositional properties, with multiple specialized modules that can be arbitrarily composed into a single, multi-task model. However, identifying the conditions that promote compositionality remains an open issue, with recent efforts concentrating mainly on linearized networks. We conduct a theoretical study that attempts to demystify compositionality in standard non-linear networks through the second-order Taylor approximation of the loss function. The proposed formulation highlights the importance of staying within the pre-training basin to achieve composable modules. Moreover, it provides the basis for two dual incremental training algorithms: the one from the perspective of multiple models trained individually, while the other aims to optimize the composed model as a whole. We probe their application in incremental classification tasks and highlight some valuable skills. In fact, the pool of incrementally learned modules not only supports the creation of an effective multi-task model but also enables unlearning and specialization in certain tasks.<|reference_end|>
@article{porrello2024a,
    title={A Second-Order Perspective on Model Compositionality and Incremental
  Learning},
    author={Angelo Porrello and Lorenzo Bonicelli and Pietro Buzzega and Monica
  Millunzi and Simone Calderara and Rita Cucchiara},
    journal={arXiv preprint arXiv:2405.16350},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.16350},
    primaryClass={cs.AI cs.LG}
}