Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression
<|reference_start|>Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression: Humans can retain old knowledge while learning new information, but Large Language Models (LLMs) often suffer from catastrophic forgetting when post-pretrained or supervised fine-tuned (SFT) on domain-specific data. Moreover, for Multimodal Large Language Models (MLLMs) which are composed of the LLM base and visual projector (e.g. LLaVA), a significant decline in performance on language benchmarks was observed compared to their single-modality counterparts. To address these challenges, we introduce a novel model-agnostic self-decompression method, Tree Generation (TG), that decompresses knowledge within LLMs into the training corpus. This paper focuses on TG-SFT, which can synthetically generate SFT data for the instruction tuning steps. By incorporating the dumped corpus during SFT for MLLMs, we significantly reduce the forgetting problem.<|reference_end|>
@article{zhang2024preserving,
    title={Preserving Knowledge in Large Language Model with Model-Agnostic
  Self-Decompression},
    author={Zilun Zhang, Yutao Sun, Tiancheng Zhao, Leigang Sha, Ruochen Xu,
  Kyusong Lee, Jianwei Yin},
    journal={arXiv preprint arXiv:2406.11354},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11354},
    primaryClass={cs.CL cs.AI cs.CV}
}