Towards Robustness and Diversity: Continual Learning in Dialog Generation with Text-Mixup and Batch Nuclear-Norm Maximization
<|reference_start|>Towards Robustness and Diversity: Continual Learning in Dialog Generation with Text-Mixup and Batch Nuclear-Norm Maximization: In our dynamic world where data arrives in a continuous stream, continual learning enables us to incrementally add new tasks/domains without the need to retrain from scratch. A major challenge in continual learning of language model is catastrophic forgetting, the tendency of models to forget knowledge from previously trained tasks/domains when training on new ones. This paper studies dialog generation under the continual learning setting. We propose a novel method that 1) uses \textit{Text-Mixup} as data augmentation to avoid model overfitting on replay memory and 2) leverages Batch-Nuclear Norm Maximization (BNNM) to alleviate the problem of mode collapse. Experiments on a $37$-domain task-oriented dialog dataset and DailyDialog (a $10$-domain chitchat dataset) demonstrate that our proposed approach outperforms the state-of-the-art in continual learning.<|reference_end|>
@article{wang2024towards,
    title={Towards Robustness and Diversity: Continual Learning in Dialog
  Generation with Text-Mixup and Batch Nuclear-Norm Maximization},
    author={Zihan Wang, Jiayu Xiao, Mengxiang Li, Zhongjiang He, Yongxiang Li,
  Chao Wang, Shuangyong Song},
    journal={arXiv preprint arXiv:2403.10894},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.10894},
    primaryClass={cs.CL}
}