Conditional Language Learning with Context
<|reference_start|>Conditional Language Learning with Context: Language models can learn sophisticated language understanding skills from fitting raw text. They also unselectively learn useless corpus statistics and biases, especially during finetuning on domain-specific corpora. In this paper, we propose a simple modification to causal language modeling called conditional finetuning, which performs language modeling conditioned on a context. We show that a context can "explain away" certain corpus statistics and make the model avoid learning them. In this fashion, conditional finetuning achieves selective learning from a corpus, learning knowledge useful for downstream tasks while avoiding learning useless corpus statistics like topic biases. This selective learning effect leads to less forgetting and better stability-plasticity tradeoff in domain finetuning, potentially benefitting lifelong learning with language models.<|reference_end|>
@article{zhang2024conditional,
    title={Conditional Language Learning with Context},
    author={Xiao Zhang, Miao Li, Ji Wu},
    journal={arXiv preprint arXiv:2406.01976},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01976},
    primaryClass={cs.CL}
}