Language-Independent Representations Improve Zero-Shot Summarization
<|reference_start|>Language-Independent Representations Improve Zero-Shot Summarization: Finetuning pretrained models on downstream generation tasks often leads to catastrophic forgetting in zero-shot conditions. In this work, we focus on summarization and tackle the problem through the lens of language-independent representations. After training on monolingual summarization, we perform zero-shot transfer to new languages or language pairs. We first show naively finetuned models are highly language-specific in both output behavior and internal representations, resulting in poor zero-shot performance. Next, we propose query-key (QK) finetuning to decouple task-specific knowledge from the pretrained language generation abilities. Then, after showing downsides of the standard adversarial language classifier, we propose a balanced variant that more directly enforces language-agnostic representations. Moreover, our qualitative analyses show removing source language identity correlates to zero-shot summarization performance. Our code is openly available.<|reference_end|>
@article{solovyev2024language-independent,
    title={Language-Independent Representations Improve Zero-Shot Summarization},
    author={Vladimir Solovyev, Danni Liu, Jan Niehues},
    journal={arXiv preprint arXiv:2404.05720},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.05720},
    primaryClass={cs.CL cs.AI}
}