Less for More: Enhancing Preference Learning in Generative Language Models with Automated Self-Curation of Training Corpora
<|reference_start|>Less for More: Enhancing Preference Learning in Generative Language Models with Automated Self-Curation of Training Corpora: Ambiguity in language presents challenges in developing more enhanced language models, particularly in preference learning, where variability among annotators results in inconsistently annotated datasets used for model alignment. To address this issue, we introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on these datasets. Our method enhances preference learning by automatically detecting and removing ambiguous annotations within the dataset. The proposed approach is validated through extensive experiments, demonstrating a marked improvement in performance across various instruction-following tasks. Our work provides a straightforward and reliable method to overcome annotation inconsistencies, serving as an initial step towards the development of more advanced preference learning techniques.<|reference_end|>
@article{lee2024less,
    title={Less for More: Enhancing Preference Learning in Generative Language
  Models with Automated Self-Curation of Training Corpora},
    author={JoonHo Lee, JuYoun Son, Juree Seok, Wooseok Jang and Yeong-Dae Kwon},
    journal={arXiv preprint arXiv:2408.12799},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.12799},
    primaryClass={cs.CL cs.AI}
}