I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses
<|reference_start|>I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses: This paper explores an intriguing observation: fine-tuning a large language model (LLM) with responses generated by a LLM often yields better results than using responses generated by humans, particularly in reasoning tasks. We conduct an in-depth investigation to understand why this occurs. Contrary to the common belief that these instances is due to the more detailed nature of LLM-generated content, our study identifies another contributing factor: an LLM is inherently more "familiar" with LLM generated responses. This familiarity is evidenced by lower perplexity before fine-tuning. We design a series of experiments to understand the impact of the "familiarity" and our conclusion reveals that this "familiarity" significantly impacts learning performance. Training with LLM-generated responses not only enhances performance but also helps maintain the model's capabilities in other reasoning tasks after fine-tuning on a specific task.<|reference_end|>
@article{ren2024i,
    title={I Learn Better If You Speak My Language: Understanding the Superior
  Performance of Fine-Tuning Large Language Models with LLM-Generated Responses},
    author={Xuan Ren and Biao Wu and Lingqiao Liu},
    journal={arXiv preprint arXiv:2402.11192},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.11192},
    primaryClass={cs.CL cs.AI}
}