Learning Dynamics of LLM Finetuning
<|reference_start|>Learning Dynamics of LLM Finetuning: Learning dynamics, which describes how the learning of specific training examples influences the model's predictions on other examples, gives us a powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning. In particular, we propose a hypothetical explanation of why specific types of hallucination are strengthened after finetuning, e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique "squeezing effect" to explain a previously observed phenomenon in off-policy direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analysis not only provides a novel perspective of understanding LLM's finetuning but also inspires a simple, effective method to improve alignment performance.<|reference_end|>
@article{ren2024learning,
    title={Learning Dynamics of LLM Finetuning},
    author={Yi Ren, Danica J. Sutherland},
    journal={arXiv preprint arXiv:2407.10490},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.10490},
    primaryClass={cs.LG cs.AI cs.CL}
}