Averaging log-likelihoods in direct alignment
<|reference_start|>Averaging log-likelihoods in direct alignment: To better align Large Language Models (LLMs) with human judgment, Reinforcement Learning from Human Feedback (RLHF) learns a reward model and then optimizes it using regularized RL. Recently, direct alignment methods were introduced to learn such a fine-tuned model directly from a preference dataset without computing a proxy reward function. These methods are built upon contrastive losses involving the log-likelihood of (dis)preferred completions according to the trained model. However, completions have various lengths, and the log-likelihood is not length-invariant. On the other side, the cross-entropy loss used in supervised training is length-invariant, as batches are typically averaged token-wise. To reconcile these approaches, we introduce a principled approach for making direct alignment length-invariant. Formally, we introduce a new averaging operator, to be composed with the optimality operator giving the best policy for the underlying RL problem. It translates into averaging the log-likelihood within the loss. We empirically study the effect of such averaging, observing a trade-off between the length of generations and their scores.<|reference_end|>
@article{grinsztajn2024averaging,
    title={Averaging log-likelihoods in direct alignment},
    author={Nathan Grinsztajn, Yannis Flet-Berliac, Mohammad Gheshlaghi Azar,
  Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash
  Chandak, Olivier Pietquin, Matthieu Geist},
    journal={arXiv preprint arXiv:2406.19188},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.19188},
    primaryClass={cs.LG}
}