A Fundamental Trade-off in Aligned Language Models and its Relation to Sampling Adaptors
<|reference_start|>A Fundamental Trade-off in Aligned Language Models and its Relation to Sampling Adaptors: The relationship between the quality of a string, as judged by a human reader, and its probability, $p(\boldsymbol{y})$ under a language model undergirds the development of better language models. For example, many popular algorithms for sampling from a language model have been conceived with the goal of manipulating $p(\boldsymbol{y})$ to place higher probability on strings that humans deem of high quality. In this article, we examine the probability--quality relationship in language models explicitly aligned to human preferences, e.g., through reinforcement learning through human feedback. We show that, when sampling corpora from an aligned language model, there exists a trade-off between the strings' average reward and average log-likelihood under the prior language model, i.e., the same model before alignment with human preferences. We provide a formal treatment of this phenomenon and demonstrate how a choice of sampling adaptor allows for a selection of how much likelihood we exchange for the reward.<|reference_end|>
@article{tan2024a,
    title={A Probability--Quality Trade-off in Aligned Language Models and its
  Relation to Sampling Adaptors},
    author={Naaman Tan, Josef Valvoda, Tianyu Liu, Anej Svete, Yanxia Qin, Kan
  Min-Yen, Ryan Cotterell},
    journal={arXiv preprint arXiv:2406.10203},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.10203},
    primaryClass={cs.CL}
}