[48] Y. Yue, J. Broder, R. Kleinberg, and T. Joachims. The k-armed dueling bandits problem.
Journal of Computer and System Sciences, 78(5):1538–1556, 2012. ISSN 0022-0000. doi: https:
//doi.org/10.1016/j.jcss.2011.12.028. URL https://www.sciencedirect.com/science/
article/pii/S0022000012000281. JCSS Special Issue: Cloud Computing 2011.

[49] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and

G. Irving. Fine-tuning language models from human preferences, 2020.

14

Author Contributions
Direct Preference Optimization