[26] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder,
P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with
human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh,
editors, Advances in Neural Information Processing Systems, volume 35, pages 27730–27744.
Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/
paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.

[27] R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In
International Conference on Learning Representations, 2018. URL https://openreview.
net/forum?id=HkAClQgA-.

[28] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and

scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
Direct Preference Optimization