reward model overoptimization, 2022. 2

[13] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming
Song, Matthieu Geist, and Stefano Ermon. Iq-learn: Inverse
soft-q learning for imitation. Neural Information Processing
Systems, 2021. 5

[14] Amelia Glaese, Nat McAleese, Maja Tr˛ebacz,

John
Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh,
Laura Weidinger, Martin Chadwick, Phoebe Thacker,
Lucy Campbell-Gillingham,
Jonathan Uesato, Po-Sen
Huang, Ramona Comanescu, Fan Yang, Abigail See,
Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz,
Jaume Sanchez Elias, Richard Green, Soˇna Mokrá, Nicholas
Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason
Gabriel, William Isaac, John Mellor, Demis Hassabis, Ko-
ray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irv-
ing.
Improving alignment of dialogue agents via targeted
human judgements, 2022. 2
DiffusionModelAlignmentUsing Direct Preference Optimization