[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda
Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna
Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol
Chen, Catherine Olsson, Christopher Olah, Danny Her-
nandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-
Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey
Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite,
Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas
Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby,
Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec,
Sheer El Showk, Stanislav Fort, Tamera Lanham, Timo-
thy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan
Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann,
Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom
Brown, and Jared Kaplan. Constitutional ai: Harmlessness
from ai feedback, 2022. 2, 3
DiffusionModelAlignmentUsing Direct Preference Optimization