Training with feedback. Training with feedback to improve the outputs of large language models,
both in terms of correctness and alignment with human preferences, is an active research direction
nowadays [70, 30, 18, 3]. One popular technique is reinforcement learning from human feedback
(RLHF) [70, 41], and RLHF-trained models have demonstrated the ability to avoid harmful outputs
when instructed to do so in the prompt [18]. Constitutional AI [3] introduces another path toward
training harmless models, where they use the pretrained model itself to create automated feedback for
both supervised learning and RLHF: for the former, a set of principles are used to guide a language
model in creating revisions of its own responses that it is then trained on, and for the latter the same
principles are used to prompt a separate model for the feedback needed for RLHF.
Another line of work trains the language model to reﬁne the initial model outputs based on external
Teaching Large Language Models to Self-Debug