ere are already promising new works not covered there.
Reinforcement Learning from Human Feedback (RLHF)
Preference learning ap-
proaches such as in Ziegler et al. [2019], Stiennon et al. [2020], Ouyang et al. [2022], Bai et al.
[2022a] train a fixed reward model from human preference data, and then use the reward
model to train via reinforcement learning (RL), e.g. via Proximal Policy Optimization (PPO)
[Schulman et al., 2017]. Thus, the reward signal in a certain sense already comes from a model
even in t