5.2

Instability of Actor-Critic Algorithms

We can also use our framework to diagnose instabilities with standard actor-critic algorithms used
for the RLHF, such as PPO. We follow the RLHF pipeline and focus on the RL fine-tuning step
outlined in Section 3. We can draw connections to the control as inference framework [20] for the
constrained RL problem outlined in 3. We assume a parameterized model πθ(y | x) and minimize
DKL[πθ(y|x) || π∗(y | x)] where π∗ is the optimal policy from Eq. 7 induced by the reward function
rϕ(y, x). With some algebra this leads to the optimization objective:

(cid:88)

y

(cid:123)(cid:122)

f (rϕ,πref,β)

(cid:18) 1

β

(cid:19)
(cid:125)

− β log

(cid:124)

πθ(y | x)
πref(y | x)

(cid:123)(cid:122)

KL

(cid:125)

(cid:21)

(10)

Eπθ(y|x)

max
πθ

rϕ(x, y) − β log

πref(y | x) exp

rϕ(x, y)

(cid:20)

(cid:124)
Direct Preference Optimization