Answers.Body: <p>Sample efficient off-policy <a href="https://spinningup.openai.com/en/latest/algorithms/sac.html" rel="nofollow noreferrer">SAC</a> learning is originally proposed for robotics continuous action space RL use cases while its alternative version could handle discrete action spaces.</p>
  <blockquote>
  <p>A central feature of SAC is entropy regularization. The policy is trained to maximize a trade-off between expected return and entropy, a measure of randomness in the policy. This has a close connection to the exploration-exploitation trade-off: increasing entropy results in more exploration, which can accelerate learning later on... An alternate version of SAC, which slightly changes the policy update rule, can be implemented to handle discrete action spaces.</p>
  </blockquote>
  <p>So the reasons why it's not used in LLM RLHF (RL with human feedback) NLP tasks are related to its main purpose which is to deal with continuous action spaces with limited samples due to its inherent entropy maximization explorative nature. LLM RLHF stage doesn't lack human provided ranking samples to inversely learn the reward model (RM), and the extremely stable clipped-objective oriented PPO on-policy learning is well suited to make sure a stable and robust learning process which is apparently desirable in RLHF after supervised fine-tuning (SFT) stage.</p>
  <p>Incidentally to speedup RLHF learning Direct Preference Optimization (DPO), Contrastive Preference Learning (CPL), and rejection sampling, etc have been proposed to replace or enhance PPO. Meta's Llama-2-chat models first use rejection sampling to fine tune with samples of higher reward compared to PPO which updates based on a single sample each time, PPO is used only at its final stage. You can read below references for these alternatives.</p>
  <p>Bai et al. (2022). <em>&quot;Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback&quot;</em><br />
  Hejna et al. (2023). <em>&quot;Contrastive Preference Learning: Learning from Human Feedback without RL&quot;</em><br />
  Rafailov et al. (2023). <em>&quot;Direct Preference Optimization: Your Language Model is Secretly a Reward Model&quot;</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-09T21:55:59.683Z
Answers.LastActivityDate: 2024-08-09T23:39:57.397Z
Answers.LastEditDate: 2024-08-09T23:39:57.397Z
Answers.UserDisplayName: cinch
Body: <p>Most language model is using online PPO or offline DPO type algorithm. Can we use soft actor critic RL to do alignment work? Any publication related can be recommended?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-09T21:27:11.433Z
Comments.Text: tokens are discrete, SAC is for continuous control... you do you
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-11T10:02:55.840Z
Comments.Text: In principle you can use [Discrete SAC](https://arxiv.org/pdf/1910.07207), Rainbow, IQN, etc, all of which attain performance comparable to PPO (at least in common RL benchmarks) - probably, PPO is favored because it is designed to be parallelizable over many nodes
Comments.UserDisplayName: Luca Anzalone
Tags: <proximal-policy-optimization><fine-tuning><off-policy-methods><on-policy-methods><rlhf>
Title: can we apply soft actor critic to language model preference optimization?
UserDisplayName: Chaoshun Hu
ThreadText: # can we apply soft actor critic to language model preference optimization?
  
  (Asked by: username_0 on 2024-08-09)
  
  Most language model is using online PPO or offline DPO type algorithm. Can we use soft actor critic RL to do alignment work? Any publication related can be recommended?
  
  **Tags:** proximal-policy-optimization, fine-tuning, off-policy-methods, on-policy-methods, rlhf
  > **Comments:**
  > **username_1**: tokens are discrete, SAC is for continuous control... you do you  
  > **username_2**: In principle you can use [Discrete SAC](https://arxiv.org/pdf/1910.07207), Rainbow, IQN, etc, all of which attain performance comparable to PPO (at least in common RL benchmarks) - probably, PPO is favored because it is designed to be parallelizable over many nodes  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2024-08-09. Score: 0 | Accepted answer
  
  Sample efficient off-policy [SAC](https://spinningup.openai.com/en/latest/algorithms/sac.html) learning is originally proposed for robotics continuous action space RL use cases while its alternative version could handle discrete action spaces.
  
  > A central feature of SAC is entropy regularization. The policy is trained to maximize a trade-off between expected return and entropy, a measure of randomness in the policy. This has a close connection to the exploration-exploitation trade-off: increasing entropy results in more exploration, which can accelerate learning later on... An alternate version of SAC, which slightly changes the policy update rule, can be implemented to handle discrete action spaces.
  
  So the reasons why it's not used in LLM RLHF (RL with human feedback) NLP tasks are related to its main purpose which is to deal with continuous action spaces with limited samples due to its inherent entropy maximization explorative nature. LLM RLHF stage doesn't lack human provided ranking samples to inversely learn the reward model (RM), and the extremely stable clipped-objective oriented PPO on-policy learning is well suited to make sure a stable and robust learning process which is apparently desirable in RLHF after supervised fine-tuning (SFT) stage.
  
  Incidentally to speedup RLHF learning Direct Preference Optimization (DPO), Contrastive Preference Learning (CPL), and rejection sampling, etc have been proposed to replace or enhance PPO. Meta's Llama-2-chat models first use rejection sampling to fine tune with samples of higher reward compared to PPO which updates based on a single sample each time, PPO is used only at its final stage. You can read below references for these alternatives.
  
  Bai et al. (2022). *"Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback"*  
  Hejna et al. (2023). *"Contrastive Preference Learning: Learning from Human Feedback without RL"*  
  Rafailov et al. (2023). *"Direct Preference Optimization: Your Language Model is Secretly a Reward Model"*
  
  
  ---
  
  