Answers.Body: <p>Indeed SQL is very similar to actor-critic method which has a soft Q-function critic network with parameter <span class="math-container">$\theta$</span> and an actor policy network with parameter <span class="math-container">$\phi$</span>, and in fact the paper <em>&quot;Equivalence Between Policy Gradients and Soft Q-Learning&quot;</em> by Schulman et al proves equivalence between the gradient of soft Q-learning within maximum entropy RL framework and natural policies with entropy regularization in its gradient estimator, and further talked about soft Q-learning:</p>
  <blockquote>
  <p>Haarnoja et al. [2017] work in the same setting of soft Q-learning as the current paper, and they are
  concerned with tasks with high-dimensional action spaces, where we would like to learn stochastic
  policies that are multi-modal, and we would like to use Q-functions for which there is no closed-form way of sampling from the Boltzmann distribution <span class="math-container">$π(a|s) ∝ π(a|s) exp(Q(s,a)/τ)$</span>. Hence, they use a method called Stein Variational Gradient Descent to derive a procedure that jointly updates the Q-function and a policy <span class="math-container">$π$</span>, which approximately samples from the Boltzmann distribution—this resembles variational inference, where one makes use of an approximate posterior distribution.</p>
  </blockquote>
  <p>Having said that, in terms of Haarnoja et al's own words about the subtle difference between SQL and actor-critic method in your first reference below, the estimated sampled action from SVGD network is not used directly to affect the next soft Q-function since the critic network uses minibatch experiences from its stored replay memory to update its parameter instead of the usual advantage function. If it's not sampled accurately enough it may not be stable and converge to an optimal final (stochastic) solution.</p>
  <blockquote>
  <p>Although the soft Q-learning algorithm proposed by Haarnoja et al. (2017) has a value function and actor network, it is not a true actor-critic algorithm: the Q-function is estimating the optimal Q-function, and the actor does not directly affect the Q-function except through the data distribution. Hence, Haarnoja et al. (2017) motivates the actor network as an approximate sampler, rather than the actor in an actor-critic algorithm. Crucially, the convergence of this method hinges on how well this sampler approximates the true posterior.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-23T11:08:42.647Z
Answers.Text: Thank you for the detailed answer. If I'm understanding this correctly, is the main point that the sampled actions used for updates to the critic network have to come directly from the actor (on-policy) and not from a replay pool (off-policy) for it to be a true actor-critic method?
Answers.UserDisplayName: frances_farmer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-23T20:58:49.287Z
Answers.Text: Not necessarily, as off-policy DDPG leverages DQN's replay buffer idea to uniformly sample minibatch experiences while being a actor-critic method (usually replay buffer only implies off policy not actor-critic). Here Haarnoja et al simply classifies true actor to be *directly* affect the Q-function like in DDPG via the usual n-step return advantage function, not via another gradient ascent of some performance metric for estimating Q-function in SQL *indirectly*. They also noted its MAP variant connection with DDPG in the same paper, thus it's not that essential clear cut for my point of view.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T10:44:31.637Z
Answers.Text: I see. I think that makes it clearer, thank you!
Answers.UserDisplayName: frances_farmer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-21T00:16:40.397Z
Answers.LastActivityDate: 2023-03-21T00:16:40.397Z
Answers.UserDisplayName: cinch
Body: <p>I've been reading these two papers from Haarnoja et. al.:</p>
  <ol>
  <li>Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement
  Learning with a Stochastic Actor</li>
  <li>Reinforcement Learning with Deep Energy-Based Policies</li>
  </ol>
  <p>As far as I can tell, Soft Q-Learning (SQL) and SAC appear very similar. Why is SQL not considered an Actor-Critic method, even though it has an action value network (critic?) and policy network (actor?)? I also cannot seem to find a consensus on the exact definition of an Actor-Critic method.</p>
  
Tags: <reinforcement-learning><actor-critic-methods><soft-actor-critic><entropy><energy-based-models>
Title: Why is Soft Q Learning not an Actor Critic method?
UserDisplayName: frances_farmer
ThreadText: # Why is Soft Q Learning not an Actor Critic method?
  
  (Asked by: username_0 on 2023-03-12)
  
  I've been reading these two papers from Haarnoja et. al.:
  
  1. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement
     Learning with a Stochastic Actor
  2. Reinforcement Learning with Deep Energy-Based Policies
  
  As far as I can tell, Soft Q-Learning (SQL) and SAC appear very similar. Why is SQL not considered an Actor-Critic method, even though it has an action value network (critic?) and policy network (actor?)? I also cannot seem to find a consensus on the exact definition of an Actor-Critic method.
  
  **Tags:** reinforcement-learning, actor-critic-methods, soft-actor-critic, entropy, energy-based-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-21. Score: 0
  
  Indeed SQL is very similar to actor-critic method which has a soft Q-function critic network with parameter $\theta$ and an actor policy network with parameter $\phi$, and in fact the paper *"Equivalence Between Policy Gradients and Soft Q-Learning"* by Schulman et al proves equivalence between the gradient of soft Q-learning within maximum entropy RL framework and natural policies with entropy regularization in its gradient estimator, and further talked about soft Q-learning:
  
  > Haarnoja et al. [2017] work in the same setting of soft Q-learning as the current paper, and they are
  > concerned with tasks with high-dimensional action spaces, where we would like to learn stochastic
  > policies that are multi-modal, and we would like to use Q-functions for which there is no closed-form way of sampling from the Boltzmann distribution $π(a|s) ∝ π(a|s) exp(Q(s,a)/τ)$. Hence, they use a method called Stein Variational Gradient Descent to derive a procedure that jointly updates the Q-function and a policy $π$, which approximately samples from the Boltzmann distribution—this resembles variational inference, where one makes use of an approximate posterior distribution.
  
  Having said that, in terms of Haarnoja et al's own words about the subtle difference between SQL and actor-critic method in your first reference below, the estimated sampled action from SVGD network is not used directly to affect the next soft Q-function since the critic network uses minibatch experiences from its stored replay memory to update its parameter instead of the usual advantage function. If it's not sampled accurately enough it may not be stable and converge to an optimal final (stochastic) solution.
  
  > Although the soft Q-learning algorithm proposed by Haarnoja et al. (2017) has a value function and actor network, it is not a true actor-critic algorithm: the Q-function is estimating the optimal Q-function, and the actor does not directly affect the Q-function except through the data distribution. Hence, Haarnoja et al. (2017) motivates the actor network as an approximate sampler, rather than the actor in an actor-critic algorithm. Crucially, the convergence of this method hinges on how well this sampler approximates the true posterior.
  
  > **Comments:**
  > **username_0**: Thank you for the detailed answer. If I'm understanding this correctly, is the main point that the sampled actions used for updates to the critic network have to come directly from the actor (on-policy) and not from a replay pool (off-policy) for it to be a true actor-critic method?  
  > **username_1**: Not necessarily, as off-policy DDPG leverages DQN's replay buffer idea to uniformly sample minibatch experiences while being a actor-critic method (usually replay buffer only implies off policy not actor-critic). Here Haarnoja et al simply classifies true actor to be *directly* affect the Q-function like in DDPG via the usual n-step return advantage function, not via another gradient ascent of some performance metric for estimating Q-function in SQL *indirectly*. They also noted its MAP variant connection with DDPG in the same paper, thus it's not that essential clear cut for my point of view.  
  > **username_0**: I see. I think that makes it clearer, thank you!  
  
  ---
  
  