Answers.Body: <p>PPO belongs to both category 2 and 3:</p>
  <ul>
  <li>It has an <em>actor-critic structure</em>, meaning that it learns the optimal policy via the actor network, and the value function via the critic network.</li>
  <li>Moreover, differently from SAC for example, the actor is learned via policy gradient (not Q-learning as in SAC). Therefore it is also policy-based.</li>
  </ul>
  <p><strong>Update:</strong> <em>defining value-based, policy-based, and actor-critic agents</em> - I'll provide an intuitive rather than technical definition.</p>
  <p>In general, a reinforcement learning agent can be equipped (let's say) with one or more components such as a (figure from <a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf" rel="nofollow noreferrer">David Silver's lecture 1</a>):</p>
  <p><a href="https://i.sstatic.net/mhDsn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mhDsn.png" alt="enter image description here" /></a></p>
  <ul>
  <li><em>Policy</em>, representing the agent's behavior function typically mapping states to actions;</li>
  <li><em>Value function</em>, i.e., a function or model that predicts the expected future reward;</li>
  <li><em>Model</em> of the environment that predicts its next state.</li>
  </ul>
  <p>Now, not all these components are required (e.g., the env's model) and some of them can be implicit (like the policy, which can be extracted from the value function.) According to which is the &quot;main&quot; component, we can categorize agents into:</p>
  <ul>
  <li><strong>Value-based</strong>, which need only to learn a value function (either the state-value or state-action one). For such agents learning the value function is enough since the policy can be &quot;extracted&quot; from it. For example, from a Q-function is very easy to derive a policy that predicts the action associated with the higher state-action value. You can also have a policy from the value function itself, but a <em>one-step model</em> of the environment is needed since the future is involved. <em>TD-learning</em> and <em>Q-learning</em> are both examples of value-based agents. Moreover, we consider value functions the state-value, <span class="math-container">$V(s)$</span>, action-value (or state-action) function, <span class="math-container">$Q(s,a)$</span>, and the advantage function, <span class="math-container">$A(s,a)$</span>.</li>
  <li><strong>Policy-based</strong> methods that learn directly a policy instead of first approximating a value function. The policy can be deterministic (i.e., a mapping) or stochastic (a probability distribution.) In both cases, a popular approach is to learn the policy's parameters by following the (deterministic) <em>policy gradient</em>, which induces a further specialization of this class of agents. Another way is to use evolutionary strategies (ES), or optimization algorithms. Classical REINFORCE and DPG are example policy-based agents.</li>
  <li><strong>Actor-critic</strong> agents, which merge both value- and policy-based methods by learning two models: the actor (policy) and critic (value function). Now the critic can be found having various roles, from variance reduction serving as a baseline (e.g., in A2C, PPO) but also as a way to learn a policy (e.g., DDPG, SAC, TD3).  The important thing is that we don't use the critic to derive an implicit policy.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-05T08:39:44.167Z
Answers.Text: Thanks for your answer Luca. So this means that there are also mixtures of the 3 basic RL algorithms (value-based methods, policy-based methods, and actor-critique-based methods)? I thought that every RL algorithm can only be assigned to one single category. Further is REINFORCE only a policy-based method and A2C only a actor-critique-based method or also mixtures?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-05T13:16:33.467Z
Answers.Text: But aren't actor-critic methods by definition also policy-based? I think that's not clear from this answer. Maybe a definition of policy-based and actor-critic methods is required here.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-06T10:36:52.120Z
Answers.Text: @nbro: Thanks for your comment. Actually I don't know. As far as I understand there are those 3 classes for RL algorithms. Apparently in REINFORCE there is no actor-critique structure while in PPO there is and both are policy-based methods. Actor-critique based methods combine value-based and policy-based methods as far as I understand
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-07T15:22:10.213Z
Answers.Text: @PeterBe In classical REINFORCE there is *no* actor-critic structure, since the state's value is estimated directly by the empirical return (or rewards-to-go if you prefer.) If you then introduce a *baseline*, for variance reduction, you may turn it into actor-critic if the baseline is a learned model of the value function.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-07T15:28:12.530Z
Answers.Text: @nbro In general the definition is a little bit shaded. E.g., Silver's [lecture 1](https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf) (see page 36) defines the actor-critic at the intersection of value- and policy-based methods. Instead, the spinning up RL's [taxonomy](https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html) lists both PPO, and A2C/A3C under "policy optimization", as a wider more general family than the policy gradient.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-08T14:06:53.337Z
Answers.Text: @LucaAnzalone: Thanks Luca for your comment and effort. But would you generally also classify RL algorithms into value-based, policy-based and actor-critic based methods?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-11T09:29:03.390Z
Answers.Text: @LucaAnzalone: Thanks for your comment. Any comments on my last comment? I'll highly appreciate every further comment from you.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-11T17:01:18.773Z
Answers.Text: @PeterBe have a read at my latest update. Hope it helps. :)
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-18T13:42:50.433Z
Answers.Text: Thanks a lot Luca for your great answer. It is definitely helpful.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-29T17:33:19.317Z
Answers.Text: @LucaAnzalone: Thanks for your answer Luca. It's me again. In your very first sentence you write "PPO belongs to both category 2 and 3". But that is not true. PPO belongs to the Actor-Critic methods as far as I understand as it combines the value-funciton approach with the policy optimization approach
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-29T19:17:39.053Z
Answers.Text: @PeterBe True: PPO is best classified as actor-critic, indeed, but it's also a policy-based method. I mean, this is a broader class than Actor-critic, which is more specific
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-04T20:01:39.227Z
Answers.LastActivityDate: 2024-01-11T17:00:49.067Z
Answers.LastEditDate: 2024-01-11T17:00:49.067Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>as far as i understand there are 3 categories of Reinforcement algorithms:</p>
  <ol>
  <li>Value-based methods (like DQN or Sarsa)</li>
  <li>Policy-based methods (like REINFORCE)</li>
  <li>Actor-critic-based methods (like A2C)</li>
  </ol>
  <p>To which of those categories does PPO (Proximal Policy Optimization) belong to? As the name suggest it is based on policy optimization but I think it also uses an actor-critic-based structure?</p>
  
Tags: <reinforcement-learning><actor-critic-methods><proximal-policy-optimization>
Title: Is PPO a policy-based method or an actor-critique-based method?
UserDisplayName: PeterBe
ThreadText: # Is PPO a policy-based method or an actor-critique-based method?
  
  (Asked by: username_0 on 2024-01-04)
  
  as far as i understand there are 3 categories of Reinforcement algorithms:
  
  1. Value-based methods (like DQN or Sarsa)
  2. Policy-based methods (like REINFORCE)
  3. Actor-critic-based methods (like A2C)
  
  To which of those categories does PPO (Proximal Policy Optimization) belong to? As the name suggest it is based on policy optimization but I think it also uses an actor-critic-based structure?
  
  **Tags:** reinforcement-learning, actor-critic-methods, proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-04. Score: 2 | Accepted answer
  
  PPO belongs to both category 2 and 3:
  
  * It has an *actor-critic structure*, meaning that it learns the optimal policy via the actor network, and the value function via the critic network.
  * Moreover, differently from SAC for example, the actor is learned via policy gradient (not Q-learning as in SAC). Therefore it is also policy-based.
  
  **Update:** *defining value-based, policy-based, and actor-critic agents* - I'll provide an intuitive rather than technical definition.
  
  In general, a reinforcement learning agent can be equipped (let's say) with one or more components such as a (figure from [David Silver's lecture 1](https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf)):
  
  [![enter image description here](https://i.sstatic.net/mhDsn.png)](https://i.sstatic.net/mhDsn.png)
  
  * *Policy*, representing the agent's behavior function typically mapping states to actions;
  * *Value function*, i.e., a function or model that predicts the expected future reward;
  * *Model* of the environment that predicts its next state.
  
  Now, not all these components are required (e.g., the env's model) and some of them can be implicit (like the policy, which can be extracted from the value function.) According to which is the "main" component, we can categorize agents into:
  
  * **Value-based**, which need only to learn a value function (either the state-value or state-action one). For such agents learning the value function is enough since the policy can be "extracted" from it. For example, from a Q-function is very easy to derive a policy that predicts the action associated with the higher state-action value. You can also have a policy from the value function itself, but a *one-step model* of the environment is needed since the future is involved. *TD-learning* and *Q-learning* are both examples of value-based agents. Moreover, we consider value functions the state-value, $V(s)$, action-value (or state-action) function, $Q(s,a)$, and the advantage function, $A(s,a)$.
  * **Policy-based** methods that learn directly a policy instead of first approximating a value function. The policy can be deterministic (i.e., a mapping) or stochastic (a probability distribution.) In both cases, a popular approach is to learn the policy's parameters by following the (deterministic) *policy gradient*, which induces a further specialization of this class of agents. Another way is to use evolutionary strategies (ES), or optimization algorithms. Classical REINFORCE and DPG are example policy-based agents.
  * **Actor-critic** agents, which merge both value- and policy-based methods by learning two models: the actor (policy) and critic (value function). Now the critic can be found having various roles, from variance reduction serving as a baseline (e.g., in A2C, PPO) but also as a way to learn a policy (e.g., DDPG, SAC, TD3). The important thing is that we don't use the critic to derive an implicit policy.
  
  > **Comments:**
  > **username_0**: Thanks for your answer username_1. So this means that there are also mixtures of the 3 basic RL algorithms (value-based methods, policy-based methods, and actor-critique-based methods)? I thought that every RL algorithm can only be assigned to one single category. Further is REINFORCE only a policy-based method and A2C only a actor-critique-based method or also mixtures?  
  > **username_2**: But aren't actor-critic methods by definition also policy-based? I think that's not clear from this answer. Maybe a definition of policy-based and actor-critic methods is required here.  
  > **username_0**: @username_2: Thanks for your comment. Actually I don't know. As far as I understand there are those 3 classes for RL algorithms. Apparently in REINFORCE there is no actor-critique structure while in PPO there is and both are policy-based methods. Actor-critique based methods combine value-based and policy-based methods as far as I understand  
  > **username_1**: @username_0 In classical REINFORCE there is *no* actor-critic structure, since the state's value is estimated directly by the empirical return (or rewards-to-go if you prefer.) If you then introduce a *baseline*, for variance reduction, you may turn it into actor-critic if the baseline is a learned model of the value function.  
  > **username_1**: @username_2 In general the definition is a little bit shaded. E.g., Silver's [lecture 1](https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf) (see page 36) defines the actor-critic at the intersection of value- and policy-based methods. Instead, the spinning up RL's [taxonomy](https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html) lists both PPO, and A2C/A3C under "policy optimization", as a wider more general family than the policy gradient.  
  > **username_0**: @username_1: Thanks username_1 for your comment and effort. But would you generally also classify RL algorithms into value-based, policy-based and actor-critic based methods?  
  > **username_0**: @username_1: Thanks for your comment. Any comments on my last comment? I'll highly appreciate every further comment from you.  
  > **username_1**: @username_0 have a read at my latest update. Hope it helps. :)  
  > **username_0**: Thanks a lot username_1 for your great answer. It is definitely helpful.  
  > **username_0**: @username_1: Thanks for your answer username_1. It's me again. In your very first sentence you write "PPO belongs to both category 2 and 3". But that is not true. PPO belongs to the Actor-Critic methods as far as I understand as it combines the value-funciton approach with the policy optimization approach  
  > **username_1**: @username_0 True: PPO is best classified as actor-critic, indeed, but it's also a policy-based method. I mean, this is a broader class than Actor-critic, which is more specific  
  
  ---
  
  