Answers.Body: <blockquote>
    <p>Keeping this taxonomy intact for model-based Dynamic programming algorithms, I would argue that value iteration is a Actor only approach, and policy iteration is a Actor-Critic approach. However, not many people discuss the term Actor-Critic when referring to Policy Iteration. How come?</p>
  </blockquote>
  
  <p>Both policy iteration and value iteration are value-based approaches. The policy in policy iteration is either arbitrary or derived from a value table. It is not modelled separately.</p>
  
  <p>To count as an Actor, the policy function needs to modelled directly as a <em>parametric</em> function of the state, not indirectly via a value assessment. You cannot use policy gradient methods to adjust an Actor's policy function unless it is possible to derive the gradient of the policy function with respect to parameters that control the relationship bewteen state and action. An Actor policy might be noted as <span class="math-container">$\pi(a|s,\theta)$</span> and the parameters <span class="math-container">$\theta$</span> are what make it possible to learn improvements.</p>
  
  <p>Policy iteration often <em>generates</em> an explicit policy, from the current value estimates. This is not a representation that can be directly manipulated, instead it is a consequence of measuring values, and there are no parameters that can be learned. Therefore the policy seen in policy iteration cannot be used as an actor in Actor-Critic or related methods.</p>
  
  <p>Another way to state this is that the policy and value functions in DP are not separate enough to be considered as an actor/critic pair. Instead they are both views of the same measurement, with the value function being closer to raw measurements and policy being a mapping of the value function to policy space.</p>
  
  <blockquote>
    <p>Also, I am not familiar with any model-based/dynamic programming like actor only approaches? Do these exist? If not, what prevents this from happening?</p>
  </blockquote>
  
  <p>The main difference between model-based dynamic programming and model-free methods like Q-learning, or SARSA, is that the dynamic programming methods directly use the full distribution model (which can be expressed as <span class="math-container">$p(r, s'|s,a)$</span>) to calculate expected bootstrapped returns.</p>
  
  <p>There is nothing in principle stopping you substituting expected returns calculated in this way into REINFORCE or Actor-Critic methods. However, it may be computationally hard to do so - these methods are often chosen when action space is large for instance. </p>
  
  <p>Basic REINFORCE using model-based expectations would be especially hard as you need an expected value calculated over all possible trajectories from each starting state - if you are going to expand the tree of all possible results to that degree, then a simple tree search algorithm would perform better, and the algorithm then resolves to a one-off planning exhaustive tree search.</p>
  
  <p>Actor-Critic using dynamic programming methods for the Critic should be viable, and I expect you could find examples of it being done in some situations. It may work well for some card or board games, if the combined action space and state space is not too large - it would behave a little like using Expected SARSA for the Critic component, except also run expectations over the state transition dynamics (whilst Expected SARSA only runs expectations over policy). You could vary the depth of this too, getting better estimates theoretically at the expense of extra computation (potentially a lot of extra computation if there is a large branching factor)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T11:49:36.267Z
Answers.Text: I think that the confusion that policy iteration is an actor-critic method lies in the fact that in actor-critic methods you use the value function to guide the search for the policy. In policy iteration, you actually use the value function to derive the policy too. I don't think it's fully clear from your answer why policy iteration couldn't be considered an actor-critic method.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T12:14:09.257Z
Answers.Text: Thanks for your elaborate answer. in policy iteration however; the policy function is usually directly represented, right? Sure, it is a quite trivial step to do so (policy improvement), but it is done nonetheless. For example, see the algorithm shown in Sutton and Barto: http://www.incompleteideas.net/sutton/book/first/4/node4.html#SECTION00130000000000000000
Answers.UserDisplayName: dan888
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T12:18:27.613Z
Answers.Text: About how REINFORCE would translate into a DP method, wouldn't this resolve to some form of policy iteration? 
  1. You generate a "trajectory" and calculate its return. In a model-based sense however; this trajectory updates the value based on all possible successor states until a terminal state, somewhat emulating a step of policy evaluation.
  2. You improve the policy based on the gradient signal from this evaluation -> policy improvement
  Sure, doing it this way doesn't intuitively sound very smart. Why not update all states in a normal Policy evaluation manner?
Answers.UserDisplayName: dan888
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T12:23:53.773Z
Answers.Text: @NeilSlater Okay, so the key difference between PI and Actor-Critic would be that PI generates a policy in every step, whereas AC modifies/adapts an older policy in every step, right? Do you happen to know of any sources that would argue similarly?
Answers.UserDisplayName: dan888
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T12:25:35.547Z
Answers.Text: @NeilSlater Exactly, I'm on the same page about REINFORCE, I just worded it  awkwardly. Generating a single trajectory isn't really possible in a DP sense as you are taking the expected value, resulting in effectively generating all the trajectories. Thanks for that!
Answers.UserDisplayName: dan888
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T12:28:57.993Z
Answers.Text: @DaniëlWillemsen I don't know of any other sources that would try to argue the case that DP is not a policy-based method. It is just a basic fact that DP is the core of value-based methods. Most RL taxonomy diagrams will show this - DP methods represent the baseline of "Generalise Policy Iteration" approaches, and don't relate to policy gradients at all
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T13:42:11.953Z
Answers.Text: @nbro: I have tried to make that clearer, the difference being the policy derviation method in DP is not a direct parametric function over state that can be adjusted by changing those parameters. I guess another way to state this is that the Actor and Critic are not separate enough, they are just transformations of the same measurement.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T11:34:24.517Z
Answers.LastActivityDate: 2020-05-13T13:41:00.513Z
Answers.LastEditDate: 2020-05-13T13:41:00.513Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <blockquote>
  <p>policy iteration is an actor-critic approach.</p>
  </blockquote>
  <p>This is a very insightful observation and I would agree with it. You can find similar statement in this <a href="https://www.cs.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Informatik/Dialog_Systems_and_Machine_Learning/Lectures_RL/L5.pdf" rel="nofollow noreferrer">lecture</a> and this NeurIPS <a href="https://proceedings.neurips.cc/paper/2013/hash/2dace78f80bc92e6d7493423d729448e-Abstract.html" rel="nofollow noreferrer">paper</a>.</p>
  <p>The actor's role is to perform policy improvement and the critic's role is to perform policy evaluation. This helps explain its advantage over value-only approaches, as PI converges faster than VI although each iteration is more expensive.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-20T06:45:20.160Z
Answers.LastActivityDate: 2022-12-20T06:45:20.160Z
Answers.UserDisplayName: MLMLH
Body: <p>One way of understanding the difference between value function approaches, policy approaches and actor-critic approaches in reinforcement learning is the following:</p>
  
  <ul>
  <li>A critic explicitly models a value function for a policy.</li>
  <li>An actor explicitly models a policy.</li>
  </ul>
  
  <p>Value function approaches, such as Q-learning, only keep track of a value function, and the policy is directly derived from that (e.g. greedily or epsilon-greedily). Therefore, these approaches can be classified as a "critic-only" approach. </p>
  
  <p>Some policy search/gradient approaches, such as REINFORCE, only use a policy representation, therefore, I would argue that this approach can be classified as an "actor-only" approach.</p>
  
  <p>Of course, many policy search/gradient approaches also use value models in addition to a policy model. These algorithms are commonly referred to as "actor-critic" approaches (well-known ones are A2C / A3C).</p>
  
  <p>Keeping this taxonomy intact for model-based dynamic programming algorithms, I would argue that <em>value iteration</em> is an actor-only approach, and <em>policy iteration</em> is an actor-critic approach. However, not many people discuss the term actor-critic when referring to policy iteration. How come?</p>
  
  <p>Also, I am not familiar with any model-based/dynamic programming like actor only approaches? Do these exist? If not, what prevents this from happening?</p>
  
Tags: <reinforcement-learning><definitions><actor-critic-methods><value-iteration><policy-iteration>
Title: Would you categorize policy iteration as an actor-critic reinforcement learning approach?
UserDisplayName: dan888
ThreadText: # Would you categorize policy iteration as an actor-critic reinforcement learning approach?
  
  (Asked by: username_0 on 2020-05-13)
  
  One way of understanding the difference between value function approaches, policy approaches and actor-critic approaches in reinforcement learning is the following:
  
  * A critic explicitly models a value function for a policy.
  * An actor explicitly models a policy.
  
  Value function approaches, such as Q-learning, only keep track of a value function, and the policy is directly derived from that (e.g. greedily or epsilon-greedily). Therefore, these approaches can be classified as a "critic-only" approach.
  
  Some policy search/gradient approaches, such as REINFORCE, only use a policy representation, therefore, I would argue that this approach can be classified as an "actor-only" approach.
  
  Of course, many policy search/gradient approaches also use value models in addition to a policy model. These algorithms are commonly referred to as "actor-critic" approaches (well-known ones are A2C / A3C).
  
  Keeping this taxonomy intact for model-based dynamic programming algorithms, I would argue that *value iteration* is an actor-only approach, and *policy iteration* is an actor-critic approach. However, not many people discuss the term actor-critic when referring to policy iteration. How come?
  
  Also, I am not familiar with any model-based/dynamic programming like actor only approaches? Do these exist? If not, what prevents this from happening?
  
  **Tags:** reinforcement-learning, definitions, actor-critic-methods, value-iteration, policy-iteration
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-05-13. Score: 3
  
  > Keeping this taxonomy intact for model-based Dynamic programming algorithms, I would argue that value iteration is a Actor only approach, and policy iteration is a Actor-Critic approach. However, not many people discuss the term Actor-Critic when referring to Policy Iteration. How come?
  
  Both policy iteration and value iteration are value-based approaches. The policy in policy iteration is either arbitrary or derived from a value table. It is not modelled separately.
  
  To count as an Actor, the policy function needs to modelled directly as a *parametric* function of the state, not indirectly via a value assessment. You cannot use policy gradient methods to adjust an Actor's policy function unless it is possible to derive the gradient of the policy function with respect to parameters that control the relationship bewteen state and action. An Actor policy might be noted as $\pi(a|s,\theta)$ and the parameters $\theta$ are what make it possible to learn improvements.
  
  Policy iteration often *generates* an explicit policy, from the current value estimates. This is not a representation that can be directly manipulated, instead it is a consequence of measuring values, and there are no parameters that can be learned. Therefore the policy seen in policy iteration cannot be used as an actor in Actor-Critic or related methods.
  
  Another way to state this is that the policy and value functions in DP are not separate enough to be considered as an actor/critic pair. Instead they are both views of the same measurement, with the value function being closer to raw measurements and policy being a mapping of the value function to policy space.
  
  > Also, I am not familiar with any model-based/dynamic programming like actor only approaches? Do these exist? If not, what prevents this from happening?
  
  The main difference between model-based dynamic programming and model-free methods like Q-learning, or SARSA, is that the dynamic programming methods directly use the full distribution model (which can be expressed as $p(r, s'|s,a)$) to calculate expected bootstrapped returns.
  
  There is nothing in principle stopping you substituting expected returns calculated in this way into REINFORCE or Actor-Critic methods. However, it may be computationally hard to do so - these methods are often chosen when action space is large for instance.
  
  Basic REINFORCE using model-based expectations would be especially hard as you need an expected value calculated over all possible trajectories from each starting state - if you are going to expand the tree of all possible results to that degree, then a simple tree search algorithm would perform better, and the algorithm then resolves to a one-off planning exhaustive tree search.
  
  Actor-Critic using dynamic programming methods for the Critic should be viable, and I expect you could find examples of it being done in some situations. It may work well for some card or board games, if the combined action space and state space is not too large - it would behave a little like using Expected SARSA for the Critic component, except also run expectations over the state transition dynamics (whilst Expected SARSA only runs expectations over policy). You could vary the depth of this too, getting better estimates theoretically at the expense of extra computation (potentially a lot of extra computation if there is a large branching factor)
  
  > **Comments:**
  > **username_2**: I think that the confusion that policy iteration is an actor-critic method lies in the fact that in actor-critic methods you use the value function to guide the search for the policy. In policy iteration, you actually use the value function to derive the policy too. I don't think it's fully clear from your answer why policy iteration couldn't be considered an actor-critic method.  
  > **username_0**: Thanks for your elaborate answer. in policy iteration however; the policy function is usually directly represented, right? Sure, it is a quite trivial step to do so (policy improvement), but it is done nonetheless. For example, see the algorithm shown in Sutton and Barto: http://www.incompleteideas.net/sutton/book/first/4/node4.html#SECTION00130000000000000000  
  > **username_0**: About how REINFORCE would translate into a DP method, wouldn't this resolve to some form of policy iteration?  1. You generate a "trajectory" and calculate its return. In a model-based sense however; this trajectory updates the value based on all possible successor states until a terminal state, somewhat emulating a step of policy evaluation. 2. You improve the policy based on the gradient signal from this evaluation -> policy improvement Sure, doing it this way doesn't intuitively sound very smart. Why not update all states in a normal Policy evaluation manner?  
  > **username_0**: @NeilSlater Okay, so the key difference between PI and Actor-Critic would be that PI generates a policy in every step, whereas AC modifies/adapts an older policy in every step, right? Do you happen to know of any sources that would argue similarly?  
  > **username_0**: @NeilSlater Exactly, I'm on the same page about REINFORCE, I just worded it  awkwardly. Generating a single trajectory isn't really possible in a DP sense as you are taking the expected value, resulting in effectively generating all the trajectories. Thanks for that!  
  > **username_1**: @DaniëlWillemsen I don't know of any other sources that would try to argue the case that DP is not a policy-based method. It is just a basic fact that DP is the core of value-based methods. Most RL taxonomy diagrams will show this - DP methods represent the baseline of "Generalise Policy Iteration" approaches, and don't relate to policy gradients at all  
  > **username_1**: @nbro: I have tried to make that clearer, the difference being the policy derviation method in DP is not a direct parametric function over state that can be adjusted by changing those parameters. I guess another way to state this is that the Actor and Critic are not separate enough, they are just transformations of the same measurement.  
  
  ---
  
  ### Answer by username_3 on 2022-12-20. Score: 0
  
  > policy iteration is an actor-critic approach.
  
  This is a very insightful observation and I would agree with it. You can find similar statement in this [lecture](https://www.cs.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Informatik/Dialog_Systems_and_Machine_Learning/Lectures_RL/L5.pdf) and this NeurIPS [paper](https://proceedings.neurips.cc/paper/2013/hash/2dace78f80bc92e6d7493423d729448e-Abstract.html).
  
  The actor's role is to perform policy improvement and the critic's role is to perform policy evaluation. This helps explain its advantage over value-only approaches, as PI converges faster than VI although each iteration is more expensive.
  
  
  ---
  
  