Answers.Body: <blockquote>
  <p>In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?</p>
  </blockquote>
  <p>One practical benefit is that critics can use TD learning to bootstrap, allowing them to learn online on each step taken, plus learn in continuing problems. Pure actor algorithms like REINFORCE, cross-entropy method, and non-RL policy-only learners, such as genetic algorithms, require episodic problems. The smallest unit those can learn from is an entire episode. That is because without a critic providing value estimates, the only way to estimate return is to sample an actual return from the end of an episode.</p>
  <p>A TD-based critic may also have lower variance, which can aid in fast learning and stability, although this is not always a benefit. TD-based critics also have bias, which can cause instability. See <a href="https://stats.stackexchange.com/questions/355820/why-do-temporal-difference-td-methods-have-lower-variance-than-monte-carlo-met">Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?</a> on Cross Validated for more details on this.</p>
  <p>In practice, RL algorithm choice is a hyperparameter. As well as impacting difficulty of implementation, CPU and other resource costs, it can impact how well learning occurs depending on the problem being attempted. Usually, the only way you can tell a method is better for your problem is to try all the valid ones and measure their performance.</p>
  <blockquote>
  <p>I think it's effective to use only actors, especially for sparse rewards. Is that correct?</p>
  </blockquote>
  <p>The sparsity of reward is not a major factor here. The hard credit assignment problem that this brings into play means that the agent has to either assign a value, or pick an action, in states which have no direct feedback. All else being equal, both the value function and optimal action choice are hard to resolve when they depend on a large number of future policy decisions and state transitions which may vary.</p>
  <p>Which approach is best will depend on whether it is easier for a statistics-based learner to approximate the mapping from state to expected future reward, or to the action. The two functions can have different degrees of complexity.</p>
  <p>For an example where a policy function (for an actor) is simpler than a value function (for a critic or value-based method), you could consider a simple chase environment where a wolf tries to catch a rabbit on a simple continuous plane surface. The agent is the wolf, and gets a reward of +1 for catching the rabbit (I won't bother with other details, there are plenty of variations you could make).</p>
  <p>In the example environment, a simple strategy for the wolf is to turn to face the rabbit and move forwards. This is easy to map from current locations and facings of the wolf and the rabbit - for ultimate ease you can express the state as the rabbit's position and velocity in polar coordinates from the wolf's perspective. Compare that to the value function - it has to predict the time it will take to reach the rabbit given a current action choice. This is a far harder function to express based on the state, and as a result may also be harder to learn.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-13T10:51:38.653Z
Answers.Text: Maybe you should at least briefly (or link to some reference that) describe(s) why 1. actor-only methods (such as REINFORCE) require episodic problems and 2. TD-based methods may have lower bias.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-13T15:05:16.367Z
Answers.Text: @nbro I expanded the first paragraph with an explanation and linked a related question on cross-validated to address your second point
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-13T16:53:06.720Z
Answers.Text: Ok, thanks! That makes this answer clearer. Another thing that I think is missing from this answer is that you seem to focus on the comparison of actor-only methods and critic-only methods, though it seems to be implied that all the info mentioned is applicable to actor-critic methods as well as critic-only methods (such as Q-learning). In any case, it may be a good idea to differentiate actor-critic methods and critic-only methods.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-12T22:59:43.283Z
Answers.LastActivityDate: 2021-01-13T16:14:52.507Z
Answers.LastEditDate: 2021-01-13T16:14:52.507Z
Answers.UserDisplayName: Neil Slater
Body: <p>In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?</p>
  <p>This is not a comparison with the Q-learning series, but probably a method of learning the game with only the actor.</p>
  <p>I think it's effective to use only actors, especially for sparse rewards. Is that correct?</p>
  <p>Please, let me know if you have any specific use cases that use only actors.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-13T10:41:11.387Z
Comments.Text: You may also be interested [this question](https://ai.stackexchange.com/q/6196/2444).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><comparison><actor-critic-methods><policy-based-methods><continuous-tasks>
Title: What are the advantages of RL with actor-critic methods over actor-only methods?
UserDisplayName: ground clown
ThreadText: # What are the advantages of RL with actor-critic methods over actor-only methods?
  
  (Asked by: username_0 on 2021-01-12)
  
  In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?
  
  This is not a comparison with the Q-learning series, but probably a method of learning the game with only the actor.
  
  I think it's effective to use only actors, especially for sparse rewards. Is that correct?
  
  Please, let me know if you have any specific use cases that use only actors.
  
  **Tags:** reinforcement-learning, comparison, actor-critic-methods, policy-based-methods, continuous-tasks
  
  > **Comments:**
  > **username_1**: You may also be interested [this question](https://ai.stackexchange.com/q/6196/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-01-12. Score: 6
  
  > In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?
  
  One practical benefit is that critics can use TD learning to bootstrap, allowing them to learn online on each step taken, plus learn in continuing problems. Pure actor algorithms like REINFORCE, cross-entropy method, and non-RL policy-only learners, such as genetic algorithms, require episodic problems. The smallest unit those can learn from is an entire episode. That is because without a critic providing value estimates, the only way to estimate return is to sample an actual return from the end of an episode.
  
  A TD-based critic may also have lower variance, which can aid in fast learning and stability, although this is not always a benefit. TD-based critics also have bias, which can cause instability. See [Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?](https://stats.stackexchange.com/questions/355820/why-do-temporal-difference-td-methods-have-lower-variance-than-monte-carlo-met) on Cross Validated for more details on this.
  
  In practice, RL algorithm choice is a hyperparameter. As well as impacting difficulty of implementation, CPU and other resource costs, it can impact how well learning occurs depending on the problem being attempted. Usually, the only way you can tell a method is better for your problem is to try all the valid ones and measure their performance.
  
  > I think it's effective to use only actors, especially for sparse rewards. Is that correct?
  
  The sparsity of reward is not a major factor here. The hard credit assignment problem that this brings into play means that the agent has to either assign a value, or pick an action, in states which have no direct feedback. All else being equal, both the value function and optimal action choice are hard to resolve when they depend on a large number of future policy decisions and state transitions which may vary.
  
  Which approach is best will depend on whether it is easier for a statistics-based learner to approximate the mapping from state to expected future reward, or to the action. The two functions can have different degrees of complexity.
  
  For an example where a policy function (for an actor) is simpler than a value function (for a critic or value-based method), you could consider a simple chase environment where a wolf tries to catch a rabbit on a simple continuous plane surface. The agent is the wolf, and gets a reward of +1 for catching the rabbit (I won't bother with other details, there are plenty of variations you could make).
  
  In the example environment, a simple strategy for the wolf is to turn to face the rabbit and move forwards. This is easy to map from current locations and facings of the wolf and the rabbit - for ultimate ease you can express the state as the rabbit's position and velocity in polar coordinates from the wolf's perspective. Compare that to the value function - it has to predict the time it will take to reach the rabbit given a current action choice. This is a far harder function to express based on the state, and as a result may also be harder to learn.
  
  > **Comments:**
  > **username_1**: Maybe you should at least briefly (or link to some reference that) describe(s) why 1. actor-only methods (such as REINFORCE) require episodic problems and 2. TD-based methods may have lower bias.  
  > **username_2**: @username_1 I expanded the first paragraph with an explanation and linked a related question on cross-validated to address your second point  
  > **username_1**: Ok, thanks! That makes this answer clearer. Another thing that I think is missing from this answer is that you seem to focus on the comparison of actor-only methods and critic-only methods, though it seems to be implied that all the info mentioned is applicable to actor-critic methods as well as critic-only methods (such as Q-learning). In any case, it may be a good idea to differentiate actor-critic methods and critic-only methods.  
  
  ---
  
  