Answers.Body: <p>In <a href="http://incompleteideas.net/book/RLbook2018.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> the authors suggest that the <em>topic</em> of reinforcement learning covers analysis and solutions to problems that can be framed in this way:</p>
  
  <blockquote>
    <p>Reinforcement learning, like many topics whose names end with “ing,” such as machine
    learning and mountaineering, is simultaneously a problem, a class of solution methods
    that work well on the problem, and the field that studies this problem and its solution
    methods. It is convenient to use a single name for all three things, but at the same time
    essential to keep the three conceptually separate. In particular, the distinction between
    problems and solution methods is very important in reinforcement learning; failing to
    make this distinction is the source of many confusions.</p>
  </blockquote>
  
  <p>And: </p>
  
  <blockquote>
    <p>Markov decision processes are intended to include just
    these three aspects—sensation, action, and goal—in their simplest possible forms without
    trivializing any of them. Any method that is well suited to solving such problems we
    consider to be a reinforcement learning method.</p>
  </blockquote>
  
  <p>So, to answer your questions, the simplest take on this is yes there is more (much more) to RL than the classic value-based optimal control methods of SARSA and Q-learning. </p>
  
  <p>Including DP and other "RL-related" algorithms in the book allows the author to show how closely related the concepts are. For example, there is little in practice that differentiates Dyna-Q (a planning algorithm closely related to Q-learning) from  experience replay. Calling one strictly "planning" and the other "reinforcement learning" and treating them as separate can reduce insight into the topic. In many cases there are hybrid methods or even a continuum between what you may initially think of as RL and "not RL" approaches. Understanding this gives you a toolkit to modify and invent algorithms.</p>
  
  <p>Having said that, the book is not the sole arbiter of what is and isn't reinforcement learning. Ultimately this is just a classification issue, and it only matters if you are communicating with someone and there is a chance for misunderstanding. If you name which algorithm you are using, it doesn't really matter whether the person you are talking to thinks it is RL or not RL. It matters what the problem is and how you propose to solve it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-30T16:06:16.180Z
Answers.LastActivityDate: 2019-04-30T16:06:16.180Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>It seems that another rather controversial point is about the inclusion of evolutionary algorithms as Reinforcement Learning ones.
  Sutton &amp; Barto do not. They argue that</p>
  <p><a href="https://i.sstatic.net/2AxGA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2AxGA.png" alt="Evolution vs Learning Sutton&amp;Barto 20202" /></a></p>
  <p>And also:</p>
  <p><a href="https://i.sstatic.net/YR04e.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YR04e.png" alt="More evolution vs learning." /></a></p>
  <p>Other people related with the subject, as the HSE University that offers a course in Coursera,  Maxim Lapan , or P. Palanisamy (both Packt's authors)  include them into the subject. Apparently they support the idea that RL is defined by changes in performance  resulting  from interaction with the environment.
  For instance, Lapan classifies the cross-entropy method as model-free, policy-based and on -policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-10T21:58:55.320Z
Answers.LastActivityDate: 2021-03-10T21:58:55.320Z
Answers.UserDisplayName: Hermes Morales
Answers.Body: <p>The dynamic programming (DP) algorithms like <em>policy iteration</em> (PI) and <em>value iteration</em> (VI) are often presented in the context of reinforcement learning (in particular, in the book <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> by Barto and Sutton) because they are very related to reinforcement learning algorithms, like <span class="math-container">$Q$</span>-learning - They are all based on the assumption that the environment/problem can be modelled as a <strong>Markov Decision Process (MDP)</strong>.</p>
  <p>However, PI and VI require that the <strong>transition model</strong> and <strong>reward functions</strong> of the underlying MDP are both known. They can also be referred to as <strong>planning</strong> algorithms, because they can be used to find a policy (which can be thought of as <strong>plan</strong>) given this transition model and reward function of the MDP, without having to take random actions in the environment. They just exploit the given rules of the environment.</p>
  <p>On the other hand, RL algorithms do <em>not usually</em> require or use the MDP's transition model, although they still require the reward function, which is fundamental. They attempt to find a policy or value function (from which the policy can be derived) by randomly interacting with the environment.</p>
  <p>There are several categories of RL algorithms. There are temporal-difference, Monte-Carlo, actor-critic, model-free, model-based, on-policy, off-policy, prediction, control, policy-based or value-based algorithms. These categories can overlap. For example, <span class="math-container">$Q$</span>-learning is a temporal-difference (TD), model-free, off-policy, control and value-based algorithm: it is based on an temporal-difference (TD) update rule, it doesn't use a model of the environment (model-free), it uses a behavioural policy that is different than the policy it learns (off-policy), it is used to find a policy (control) and it attempts to approximate a value function rather than directly the policy (value-based).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-30T14:59:24.643Z
Answers.Text: Can you further explain your last sentence ? are you talking about variations like TD[lambda]  or policy gradient methods? or something else?
  Also  do you have any recommendation of where I can further read about RL, knowing I already explored the Barto and Sutton book?
Answers.UserDisplayName: Miguel Saraiva
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-30T15:19:37.863Z
Answers.Text: @MiguelSaraiva I have updated my answer. I would recommend that you read that book again and more carefully (and that you start implementing some of those algorithms to get full understanding of those concepts). This is a decent book. However, RL comprises a lot of concepts and dense terminology, which often confuse beginners. Have also a look at this question: https://ai.stackexchange.com/q/6997/2444, in particular, my answer: https://ai.stackexchange.com/a/7005/2444.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-30T15:54:32.317Z
Answers.Text: I have seen some of the videos in that series by Silver already and they are indeed good. Thank you for your help.
Answers.UserDisplayName: Miguel Saraiva
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-30T14:54:56.310Z
Answers.LastActivityDate: 2023-11-17T01:01:15.000Z
Answers.LastEditDate: 2023-11-17T01:01:15.000Z
Answers.UserDisplayName: nbro
Body: <p>What are the areas/algorithms that belong to reinforcement learning?</p>
  <p>TD(0), Q-Learning and SARSA are all temporal-difference algorithms, which belong to the reinforcement learning area, but is there more to it?</p>
  <p>Are the dynamic programming algorithms, such as policy iteration and value iteration, considered as part of reinforcement learning? Or are these just the basis for the temporal-difference algorithms, which are the only RL algorithms?</p>
  
Tags: <reinforcement-learning><terminology><temporal-difference-methods><planning><dynamic-programming>
Title: What algorithms are considered reinforcement learning algorithms?
UserDisplayName: Miguel Saraiva
ThreadText: # What algorithms are considered reinforcement learning algorithms?
  
  (Asked by: username_0 on 2019-04-30)
  
  What are the areas/algorithms that belong to reinforcement learning?
  
  TD(0), Q-Learning and SARSA are all temporal-difference algorithms, which belong to the reinforcement learning area, but is there more to it?
  
  Are the dynamic programming algorithms, such as policy iteration and value iteration, considered as part of reinforcement learning? Or are these just the basis for the temporal-difference algorithms, which are the only RL algorithms?
  
  **Tags:** reinforcement-learning, terminology, temporal-difference-methods, planning, dynamic-programming
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2019-04-30. Score: 14 | Accepted answer
  
  The dynamic programming (DP) algorithms like *policy iteration* (PI) and *value iteration* (VI) are often presented in the context of reinforcement learning (in particular, in the book [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2020.pdf) by Barto and Sutton) because they are very related to reinforcement learning algorithms, like $Q$-learning - They are all based on the assumption that the environment/problem can be modelled as a **Markov Decision Process (MDP)**.
  
  However, PI and VI require that the **transition model** and **reward functions** of the underlying MDP are both known. They can also be referred to as **planning** algorithms, because they can be used to find a policy (which can be thought of as **plan**) given this transition model and reward function of the MDP, without having to take random actions in the environment. They just exploit the given rules of the environment.
  
  On the other hand, RL algorithms do *not usually* require or use the MDP's transition model, although they still require the reward function, which is fundamental. They attempt to find a policy or value function (from which the policy can be derived) by randomly interacting with the environment.
  
  There are several categories of RL algorithms. There are temporal-difference, Monte-Carlo, actor-critic, model-free, model-based, on-policy, off-policy, prediction, control, policy-based or value-based algorithms. These categories can overlap. For example, $Q$-learning is a temporal-difference (TD), model-free, off-policy, control and value-based algorithm: it is based on an temporal-difference (TD) update rule, it doesn't use a model of the environment (model-free), it uses a behavioural policy that is different than the policy it learns (off-policy), it is used to find a policy (control) and it attempts to approximate a value function rather than directly the policy (value-based).
  
  > **Comments:**
  > **username_0**: Can you further explain your last sentence ? are you talking about variations like TD[lambda]  or policy gradient methods? or something else? Also  do you have any recommendation of where I can further read about RL, knowing I already explored the Barto and Sutton book?  
  > **username_3**: @MiguelSaraiva I have updated my answer. I would recommend that you read that book again and more carefully (and that you start implementing some of those algorithms to get full understanding of those concepts). This is a decent book. However, RL comprises a lot of concepts and dense terminology, which often confuse beginners. Have also a look at this question: https://ai.stackexchange.com/q/6997/2444, in particular, my answer: https://ai.stackexchange.com/a/7005/2444.  
  > **username_0**: I have seen some of the videos in that series by Silver already and they are indeed good. Thank you for your help.  
  
  ---
  
  ### Answer by username_1 on 2019-04-30. Score: 4
  
  In [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2018.pdf) the authors suggest that the *topic* of reinforcement learning covers analysis and solutions to problems that can be framed in this way:
  
  > Reinforcement learning, like many topics whose names end with “ing,” such as machine
  > learning and mountaineering, is simultaneously a problem, a class of solution methods
  > that work well on the problem, and the field that studies this problem and its solution
  > methods. It is convenient to use a single name for all three things, but at the same time
  > essential to keep the three conceptually separate. In particular, the distinction between
  > problems and solution methods is very important in reinforcement learning; failing to
  > make this distinction is the source of many confusions.
  
  And:
  
  > Markov decision processes are intended to include just
  > these three aspects—sensation, action, and goal—in their simplest possible forms without
  > trivializing any of them. Any method that is well suited to solving such problems we
  > consider to be a reinforcement learning method.
  
  So, to answer your questions, the simplest take on this is yes there is more (much more) to RL than the classic value-based optimal control methods of SARSA and Q-learning.
  
  Including DP and other "RL-related" algorithms in the book allows the author to show how closely related the concepts are. For example, there is little in practice that differentiates Dyna-Q (a planning algorithm closely related to Q-learning) from experience replay. Calling one strictly "planning" and the other "reinforcement learning" and treating them as separate can reduce insight into the topic. In many cases there are hybrid methods or even a continuum between what you may initially think of as RL and "not RL" approaches. Understanding this gives you a toolkit to modify and invent algorithms.
  
  Having said that, the book is not the sole arbiter of what is and isn't reinforcement learning. Ultimately this is just a classification issue, and it only matters if you are communicating with someone and there is a chance for misunderstanding. If you name which algorithm you are using, it doesn't really matter whether the person you are talking to thinks it is RL or not RL. It matters what the problem is and how you propose to solve it.
  
  
  ---
  
  ### Answer by username_2 on 2021-03-10. Score: 1
  
  It seems that another rather controversial point is about the inclusion of evolutionary algorithms as Reinforcement Learning ones.
  Sutton & Barto do not. They argue that
  
  [![Evolution vs Learning Sutton&Barto 20202](https://i.sstatic.net/2AxGA.png)](https://i.sstatic.net/2AxGA.png)
  
  And also:
  
  [![More evolution vs learning.](https://i.sstatic.net/YR04e.png)](https://i.sstatic.net/YR04e.png)
  
  Other people related with the subject, as the HSE University that offers a course in Coursera, Maxim Lapan , or P. Palanisamy (both Packt's authors) include them into the subject. Apparently they support the idea that RL is defined by changes in performance resulting from interaction with the environment.
  For instance, Lapan classifies the cross-entropy method as model-free, policy-based and on -policy.
  
  
  ---
  
  