Answers.Body: <p>I highly recommend looking at <a href="http://www.incompleteideas.net/book/RLbook2018.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> by Richard Sutton and Andrew Barto.</p>
  
  <p>In it they write:</p>
  
  <blockquote>
    <p>Reinforcement learning, like many topics whose names end with “ing,” such as machine
    learning and mountaineering, is simultaneously a problem, a class of solution methods
    that work well on the problem, and the field that studies this problem and its solution
    methods. It is convenient to use a single name for all three things, but at the same time
    essential to keep the three conceptually separate. In particular, the distinction between
    problems and solution methods is very important in reinforcement learning; failing to
    make this distinction is the source of many confusions.</p>
  </blockquote>
  
  <p>On the solution side of things, rather than a family tree of reinforcement learning, there is a spectrum of different approaches to the problem.</p>
  
  <p><a href="https://i.sstatic.net/PqupD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PqupD.png" alt="A slice through the space of RL methods, highlighting the two of the most important dimensions explored in the Introduction of Reinforcement Learning"></a></p>
  
  <p>Many of the points within the spectrum above are TD(<span class="math-container">$\lambda$</span>) methods. When the state space is large, one might use a neural network to help generalize across similar states, such as DQN for instance.</p>
  
  <p>It should be noted that there are many applications of reinforcement learning algorithms that do not involve training an agent to perform a task. Instead one might want to evaluate an existing agent or use a general value function to predict an event using an agent's policy. </p>
  
  <p>Nevertheless, many popular problems consist of training an agent to perform a task. These algorithms can be categorized into a "family tree" based either on what values they are updating or on what problem they are tackling. (Source : <a href="https://github.com/NervanaSystems/coach" rel="nofollow noreferrer">https://github.com/NervanaSystems/coach</a>)</p>
  
  <ul>
  <li><p>On the left are Value Based methods which update the value function of a policy and find policies that optimize said value function.</p></li>
  <li><p>To the right of those are policy gradient methods which optimize for the policies directly without explicitly finding a value function.</p></li>
  <li><p>To the far right is Imitation Learning whereby one wants to copy a policy given demonstrations of an existing but unacceptable policy.</p></li>
  <li><p>And in the middle is Direct Future Prediction which is its own weird beast that I found out while finding that picture. Here is the <a href="https://arxiv.org/pdf/1611.01779.pdf" rel="nofollow noreferrer">arxiv paper</a> that describes DFP.</p></li>
  </ul>
  
  <p><a href="https://i.sstatic.net/uKOTj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uKOTj.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-17T20:59:09.300Z
Answers.LastActivityDate: 2020-02-17T22:27:00.977Z
Answers.LastEditDate: 2020-02-17T22:27:00.977Z
Answers.UserDisplayName: Jaden Travnik
Body: <p>Can anyone point me in the direction of a nice graph that depicts the "family tree", or hierarchy, of RL algorithms (or models)? For example, it splits the learning into TD and Monte Carlo methods, under which is listed all of the algorithms with their respective umbrella terms. Beneath each algorithm is shown modifications to those algorithms, etc. I'm having difficulty picturing where everything lies within the RL landscape.</p>
  
Tags: <machine-learning><reinforcement-learning>
Title: Is there a family tree for reinforcement learning algorithms?
UserDisplayName: mason7663
ThreadText: # Is there a family tree for reinforcement learning algorithms?
  
  (Asked by: username_0 on 2020-02-17)
  
  Can anyone point me in the direction of a nice graph that depicts the "family tree", or hierarchy, of RL algorithms (or models)? For example, it splits the learning into TD and Monte Carlo methods, under which is listed all of the algorithms with their respective umbrella terms. Beneath each algorithm is shown modifications to those algorithms, etc. I'm having difficulty picturing where everything lies within the RL landscape.
  
  **Tags:** machine-learning, reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-17. Score: 5 | Accepted answer
  
  I highly recommend looking at [Reinforcement Learning: An Introduction](http://www.incompleteideas.net/book/RLbook2018.pdf) by Richard Sutton and Andrew Barto.
  
  In it they write:
  
  > Reinforcement learning, like many topics whose names end with “ing,” such as machine
  > learning and mountaineering, is simultaneously a problem, a class of solution methods
  > that work well on the problem, and the field that studies this problem and its solution
  > methods. It is convenient to use a single name for all three things, but at the same time
  > essential to keep the three conceptually separate. In particular, the distinction between
  > problems and solution methods is very important in reinforcement learning; failing to
  > make this distinction is the source of many confusions.
  
  On the solution side of things, rather than a family tree of reinforcement learning, there is a spectrum of different approaches to the problem.
  
  [![A slice through the space of RL methods, highlighting the two of the most important dimensions explored in the Introduction of Reinforcement Learning](https://i.sstatic.net/PqupD.png)](https://i.sstatic.net/PqupD.png)
  
  Many of the points within the spectrum above are TD($\lambda$) methods. When the state space is large, one might use a neural network to help generalize across similar states, such as DQN for instance.
  
  It should be noted that there are many applications of reinforcement learning algorithms that do not involve training an agent to perform a task. Instead one might want to evaluate an existing agent or use a general value function to predict an event using an agent's policy.
  
  Nevertheless, many popular problems consist of training an agent to perform a task. These algorithms can be categorized into a "family tree" based either on what values they are updating or on what problem they are tackling. (Source : <https://github.com/NervanaSystems/coach>)
  
  * On the left are Value Based methods which update the value function of a policy and find policies that optimize said value function.
  * To the right of those are policy gradient methods which optimize for the policies directly without explicitly finding a value function.
  * To the far right is Imitation Learning whereby one wants to copy a policy given demonstrations of an existing but unacceptable policy.
  * And in the middle is Direct Future Prediction which is its own weird beast that I found out while finding that picture. Here is the [arxiv paper](https://arxiv.org/pdf/1611.01779.pdf) that describes DFP.
  
  [![enter image description here](https://i.sstatic.net/uKOTj.png)](https://i.sstatic.net/uKOTj.png)
  
  
  ---
  
  