Answers.Body: <blockquote>
  <p>The basic algorithms (MC control, Q-learning, or Dyna-Q) seemed to all be based on solving whichever specific maze the agent was trained on.</p>
  </blockquote>
  <p>All RL algorithms are based on creating solutions to a defined state and action space. If you limit your state space representation and training to a single maze, then that is what will be learned. This is no different from other machine learning approaches - they learn the traits of a population by being shown samples from that population (not just one example). They also need to be built for the range of input parameters that you need them to solve.</p>
  <p>In the case of RL, and your maze solver, that means the state representation needs to cover all possible mazes, not just a location in a single maze (there are ways to internalise some of the representation to the learning process such as using RNNs, but that is not relevant to the main answer here).</p>
  <p>The toy environments in Sutton &amp; Barto are often trivial to solve using non-RL approaches. They are not demonstrations of what RL can do, instead they have been chosen to explain how a particular issue related to learning works. Sutton &amp; Barto does include a chapter on more interesting and advanced uses of RL - that is chapter 16 &quot;Applications and Case Studies&quot; <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">in the second edition</a>.</p>
  <blockquote>
  <p>When I started learning RL, I thought that it would be more generalizable.</p>
  </blockquote>
  <p>It is, but without some kind of pre-training to support generalisation from a low number of examples, you have to:</p>
  <ul>
  <li><p>Model the general problem</p>
  </li>
  <li><p>Train the agent on the general problem</p>
  </li>
  </ul>
  <p>RL agents trained from scratch on new problems can seem very inefficient compared to learning by living creatures that RL roughly parallels. However, RL is not a model for general intelligence, but for learning through trial and error. Most examples start from no knowledge, not even basic priors for a maze such as the grid layout or the generalisable knowledge of movement and location.</p>
  <p>If you do provide a more general problem definition and training examples, and use a function approximator that can generalise internally (such as a neural network), then an agent can learn to solve problems in a more general sense and <em>may</em> also generate internal representations that (approximately) match up to common factors in the general problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-08T07:45:48.677Z
Answers.LastActivityDate: 2021-06-08T13:29:44.900Z
Answers.LastEditDate: 2021-06-08T13:29:44.900Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have read most of Sutton and Barto's introductory text on reinforcement learning. I thought I would try to apply some of the RL algorithms in the book to a previous assignment I had done on <a href="https://www.sokobanonline.com/" rel="nofollow noreferrer">Sokoban</a>, in which you are in a maze-like grid environment, trying to stack three snowballs into a snowman on a predefined location on the grid.</p>
  <p>The basic algorithms (MC control, Q-learning, or Dyna-Q) seemed to all be based on solving whichever specific maze the agent was trained on. For example, the transition probabilities of going from coordinate (1,2) to (1,3) would be different for different mazes (since in one maze, we could have an obstacle at (1,3)). An agent that calculates its rewards based on one maze using these algorithms doesn't seem like it would know what to do given a totally different maze. It would have to <em>retrain</em>: 1) either take real life actions to relearn from scratch how to navigate a maze, or 2) be given the model of the maze, either exact or approximate (which seems infeasible in a real life setting) so that planning without taking actions is possible.</p>
  <p>When I started learning RL, I thought that it would be more generalizable. This leads me to the question: Is this problem covered in multi-task RL? How would you categorize the various areas of RL in terms of the general problem that it is looking to solve?</p>
  
Tags: <reinforcement-learning>
Title: What are the various problems RL is trying to solve?
UserDisplayName: Snowball
ThreadText: # What are the various problems RL is trying to solve?
  
  (Asked by: username_0 on 2021-06-08)
  
  I have read most of Sutton and Barto's introductory text on reinforcement learning. I thought I would try to apply some of the RL algorithms in the book to a previous assignment I had done on [Sokoban](https://www.sokobanonline.com/), in which you are in a maze-like grid environment, trying to stack three snowballs into a snowman on a predefined location on the grid.
  
  The basic algorithms (MC control, Q-learning, or Dyna-Q) seemed to all be based on solving whichever specific maze the agent was trained on. For example, the transition probabilities of going from coordinate (1,2) to (1,3) would be different for different mazes (since in one maze, we could have an obstacle at (1,3)). An agent that calculates its rewards based on one maze using these algorithms doesn't seem like it would know what to do given a totally different maze. It would have to *retrain*: 1) either take real life actions to relearn from scratch how to navigate a maze, or 2) be given the model of the maze, either exact or approximate (which seems infeasible in a real life setting) so that planning without taking actions is possible.
  
  When I started learning RL, I thought that it would be more generalizable. This leads me to the question: Is this problem covered in multi-task RL? How would you categorize the various areas of RL in terms of the general problem that it is looking to solve?
  
  **Tags:** reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-08. Score: 2 | Accepted answer
  
  > The basic algorithms (MC control, Q-learning, or Dyna-Q) seemed to all be based on solving whichever specific maze the agent was trained on.
  
  All RL algorithms are based on creating solutions to a defined state and action space. If you limit your state space representation and training to a single maze, then that is what will be learned. This is no different from other machine learning approaches - they learn the traits of a population by being shown samples from that population (not just one example). They also need to be built for the range of input parameters that you need them to solve.
  
  In the case of RL, and your maze solver, that means the state representation needs to cover all possible mazes, not just a location in a single maze (there are ways to internalise some of the representation to the learning process such as using RNNs, but that is not relevant to the main answer here).
  
  The toy environments in Sutton & Barto are often trivial to solve using non-RL approaches. They are not demonstrations of what RL can do, instead they have been chosen to explain how a particular issue related to learning works. Sutton & Barto does include a chapter on more interesting and advanced uses of RL - that is chapter 16 "Applications and Case Studies" [in the second edition](http://incompleteideas.net/book/RLbook2020.pdf).
  
  > When I started learning RL, I thought that it would be more generalizable.
  
  It is, but without some kind of pre-training to support generalisation from a low number of examples, you have to:
  
  * Model the general problem
  * Train the agent on the general problem
  
  RL agents trained from scratch on new problems can seem very inefficient compared to learning by living creatures that RL roughly parallels. However, RL is not a model for general intelligence, but for learning through trial and error. Most examples start from no knowledge, not even basic priors for a maze such as the grid layout or the generalisable knowledge of movement and location.
  
  If you do provide a more general problem definition and training examples, and use a function approximator that can generalise internally (such as a neural network), then an agent can learn to solve problems in a more general sense and *may* also generate internal representations that (approximately) match up to common factors in the general problem.
  
  
  ---
  
  