Answers.Body: <p>Check out <a href="https://arxiv.org/abs/1707.06203" rel="nofollow noreferrer">Imagination-Augmented Agents</a> paper - seems like it does what you are talking about. The agent itself is the standard A3C that you are familiar with. The novelty is the &quot;imagination&quot; environment model which is trained to predict the  behavior of the environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-03T13:30:36.487Z
Answers.LastActivityDate: 2021-04-03T13:30:36.487Z
Answers.UserDisplayName: Kostya
Answers.Body: <p>Yes, there are algorithms that try to predict the next state. Usually this will be a model based algorithm -- this is where the agent tries to make use of a model of the environment to help it learn. I'm not sure on the best resource to learn about this but my go-to recommendation is always the Sutton and Barto book.</p>
  <p><a href="https://arxiv.org/abs/2006.00900" rel="nofollow noreferrer">This paper</a> introduces PlanGAN; the idea of this model is to use a GAN to generate a trajectory. This will include not only predicting the next state but all future states in a trajectory.</p>
  <p><a href="https://arxiv.org/abs/1507.00814" rel="nofollow noreferrer">This paper</a> introduces a novelty function to incentivise the agent to visit unexplored states. The idea is that for unexplored states, a model that predicts the next state from the state-action tuple will have high error (measured by Euclidean distance from true next state) and they add this error to the original reward to make a modified reward.</p>
  <p><a href="https://arxiv.org/abs/1912.01603" rel="nofollow noreferrer">This paper</a> introduces Dreamer. This is where <em>all</em> learning is done in a latent space and so the transition dynamics of this latent space must be learned, another example of needing to learn the next state.</p>
  <p>These are just some examples of papers that try to predict the next state, there are many more out there that I would recommend you look for.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-03T14:01:12.117Z
Answers.LastActivityDate: 2021-04-03T14:01:12.117Z
Answers.UserDisplayName: David
Body: <p>So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially.</p>
  <p>If I remember correctly, all these algorithms focus on the value of the action and try to get the maximum one. <strong>Is there an RL algorithm that also tries to predict what the next state will be, given a possible action that the agent would take?</strong></p>
  <p>Then, in parallel to the constant training for getting the best reward, there will also be constant training to predict the next state as accurately as possible? And then have that prediction of the next state always be passed as an input into the NN that decides on the action to take. Seems like a useful piece of information.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-03T18:17:36.497Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/122628/discussion-on-question-by-ram-rachum-are-there-rl-algorithms-that-also-try-to-pr).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><model-based-methods><algorithm-request>
Title: Are there RL algorithms that also try to predict the next state?
UserDisplayName: Ram Rachum
ThreadText: # Are there RL algorithms that also try to predict the next state?
  
  (Asked by: username_0 on 2021-04-01)
  
  So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially.
  
  If I remember correctly, all these algorithms focus on the value of the action and try to get the maximum one. **Is there an RL algorithm that also tries to predict what the next state will be, given a possible action that the agent would take?**
  
  Then, in parallel to the constant training for getting the best reward, there will also be constant training to predict the next state as accurately as possible? And then have that prediction of the next state always be passed as an input into the NN that decides on the action to take. Seems like a useful piece of information.
  
  **Tags:** reinforcement-learning, deep-rl, model-based-methods, algorithm-request
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/122628/discussion-on-question-by-ram-username_0-are-there-rl-algorithms-that-also-try-to-pr).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2021-04-03. Score: 4 | Accepted answer
  
  Yes, there are algorithms that try to predict the next state. Usually this will be a model based algorithm -- this is where the agent tries to make use of a model of the environment to help it learn. I'm not sure on the best resource to learn about this but my go-to recommendation is always the Sutton and Barto book.
  
  [This paper](https://arxiv.org/abs/2006.00900) introduces PlanGAN; the idea of this model is to use a GAN to generate a trajectory. This will include not only predicting the next state but all future states in a trajectory.
  
  [This paper](https://arxiv.org/abs/1507.00814) introduces a novelty function to incentivise the agent to visit unexplored states. The idea is that for unexplored states, a model that predicts the next state from the state-action tuple will have high error (measured by Euclidean distance from true next state) and they add this error to the original reward to make a modified reward.
  
  [This paper](https://arxiv.org/abs/1912.01603) introduces Dreamer. This is where *all* learning is done in a latent space and so the transition dynamics of this latent space must be learned, another example of needing to learn the next state.
  
  These are just some examples of papers that try to predict the next state, there are many more out there that I would recommend you look for.
  
  
  ---
  
  ### Answer by username_2 on 2021-04-03. Score: 2
  
  Check out [Imagination-Augmented Agents](https://arxiv.org/abs/1707.06203) paper - seems like it does what you are talking about. The agent itself is the standard A3C that you are familiar with. The novelty is the "imagination" environment model which is trained to predict the behavior of the environment.
  
  
  ---
  
  