Answers.Body: <p>It sounds like what you're suggesting is similar to what is done in methods that use a planner. These methods looks to learn the dynamics of the MDP to use to plan during training; that is they want to be able to learn the transition probabilities <span class="math-container">$p(s'| s, a)$</span>.</p>
  <p>In <a href="https://arxiv.org/abs/1911.08453" rel="nofollow noreferrer">this</a> paper that I read recently they note that learning to predict environment dynamics when the state/action space is high dimensional, as is the case with images, is difficult; so whilst it may be possible in theory it would be difficult to do and if you were predicting many steps into the future then the error would compound.</p>
  <p>A way around this, as is done in the referenced paper, is to use predict environment dynamics in a latent space. This means that they use a latent variable to predict the next state using e.g. Variational Autoencoders.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-14T11:33:43.697Z
Answers.LastActivityDate: 2021-01-14T11:33:43.697Z
Answers.UserDisplayName: David
Body: <p>During the course of training a DQN agent, all visited states are stored in a replay buffer. Therefore would it be practically possible for a CNN, given a reasonable amount of data, to predict the next RL state (in the form of an image) for a given action?</p>
  <p>For example, take the game of Atari as shown below -
  <a href="https://i.sstatic.net/lciZa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lciZa.png" alt="enter image description here" /></a>
  Here the agent can take 2 major actions - go left and go right. Would a CNN be generate the image of the bar going right/left for the respective actions?
  My practical knowledge of CNNs is quite limited and therefore I'm trying to gauge the abilities of CNNs before I take up a project.</p>
  
Tags: <reinforcement-learning><convolutional-neural-networks>
Title: Can a convolutional network predict states for a RL Agent
UserDisplayName: desert_ranger
ThreadText: # Can a convolutional network predict states for a RL Agent
  
  (Asked by: username_0 on 2021-01-14)
  
  During the course of training a DQN agent, all visited states are stored in a replay buffer. Therefore would it be practically possible for a CNN, given a reasonable amount of data, to predict the next RL state (in the form of an image) for a given action?
  
  For example, take the game of Atari as shown below -
  [![enter image description here](https://i.sstatic.net/lciZa.png)](https://i.sstatic.net/lciZa.png)
  Here the agent can take 2 major actions - go left and go right. Would a CNN be generate the image of the bar going right/left for the respective actions?
  My practical knowledge of CNNs is quite limited and therefore I'm trying to gauge the abilities of CNNs before I take up a project.
  
  **Tags:** reinforcement-learning, convolutional-neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-14. Score: 2 | Accepted answer
  
  It sounds like what you're suggesting is similar to what is done in methods that use a planner. These methods looks to learn the dynamics of the MDP to use to plan during training; that is they want to be able to learn the transition probabilities $p(s'| s, a)$.
  
  In [this](https://arxiv.org/abs/1911.08453) paper that I read recently they note that learning to predict environment dynamics when the state/action space is high dimensional, as is the case with images, is difficult; so whilst it may be possible in theory it would be difficult to do and if you were predicting many steps into the future then the error would compound.
  
  A way around this, as is done in the referenced paper, is to use predict environment dynamics in a latent space. This means that they use a latent variable to predict the next state using e.g. Variational Autoencoders.
  
  
  ---
  
  