Answers.Body: <p>In short, it is much easier for the agent to learn from a smaller dimensional state space. This is because the agent must also do representation learning; i.e. it must also infer <em>what</em> the state is telling it as part of the learning process. If you think of the architecture used in DQN to solve Atari, they had a CNN that outputted a vector which was then passed through some dense layers. Here the representation learning was done by the CNN and was trained using an end-to-end approach i.e. all updates to the network weights were done through the reinforcement learning objective; that is there is no supervised or unsupervised learning that takes place.</p>
  <p>This can be particularly difficult when you combine images with sparse rewards as there is not a lot of feedback so the representation learning can take a long time. <a href="https://arxiv.org/abs/2009.08319" rel="nofollow noreferrer">This paper</a> gives a good description of the problem of decoupling representation learning from reinforcement learning with a nice solution.</p>
  <p>The other main 'problem setting' I have seen images replaced with a latent state is when the authors are looking at planning. The problem with doing any kind of planning is that a model of the transition dynamics, <span class="math-container">$p(s' | s, a)$</span>, is needed. For high dimensional state spaces such as images, this can be very difficult to predict and even relatively small errors will quickly compound so if you use the model to predict multiple time steps into the future the planner is useless because of these compounding errors. I think there is a discussion on this in <a href="https://arxiv.org/abs/1911.08453" rel="nofollow noreferrer">this paper</a> (certainly there will be references therein that point you in the right direction).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-19T11:24:25.330Z
Answers.LastActivityDate: 2021-01-19T11:24:25.330Z
Answers.UserDisplayName: David
Body: <p>I have seen many papers using autoencoders to replace images (states) with latent representations. Some of those methods have shown higher rewards using such techniques. However, I do not understand how this helps the RL agent learn better. Perhaps viewing latent representations allows the agent to generalize to novel states more quickly?</p>
  <p>Here are 2 papers I have read -</p>
  <ul>
  <li><a href="https://arxiv.org/abs/1509.06113" rel="nofollow noreferrer">Deep Spatial Autoencoders for Visuomotor Learning</a></li>
  <li><a href="https://arxiv.org/abs/1806.00630" rel="nofollow noreferrer">DAQN: Deep Auto-encoder and Q-Network</a></li>
  </ul>
  
Tags: <reinforcement-learning><papers><autoencoders><generalization><state-spaces>
Title: How does replacing states with latent representations help RL agents?
UserDisplayName: desert_ranger
ThreadText: # How does replacing states with latent representations help RL agents?
  
  (Asked by: username_0 on 2021-01-18)
  
  I have seen many papers using autoencoders to replace images (states) with latent representations. Some of those methods have shown higher rewards using such techniques. However, I do not understand how this helps the RL agent learn better. Perhaps viewing latent representations allows the agent to generalize to novel states more quickly?
  
  Here are 2 papers I have read -
  
  * [Deep Spatial Autoencoders for Visuomotor Learning](https://arxiv.org/abs/1509.06113)
  * [DAQN: Deep Auto-encoder and Q-Network](https://arxiv.org/abs/1806.00630)
  
  **Tags:** reinforcement-learning, papers, autoencoders, generalization, state-spaces
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-19. Score: 3 | Accepted answer
  
  In short, it is much easier for the agent to learn from a smaller dimensional state space. This is because the agent must also do representation learning; i.e. it must also infer *what* the state is telling it as part of the learning process. If you think of the architecture used in DQN to solve Atari, they had a CNN that outputted a vector which was then passed through some dense layers. Here the representation learning was done by the CNN and was trained using an end-to-end approach i.e. all updates to the network weights were done through the reinforcement learning objective; that is there is no supervised or unsupervised learning that takes place.
  
  This can be particularly difficult when you combine images with sparse rewards as there is not a lot of feedback so the representation learning can take a long time. [This paper](https://arxiv.org/abs/2009.08319) gives a good description of the problem of decoupling representation learning from reinforcement learning with a nice solution.
  
  The other main 'problem setting' I have seen images replaced with a latent state is when the authors are looking at planning. The problem with doing any kind of planning is that a model of the transition dynamics, $p(s' | s, a)$, is needed. For high dimensional state spaces such as images, this can be very difficult to predict and even relatively small errors will quickly compound so if you use the model to predict multiple time steps into the future the planner is useless because of these compounding errors. I think there is a discussion on this in [this paper](https://arxiv.org/abs/1911.08453) (certainly there will be references therein that point you in the right direction).
  
  
  ---
  
  