We consider the problem of adapting an existing policy when the environment representation changes. Upon a change of the encoding of the observations the agent can no longer make use of its policy as it cannot correctly interpret the new observations. This paper proposes Greedy State Representation Learning (GSRL) to transfer the original policy by translating the environment representation back into its original encoding. To achieve this GSRL samples observations from both the environment and a dynamics model trained from prior experience. This generates pairs of state encodings, i.e., a new representation from the environment and a (biased) old representation from the forward model, that allow us to bootstrap a neural network model for state translation. Although early translations are unsatisfactory (as expected), the agent eventually learns a valid translation as it minimizes the error between expected and observed environment dynamics. Our experiments show the efficiency of our approach and that it translates the policy in considerably less steps than it would take to retrain the policy.
The authors propose a means to adapt to new state representations during reinforcement learning. The method works by learning a translation model that translates new state representations to old state representations. The authors evaluate the method on the MountainCar environment and show that the adaptation model is more efficient than training a new policy from scratch.
My concerns with this work are as follows:
- I don't find the type of changes to state representations very useful (e.g. Section 6, velocity *=2, position /= 2) nor practical. Moreover, learning a translation model to recover the old representation seems easy given this type of simple perturbations. This perturbation is fundamentally different than those used to motivate the problem (e.g. "a robot whose sensors break down" or "whose sensor outputs degrade", in the introduction).
- The authors only evaluate with one type of simple perturbation on one environment, hence I am skeptical regarding the generalizability of this work.
- The premise of this work is to not store old transitions (Section 1, the paragraph "the most simplistic idea is to..."), however this model does store old transitions because it uses prioritized experience replay. In this case there is no argument against using this data to train the translation model.
- A comparison that is missing from the paper is to fine-tune the existing model. I believe this is a more fair comparison in terms of sample-efficiency than training from scratch.
Other comments:
- For the title, to call the adaptation to slightly different state representations "policy adaptation" is a stretch.
- There are a lot of tangential information in the introduction on things like sample efficiency and model-free vs. model-based RL. This is distracting.
- The paper is excessively long.