Self-Supervised Policy Adaptation
We consider the problem of adapting an existing policy when the environment representation changes. Upon a change of the encoding of the observations the agent can no longer make use of its policy as it cannot correctly interpret the new observations. This paper proposes Greedy State Representation Learning (GSRL) to transfer the original policy by translating the environment representation back into its original encoding. To achieve this GSRL samples observations from both the environment and a dynamics model trained from prior experience. This generates pairs of state encodings, i.e., a new representation from the environment and a (biased) old representation from the forward model, that allow us to bootstrap a neural network model for state translation. Although early translations are unsatisfactory (as expected), the agent eventually learns a valid translation as it minimizes the error between expected and observed environment dynamics. Our experiments show the efficiency of our approach and that it translates the policy in considerably less steps than it would take to retrain the policy.
The paper investigates a setting in which the observation function changes while the underlying environmental dynamic stays the same. In order to re-use the policy which was trained on the old observation function, they propose to learn a mapping function to map the new observations to the old ones. I believe the work is interesting as generalization and reducing the sample complexity of learning policies, for example through re-use of old policies, is of high current interest. However, I believe this paper requires more work to show the feasibility of the proposed approach. In particular: - The proposed method has the problem that matching is done *locally* and without any guarantee that the mapping function will converge to the correct mapping. This is not a problem of the method itself but of the challenging problem setup. The authors discuss this and propose two approaches to alleviate this. However, no experimental evidence is shown whether and to what extend this prevents wrong local minima, as I don*t believe the mountain car has such problems? - Evaluation is only done on the mountain-car experiment, which is not sufficient to show feasibility in general - The learning curves in the experiment seem highly volatile and unstable. I*m not sure why this is the case, maybe wrong hyperparameters or a bug in the code? - Lastly, the paper is over the recommended limit of 8 pages and could, in my opinion, made more concise at many points to shorten it and also make it easier to read. In summary: I believe this interesting work, but requires more experiments in different environments and additional ablation studies to show the feasibility of the proposed method. Making the writing more concise would, in my opinion, not only shorten the paper but also make it easier to read.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Self-Supervised Policy Adaptation
We consider the problem of adapting an existing policy when the environment representation changes. Upon a change of the encoding of the observations the agent can no longer make use of its policy as it cannot correctly interpret the new observations. This paper proposes Greedy State Representation Learning (GSRL) to transfer the original policy by translating the environment representation back into its original encoding. To achieve this GSRL samples observations from both the environment and a dynamics model trained from prior experience. This generates pairs of state encodings, i.e., a new representation from the environment and a (biased) old representation from the forward model, that allow us to bootstrap a neural network model for state translation. Although early translations are unsatisfactory (as expected), the agent eventually learns a valid translation as it minimizes the error between expected and observed environment dynamics. Our experiments show the efficiency of our approach and that it translates the policy in considerably less steps than it would take to retrain the policy.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper investigates a setting in which the observation function changes while the underlying environmental dynamic stays the same. In order to re-use the policy which was trained on the old observation function, they propose to learn a mapping function to map the new observations to the old ones. I believe the work is interesting as generalization and reducing the sample complexity of learning policies, for example through re-use of old policies, is of high current interest. However, I believe this paper requires more work to show the feasibility of the proposed approach. In particular: - The proposed method has the problem that matching is done *locally* and without any guarantee that the mapping function will converge to the correct mapping. This is not a problem of the method itself but of the challenging problem setup. The authors discuss this and propose two approaches to alleviate this. However, no experimental evidence is shown whether and to what extend this prevents wrong local minima, as I don*t believe the mountain car has such problems? - Evaluation is only done on the mountain-car experiment, which is not sufficient to show feasibility in general - The learning curves in the experiment seem highly volatile and unstable. I*m not sure why this is the case, maybe wrong hyperparameters or a bug in the code? - Lastly, the paper is over the recommended limit of 8 pages and could, in my opinion, made more concise at many points to shorten it and also make it easier to read. In summary: I believe this interesting work, but requires more experiments in different environments and additional ablation studies to show the feasibility of the proposed method. Making the writing more concise would, in my opinion, not only shorten the paper but also make it easier to read.
assistant
The paper investigates a setting in which the observation function changes while the underlying environmental dynamic stays the same. In order to re-use the policy which was trained on the old observation function, they propose to learn a mapping function to map the new observations to the old ones. I believe the work is interesting as generalization and reducing the sample complexity of learning policies, for example through re-use of old policies, is of high current interest. However, I believe this paper requires more work to show the feasibility of the proposed approach. In particular: - The proposed method has the problem that matching is done *locally* and without any guarantee that the mapping function will converge to the correct mapping. This is not a problem of the method itself but of the challenging problem setup. The authors discuss this and propose two approaches to alleviate this. However, no experimental evidence is shown whether and to what extend this prevents wrong local minima, as I don*t believe the mountain car has such problems? - Evaluation is only done on the mountain-car experiment, which is not sufficient to show feasibility in general - The learning curves in the experiment seem highly volatile and unstable. I*m not sure why this is the case, maybe wrong hyperparameters or a bug in the code? - Lastly, the paper is over the recommended limit of 8 pages and could, in my opinion, made more concise at many points to shorten it and also make it easier to read. In summary: I believe this interesting work, but requires more experiments in different environments and additional ablation studies to show the feasibility of the proposed method. Making the writing more concise would, in my opinion, not only shorten the paper but also make it easier to read.