Preferences Implicit in the State of the World
Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not only specify what to do, but also the much larger space of what not to do. It is easy to forget these preferences, since these preferences are already satisfied in our environment. This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want. We can therefore use this implicit preference information from the state to fill in the blanks. We develop an algorithm based on Maximum Causal Entropy IRL and use it to evaluate the idea in a suite of proof-of-concept environments designed to show its properties. We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized. Our code can be found at https://github.com/HumanCompatibleAI/rlsp.
 This work proposes a way to infer the implicit information in the initial state using IRL and combine the inferred reward with a specified reward to achieve better performance in a few simulated environments where the specified reward is not sufficient to solve the task. The main novelty of this work is to reformulate the Maximum Causal Entropy IRL objective using just the initial state as the end state of an expert trajectory to infer the underlying preference. Overall the proposed approach is impressive and the intuition behind the paper is novel and easy to understand. My main concerns are the following: - All the simulated experiments are able to demonstrate the effectiveness of the method, though they seem to be a bit too simplistic, e.g. known dynamics. As mentioned in Section 7, more real-environment experiments would make this method a lot stronger. - The way of choosing the distribution s_{-T} seems to require some sort of human preference, e.g. in the apple collection case, s_{-T} has to be sampled from the distribution where there*s no apple in the basket in order to make the algorithm to work. This assumption seems to make the implicit information of the initial state not so *implicit*. Besides, it*s unclear how to choose the horizon T. It would be interesting to see how the value of T affects the performance.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Preferences Implicit in the State of the World
Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not only specify what to do, but also the much larger space of what not to do. It is easy to forget these preferences, since these preferences are already satisfied in our environment. This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want. We can therefore use this implicit preference information from the state to fill in the blanks. We develop an algorithm based on Maximum Causal Entropy IRL and use it to evaluate the idea in a suite of proof-of-concept environments designed to show its properties. We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized. Our code can be found at https://github.com/HumanCompatibleAI/rlsp.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This work proposes a way to infer the implicit information in the initial state using IRL and combine the inferred reward with a specified reward to achieve better performance in a few simulated environments where the specified reward is not sufficient to solve the task. The main novelty of this work is to reformulate the Maximum Causal Entropy IRL objective using just the initial state as the end state of an expert trajectory to infer the underlying preference. Overall the proposed approach is impressive and the intuition behind the paper is novel and easy to understand. My main concerns are the following: - All the simulated experiments are able to demonstrate the effectiveness of the method, though they seem to be a bit too simplistic, e.g. known dynamics. As mentioned in Section 7, more real-environment experiments would make this method a lot stronger. - The way of choosing the distribution s_{-T} seems to require some sort of human preference, e.g. in the apple collection case, s_{-T} has to be sampled from the distribution where there*s no apple in the basket in order to make the algorithm to work. This assumption seems to make the implicit information of the initial state not so *implicit*. Besides, it*s unclear how to choose the horizon T. It would be interesting to see how the value of T affects the performance.
assistant
 This work proposes a way to infer the implicit information in the initial state using IRL and combine the inferred reward with a specified reward to achieve better performance in a few simulated environments where the specified reward is not sufficient to solve the task. The main novelty of this work is to reformulate the Maximum Causal Entropy IRL objective using just the initial state as the end state of an expert trajectory to infer the underlying preference. Overall the proposed approach is impressive and the intuition behind the paper is novel and easy to understand. My main concerns are the following: - All the simulated experiments are able to demonstrate the effectiveness of the method, though they seem to be a bit too simplistic, e.g. known dynamics. As mentioned in Section 7, more real-environment experiments would make this method a lot stronger. - The way of choosing the distribution s_{-T} seems to require some sort of human preference, e.g. in the apple collection case, s_{-T} has to be sampled from the distribution where there*s no apple in the basket in order to make the algorithm to work. This assumption seems to make the implicit information of the initial state not so *implicit*. Besides, it*s unclear how to choose the horizon T. It would be interesting to see how the value of T affects the performance.