Inverse Reward Design
Autonomous agents optimize the reward function we give them. What they don't know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.
The authors develop "Inverse Reward Design" which is intended to help mitigate unintended consequences when designing reward functions that will guide agent behaviour. This is a highly relevant topic for the NIPS audience and a very interesting idea.
        
        The main potential contribution of this paper would be the clarity of thought it could bring to how issues of invariance can speak to the IRL problem. However, I find that the current presentation to be unclear at many points, particularly through the development of the main ideas. I have done my best to detail those points below.
        
        My main concern is that the presentation leading up to and including 4.2 is very confused.
        
        The authors define P(\tilde w | w^*). (They always condition on the same \tilde M so I am dropping that.)
        
        At l.148, the authors state that through Bayes' rule, get  P(w^* | \tilde w) \propto P(\tilde w | w^*) P(w^*)
        
        In the text, the display just after l.148 is where things start to get confused, by the use of P(w) to mean P(w^* = w)
        
        The full Bayes' rule statement is of course P(w^* | \tilde w) = P(\tilde w | w^*) P(w^*) / P(\tilde w), or 
        P(w^* | \tilde w) = P(\tilde w | w^*) P(w^*) / \int_\w^* P(\tilde w | w^*) P(w^*) dw^*
        
        I am confused because 1) the authors don't address the prior P(w^*) and 2) they state that the normalizing constant integrates "over the space of possible proxy rewards" and talk about how this is intractable "if \tilde{w} lies in an infinite or large finite set." Since w^* and \tilde{w} live in the same space (I believe) this is technically correct, but a very strange statement, since the integral in question is over w^*. I am concerned that there is substantial confusion here at the core idea of the whole paper. I suspect that this has resulted just from notational confusion, which often arises when shorthand [P(w^*)] and longhand [P(w^* = w)] notation is used for probabilities. I also suspect that whatever was implemented for the experiments was probably the right thing. However, I cannot assess this in the current state of the paper.
        
        Finally, I would suggest that the authors address two questions that I think are natural in this setting. 1) What do we do about the prior P(w^*)? (Obviously there won't be a universal answer for this, but I imagine the authors have some thoughts.) 2) 
        
        Below are additional comments:
        
        l.26 - The looping example isn't given in enough detail to help the reader. The example should stand alone from the cited paper.
        Figure 1 - I understand the podium in the figure on the right, but the meaning of the numbers is not clear. E.g. is the first number supposed to convey [2][0] or twenty?
        
        l.50 - "Extracting the true reward is..." - Do you mean "We define..."
        
        Figure 2: uknown
        
        l.134 - I think somewhere *much* earlier it needs to be made clear that this is a feature-based reward function that generalizes over different kinds of states.
        
        l.138 - \xi and \phi(\xi) are not precisely defined. I don't know what it means for an agent to "select trajectory \xi"
        
        l.145 - Notation in (1) and (2) is confusing. I find conditioning on \xi drawn from ... inside the [] strange. Subscript the expectation with it instead?
        
        l.148 - P(w) should be P(w|\tilde{M}) unless you make additional assumptions. (true also in (3)) On the other hand, since as far as I can tell everything in the paper conditions on \tilde{M}, you may as well just drop it.
        
        l.149 - Probabilities are normalized. If you're going to talk about the normalizing constant, write it explicitly rather than using \propto.
        
        l.150 - Is this the same w? Looks different.
        
        l.151 - I think you want = rather than \propto. Something is wrong in the integral in (3); check the variable of integration. There are \bar\phi and \tilde\phi and it's not clear what is happening.
                
        l.158 - "constrained to lie on a hypercube" - Just say this in math so that it's precise.
        
        l.166 - Can't (4) just go from 1 to N?
        
        l.228 - I wouldn't open a figure caption with "Our proof-of-concept domains make the unrealistic assumption." Furthermore, I have no idea how the figure caption relates to the figure.
      
=== UPDATE
The authors have clarified section 4 and I understand much better what they were after. I have updated my score accordingly.