Policy Shaping: Integrating Human Feedback with Reinforcement Learning
A long term goal of Interactive Reinforcement Learning is to incorporate nonexpert human feedback to solve complex tasks. Some state-of-the-art methods have approached this problem by mapping human information to rewards and values and iterating over them to compute better control policies. In this paper we argue for an alternate, more effective characterization of human feedback: Policy Shaping. We introduce Advise, a Bayesian approach that attempts to maximize the information gained from human feedback by utilizing it as direct policy labels. We compare Advise to state-of-the-art approaches and show that it can outperform them and is robust to infrequent and inconsistent human feedback.
This paper presents a new method for using human 
      feedback to improve a reinforcement-learning agent. The novelty of the 
      approach is to transform human feedback into a potentially inconsistent 
      estimate on the optimality of an action, instead of a reward as is often 
      the case. The resultant algorithm outperforms the previous state of the 
      art in a pair of toy domains. I thought this was an excellent paper, which 
      appropriately motivated the problem, clearly introduced a new idea and 
      then compared performance to other state-of-the-art algorithms (and not 
      just strawmen). I mostly have suggestions for improvement. - I 
      really liked the use of simulated human teacher, which could be 
      manipulated systematically to change likelihood and consistency of 
      feedback. One thing I would have liked to see is much lower likelihoods of 
      feedback (< 1%) - Something that worries me is that people may 
      be systematically inconsistent in their feedback. In psychology, one of 
      the most common uses of reward shaping is in the process of training a new 
      behaviour through successive approximation. That is, let’s say you want a 
      rat to pull a chain. First, you would reward the rat for getting near the 
      chain, then for touching it, and finally for pulling on it. At each step, 
      you deliberately stop rewarding the earlier step. How would Advise deal 
      with this type of systematic inconsistency in the human feedback (which is 
      the type of feedback they might get from an expert human trainer)? 
      - Sec 4.2: I found the assumption that humans only know one 
      optimal action to be a bit too strong. What happens to the algorithm if 
      that assumption is relaxed? Is performance compromised if the human 
      teacher vacillates between shaping two different optimal actions? Maybe it 
      should? A few words on this issue would be nice. - One other issue 
      that arises in working with human feedback is delay. Much inconsistency 
      may simple be due to people not responding at the same rate each 
      time—i.e., giving positive feedback only after another intervening action. 
      I think this might actually be another reason that the Advise approach 
      (which allows for inconsistency) is stronger than the other alternatives 
      considered. Minor things: line 053: “from MDP reward” is 
      an awkward construction sec 5.1. How do you win in Pac-Man? Eat both 
      dots? Not specified. Table 1 (and figures). Second column would be 
      clearer as “Reduced Frequency” instead of “Reduced Feedback”. Also, the 
      ordering of conditions (from left to right) is different in Table 1 than 
      the subsequent figures. lines 234-247. The relation between control 
      sharing and action biasing could be made a little clearer. lines 294. 
      prior estimate of the (missing of) Figure 2. Other than the ideal 
      case, why choose to plot only those cases where Advise does not help (if I 
      am reading Table 1 correctly)? line 369-370. More accurately, it’s 
      probably best to take the closest overestimate of C (i.e., err upward). 
      Figures 4 and 5: The text in the figures (esp the axis labels) was way 
      too small.