Learning from Limited Demonstrations
We propose a Learning from Demonstration (LfD) algorithm which leverages expert data, even if they are very few or inaccurate. We achieve this by using both expert data, as well as reinforcement signals gathered through trial-and-error interactions with the environment. The key idea of our approach, Approximate Policy Iteration with Demonstration (APID), is that expert's suggestions are used to define linear constraints which guide the optimization performed by Approximate Policy Iteration. We prove an upper bound on the Bellman error of the estimate computed by APID at each iteration. Moreover, we show empirically that APID outperforms pure Approximate Policy Iteration, a state-of-the-art LfD algorithm, and supervised learning in a variety of scenarios, including when very few and/or suboptimal demonstrations are available. Our experiments include simulations as well as a real robot path-finding task.
formulated as a coupled constraint convex optimization, in which expert demonstrations define a set of linear constraints in API. The optimization is formulated in a way that permits mistakes in the demonstrations provided by the expert, and also accommodates variable availability of demonstrations (i.e., just an initial batch or continued demonstrations). We provide a theoretical analysis describing an upper bound on the Bellman error achievable by our approach.
We evaluate our algorithm in a simulated environment under various scenarios, such as varying the quality and quantity of expert demonstrations. In all cases, we compare our algorithm with LeastSquare Policy Iteration (LSPI) [6], a popular API method, as well as with a state-of-the-art LfD method, Dataset Aggregation (DAgger) [1]. We also evaluate the algorithm's practicality in a real robot path finding task, where there are few demonstrations, and exploration data is expensive due to limited time. In all of the experiments, our method outperformed LSPI, using fewer exploration data and exhibiting significantly less variance. Our method also significantly outperformed Dataset Aggregation (DAgger), a state-of-art LfD algorithm, in cases where the expert demonstrations are few or suboptimal.
Although the author response has addressed my 
      questions or clarified my understanding, I am leaving the original 
      review intact below for reference and posterity. In addition, I will 
      note two things in response to the response: - With respect to 
      TAMER and similar reward-based methods, I believe you could convert 
      demonstrations to oracle-like reward. Perhaps that is a thought for 
      later. - In the case of adversarial demonstrations, I was really 
      thinking of the case where they are adversarial but you do not know it 
      (or if you like, imagine that they are very, very poor quality 
      demonstrations) so that alpha isn't 0. ---original--- The 
      paper describes an extension to the modified BRM algorithm proposed by 
      Antos et al. by incorporating human demonstrations as constraints. The 
      idea is elegant and overall the paper shows that it is a promising 
      area of research. I do have some questions/concerns: Did the 
      authors consider using the modified BRM instead of the LSTD method? 
      Work by Farahmand et al. (Reference 5) states that when the linear 
      parameterized functions are used, the solution computed by the 
      modified BRM is *equivalent* to LSPI; however, they go on to state 
      that this proposition is not true when regularized versions are used 
      for LSTD and BRM. It is potentially possible that you get different 
      results when using modified BRM. It seems that the difference 
      between APID and LSPI is that the former uses some form of 
      regularization as well as a constraint on human demonstrations, else 
      they would provide the same answer (as shown in references 5 and 13). 
      It is not clear if a comparison to LSPI is not entirely meaningful 
      here because essentially more domain knowledge is being fed into APID. 
      Along those lines, in the experiments, did the authors use a 
      regularized of LSPI for the comparison? Perhaps a comparison that 
      would strengthen the paper is the inclusion of another established LfD 
      method: TAMER by Knox and Stone. TAMER and some of its variants have 
      been known to perform well in conditions with limited human input and 
      work under similar conditions (using human information as well as 
      direct experience). A comparison to this algorithm would help to 
      position the work well in the space of RL and LfD. What aspect 
      of the algorithm particularly takes into account the limited number of 
      demonstrations? There does not appear to a specific aspect of the 
      algorithm that maximizes the information from the humans (in fact, any 
      information loss from the humans is made up using direct interaction 
      in the world). It is simply using information from experts as 
      constraints. What kind of optimization do you use for the modified 
      BRM algorithm? In equation 4, hQ can be obtained in closed form, but 
      how about Q^ (how is w optimized?). How do you set the optimization 
      and regularization parameters? How close to optimal should the 
      humans be? Can you comment on the performance if the human was 
      adversarial? In other words, to what extent can your assumptions on 
      optimality of human traces be relaxed? How do you estimate the kind of 
      expert from whom you are getting demonstrations? The intro is 
      clear and the approach is explained well. The authors have also 
      highlighted aspects of the approach in their experiments. By contrast, 
      the theoretical aspects of the work are not as self-contained. I 
      believe the paper would be improved by adding further relevant details 
      for that part of the paper (obviously, space is at a premium in 
      general).