Imitation learning is an effective alternative approach to learn a policy when the reward function is sparse. In this paper, we consider a challenging setting where an agent has access to a sparse reward function and state-only expert observations. We propose a method which gradually balances between the imitation learning cost and the reinforcement learning objective. Built upon an existing imitation learning method, our approach works with state-only observations. We show, through navigation scenarios, that (i) an agent is able to efficiently leverage sparse rewards to outperform standard state-only imitation learning, (ii) it can learn a policy even when learner's actions are different from the expert, and (iii) the performance of the agent is not bounded by that of the expert due to the optimized usage of sparse rewards.
The draft proposes a heuristic combining environment rewards with an IRL-style rewards recovered from expert demonstrations, seeking to extend the GAIL approach to IRL to the case of mismatching action spaces between the expert and the learner. The interesting contribution is, in my opinion, the self-exploration parameter that reduces the reliance of learning on demonstrations once they have been learned sufficiently well.
Questions:
- In general, it's known that behavioral cloning, of which this work seem to be an example in so much it learns state distributions that are indistinguishable from the expert ones, can fail spectacularly because of the distribution shift (Kaariainen@ALW06, Ross&Bagnell@AISTATS10, Ross&Bagnell@AISTATS11). Can you comment if GAN-based methods are immune or susceptible to this?
   
- Would this work for tasks where the state-space has to be learned together with the policy? E.g. image captioning tasks or Atari games.
- Is it possible to quantify the ease of learning or the frequency of use of the "new" actions, i.e. $A^l \setminus A^e$?. Won't learning these actions effectively be as difficult as RL with sparse rewards? Say, in a grid world where 4-way diagonal moves allow reaching the goal faster, learner is king 8-way, demonstrations come from a 4-way expert, rewards are sparse and each step receives a -1 reward and the final goal is large positive -- does the learner's final policy actually use the diagonals and when?
Related work:
   
- Is it possible to make a connection to (data or policy) aggregation methods in IL. Such methods (e.g. Chang et al.@ICML15) can also sometimes learn policies better than the expert.
Experiments:
- why GAIL wasn't evaluated in Fig. 3 and Fig. 4?
Minor:
- what's BCE in algorithm 1?    
- Fig.1: "the the"
- sec 3.2: but avoid -> but avoids
- sec 3.2: be to considers -> be to consider
- sec 3.2: any hyperparameter -> any hyperparameters
- colors in Fig 2 are indistinguishable
- Table 1: headers saying which method is prior work and which is contribution would be helpful
- Fig. 3: if possible try to find a way of communicating the relation of action spaces between expert and learner (e.g. a subset of/superset of). Using the same figure to depict self-exploration make it complicated to analyse.
- sec 3.2: wording in the last paragraph on p.4 (positive scaling won't _make_ anything positive if it wasn't before)