Improving Generative Adversarial Imitation Learning with Non-expert Demonstrations
Imitation learning aims to learn an optimal policy from expert demonstrations and its recent combination with deep learning has shown impressive performance. However, collecting a large number of expert demonstrations for deep learning is time-consuming and requires much expert effort. In this paper, we propose a method to improve generative adversarial imitation learning by using additional information from non-expert demonstrations which are easier to obtain. The key idea of our method is to perform multiclass classification to learn discriminator functions where non-expert demonstrations are regarded as being drawn from an extra class. Experiments in continuous control tasks demonstrate that our method learns better policies than the generative adversarial imitation learning baseline when the number of expert demonstrations is small.
 This paper proposes M-GAIL, which performs imitation learning from expert as well as sub-optimal demonstrations. This work builds off of GAIL (Ho et. al 2016) but modifies the discriminator with an additional class corresponding to sub-optimal demonstrations. The authors show empirically that including these sub-optimal demonstrations into the training process leads to faster and improved learning. While allowing the use of sub-optimal demonstrations is important and could have useful benefits in practice, I find the new algorithmic formulation unnecessary when compared to previous works. One of the theoretical benefits of MaxCausalEnt IRL (and by extension GAIL) is that because the model is probabilistic, not all demonstrations need to be reward-maximizing. The definition of optimality is relaxed from meaning exact reward maximization to coming from some optimal distribution over trajectories. Correct me if I am wrong, but in the case of this work, the proposed algorithm seems equivalent to adding the "sub-optimal" demonstrations to the expert demonstrations, but down-weighting each sub-optimal demonstration be a factor of .lambda. Thus, I don*t believe we need to introduce an entirely new algorithm, with the concept of a 3rd class, to solve this problem. The existing MaxEnt frameworks seem to handle the notion of sub-optimality proposed in this paper just as well, interpreting the "sub-optimal" demonstrations as low-probability expert demonstrations. This feels cleaner and more intuitive than the current proposed explanation as maximizing a mixture of two reward functions (in the IRL view) or minimizing occupancy measure divergence over 3 distributions (in the IL view). If this equivalence is true, I would encourage the authors to include this discussion in the main paper, and if not, discuss the differences and possibly compare against this simple strategy as a baseline. I find the empirical results quite interesting, even though the gains seem small. Including sub-optimal demonstrations to learn from could be a nice trick to improve the learning of GAIL-like algorithms, which is nice to know. I am curious if the authors tried annealing the .lambda term from 1.0 to 0.0, as lambda=1.0 is likely easier to learn from, but lambda=0.0 would have better asymptotic performance.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Improving Generative Adversarial Imitation Learning with Non-expert Demonstrations
Imitation learning aims to learn an optimal policy from expert demonstrations and its recent combination with deep learning has shown impressive performance. However, collecting a large number of expert demonstrations for deep learning is time-consuming and requires much expert effort. In this paper, we propose a method to improve generative adversarial imitation learning by using additional information from non-expert demonstrations which are easier to obtain. The key idea of our method is to perform multiclass classification to learn discriminator functions where non-expert demonstrations are regarded as being drawn from an extra class. Experiments in continuous control tasks demonstrate that our method learns better policies than the generative adversarial imitation learning baseline when the number of expert demonstrations is small.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes M-GAIL, which performs imitation learning from expert as well as sub-optimal demonstrations. This work builds off of GAIL (Ho et. al 2016) but modifies the discriminator with an additional class corresponding to sub-optimal demonstrations. The authors show empirically that including these sub-optimal demonstrations into the training process leads to faster and improved learning. While allowing the use of sub-optimal demonstrations is important and could have useful benefits in practice, I find the new algorithmic formulation unnecessary when compared to previous works. One of the theoretical benefits of MaxCausalEnt IRL (and by extension GAIL) is that because the model is probabilistic, not all demonstrations need to be reward-maximizing. The definition of optimality is relaxed from meaning exact reward maximization to coming from some optimal distribution over trajectories. Correct me if I am wrong, but in the case of this work, the proposed algorithm seems equivalent to adding the "sub-optimal" demonstrations to the expert demonstrations, but down-weighting each sub-optimal demonstration be a factor of .lambda. Thus, I don*t believe we need to introduce an entirely new algorithm, with the concept of a 3rd class, to solve this problem. The existing MaxEnt frameworks seem to handle the notion of sub-optimality proposed in this paper just as well, interpreting the "sub-optimal" demonstrations as low-probability expert demonstrations. This feels cleaner and more intuitive than the current proposed explanation as maximizing a mixture of two reward functions (in the IRL view) or minimizing occupancy measure divergence over 3 distributions (in the IL view). If this equivalence is true, I would encourage the authors to include this discussion in the main paper, and if not, discuss the differences and possibly compare against this simple strategy as a baseline. I find the empirical results quite interesting, even though the gains seem small. Including sub-optimal demonstrations to learn from could be a nice trick to improve the learning of GAIL-like algorithms, which is nice to know. I am curious if the authors tried annealing the .lambda term from 1.0 to 0.0, as lambda=1.0 is likely easier to learn from, but lambda=0.0 would have better asymptotic performance.
assistant
 This paper proposes M-GAIL, which performs imitation learning from expert as well as sub-optimal demonstrations. This work builds off of GAIL (Ho et. al 2016) but modifies the discriminator with an additional class corresponding to sub-optimal demonstrations. The authors show empirically that including these sub-optimal demonstrations into the training process leads to faster and improved learning. While allowing the use of sub-optimal demonstrations is important and could have useful benefits in practice, I find the new algorithmic formulation unnecessary when compared to previous works. One of the theoretical benefits of MaxCausalEnt IRL (and by extension GAIL) is that because the model is probabilistic, not all demonstrations need to be reward-maximizing. The definition of optimality is relaxed from meaning exact reward maximization to coming from some optimal distribution over trajectories. Correct me if I am wrong, but in the case of this work, the proposed algorithm seems equivalent to adding the "sub-optimal" demonstrations to the expert demonstrations, but down-weighting each sub-optimal demonstration be a factor of .lambda. Thus, I don*t believe we need to introduce an entirely new algorithm, with the concept of a 3rd class, to solve this problem. The existing MaxEnt frameworks seem to handle the notion of sub-optimality proposed in this paper just as well, interpreting the "sub-optimal" demonstrations as low-probability expert demonstrations. This feels cleaner and more intuitive than the current proposed explanation as maximizing a mixture of two reward functions (in the IRL view) or minimizing occupancy measure divergence over 3 distributions (in the IL view). If this equivalence is true, I would encourage the authors to include this discussion in the main paper, and if not, discuss the differences and possibly compare against this simple strategy as a baseline. I find the empirical results quite interesting, even though the gains seem small. Including sub-optimal demonstrations to learn from could be a nice trick to improve the learning of GAIL-like algorithms, which is nice to know. I am curious if the authors tried annealing the .lambda term from 1.0 to 0.0, as lambda=1.0 is likely easier to learn from, but lambda=0.0 would have better asymptotic performance.