Imitation learning aims to learn an optimal policy from expert demonstrations and its recent combination with deep learning has shown impressive performance. However, collecting a large number of expert demonstrations for deep learning is time-consuming and requires much expert effort. In this paper, we propose a method to improve generative adversarial imitation learning by using additional information from non-expert demonstrations which are easier to obtain. The key idea of our method is to perform multiclass classification to learn discriminator functions where non-expert demonstrations are regarded as being drawn from an extra class. Experiments in continuous control tasks demonstrate that our method learns better policies than the generative adversarial imitation learning baseline when the number of expert demonstrations is small.
The paper proposes a modification of GAIL (Ho & Ermon, 2016) to make use of non-expert data. The non-expert data is used by training a classifier to classify between roll-outs of the current policy, expert demonstrations and non-expert demonstrations. Similar to GAIL, the policy is iteratively updated using TRPO with a cost that is given by the log probability of predicting the policy. The use of non-expert data acts as regularization in order to learn better features similar to universum prescription (Zhang & LeCun, 2017).  
The paper is well-written and very clear. The general problem setting is interesting, but I think it is of rather little significance, because I do not see many clear applications. The evaluation focuses on simulated robots, however gathering non-expert data on real robots would be very expensive, so the approach would not make a lot of sense here (even if we replace TRPO a more sample efficient rl method). The paper mentions the game of Go, but learning a policy on such large state spaces is not feasible without major modification and significant computational effort. However, the paper also mentions autonomous driving, which might be a more convincing application, because we can have a lot of demonstrations that we do not want to label as expert trajectories. I think the paper would profit a lot from having an experiment where the importance of making use of non-expert data becomes evident.
The approach seems sound, although I think that we can not expect much benefit from using the unlabelled data in the proposed way. By not making any assumptions on the non-expert data, they do not carry any information about the objective; the information that they carry about the system dynamics is not exploited for the RL update. Instead, the use of non-expert data is restricted to learning better features for discriminating between the agent and the expert. However, non-expert data is typically not cheaper than policy roll-outs and better features could also be learned by using more samples from the policy. The experiments also show only slight benefits, especially when comparing the final performance (instead of total returns) and accounting for the additional system interactions needed for generating the non-expert data. To make the comparison fairer, we could consider using a few more system interactions (variable K in the paper) per iteration for standard GAIL, so that the total number of function evaluations would match those of M-GAIL after a certain number of iterations. Especially if K is appropriately tuned, it is not clear whether we could still show an advantage of M-GAIL. It would also be interesting to show, whether we can benefit from using the policy roll-outs of previous iterations as non-expert data for the current iteration (in the traditional IL setting where no non-expert data is available a priori).
The main weakness of the paper is, that the novelty seems marginal. Instead of doing binary classification with cross-entropy loss, we're doing three-class classification with cross-entropy loss and use it for binary classification (by throwing away the auxiliary logit for predicting the non-expert class). Did I miss any other difference to GAIL? We can argue whether the policy objective is different (to me, H_\phi of M-GAIL corresponds to the discriminator of GAIL and the objective is exactly the same), however, even if we call it a minor modification, we would have very little novelty in the approach. As the paper does also not compensate for this with very good results or thorough theoretical analysis, I think that the contribution is too minor.
I do not see the point of section 4.4. and the related appendix A2. For all I understand, it proves that when using lambda=0 (standard GAIL, right?), the proof of Fu et al. (2018) for GAIL is valid (i.e. we learn a completely useless reward function that does not carry any additional information compared to the policy), and when using lambda!=0 we learn something different. I don't see the the purpose of this statement and I don't think that it needs to be proven. The paper argues, that for small lambda we can treat the discriminator logits as approximations of these (completely useless) reward functions--without providing any bound. As I do not see why this would be useful, I think the section should be removed. 
Minor:
Typo: "[...]due to its dependent[sic] on the linearity of reward functions and good feature engineering"