The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown. We propose a new IL paradigm called Variational Imitation Learning with Diverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators' expertise with a probabilistic graphical model and estimate it along with a reward function. We show that a naive estimation approach is not suitable to large state and action spaces, and fix this issue by using a variational approach that can be easily implemented using existing reinforcement-learning methods. Experiments on continuous-control benchmarks and real-world crowdsourced demonstrations denote that VILD outperforms stateof-the-art methods. Our work enables scalable and data-efficient IL under more realistic settings than before.
This paper proposes an imitation learning algorithm for the setting where the demonstration data consists of trajectories from sources of varying expertise. The authors proceed by defining a parameterized model of the (demonstration) trajectory distribution (Equation 2), which uses the MaxEnt-RL model for the optimal policy, and a distribution (p_w) to model the level of expertise. Imitation learning is then reduced to maximum-likelihood training under the provided demonstrations. Using appropriate variational distributions and model specification, the MLE objective is transformed to the VILD objective (Equation 5), which can be optimized with gradient descent. Expertise-level (p_w) is modeled as a Gaussian blur over the optimal action, wherein the variance is correlated with expertise (lower is better). Furthermore, a truncated IS approach is proposed for learning a better reward function. It samples more frequently from the experts that have a higher estimated expertise.
Overall, I really enjoyed reading the paper. The writing and the presentation of material (both background and novel solutions) is clean and concise. The Appendix, with all the derivations and the summarizations of the related approaches, is very informative. I would like the authors to comment on the following:
1.	It is claimed in Section 1 that prior approaches for imperfect imitation learning rely on auxiliary information from the expert, in the form of confidence scores or ranking, while VILD doesn’t use any. In my opinion, the fact that VILD uses “labeled” expert demonstrations (i.e. each demonstration is tagged with a number {1..K}) classifies as auxiliary information. Contrary to approaches such as InfoGAIL, which infer the latent structure of the expert demonstrations in a completely unsupervised fashion, VILD fixes the demonstrations-model and instead attempts to learn the parameters corresponding to this model – this holds exactly for the Gaussian policy, and approximately for the TSD policy setting.
2.	The difference in performance of VILD w/ and w/o IS is surprising. I understand the motivation in Section 3.4 that IS should help to improve the convergence rate, but for benchmarks like HalfCheetah, Walker, the performance seems to have saturated to a significantly lower value. I would like to know if the authors have some thoughts on this wide discrepancy w/ and w/o IS. 
3.	Baselines – I’m not sure if InfoGAIL with a uniform prior on the context is a fair comparison to VILD-IS. Since VILD-IS changes the demonstrator sampling from uniform to expertise-dependent, one could do something similar for InfoGAIL – e.g. after training, report the best performing context, or sample context based on a performance-dependent distribution.   
4.	Sample-efficiency in terms of expert data – Is the number of trajectories that are collected from each of 10 demonstrators reported somewhere?