A noisy and diverse demonstration set may hinder the performances of an agent aiming to acquire certain skills via imitation learning. However, state-of-the-art imitation learning algorithms often assume the optimality of the given demonstration set. In this paper, we address such optimal assumption by learning only from the most suitable demonstrations in a given set. Suitability of a demonstration is estimated by whether imitating it produce desirable outcomes for achieving the goals of the tasks. For more efficient demonstration suitability assessments, the learning agent should be capable of imitating a demonstration as quick as possible, which shares similar spirit with fast adaptation in the meta-learning regime. Our framework, thus built on top of Model-Agnostic Meta-Learning, evaluates how desirable the imitated outcomes are, after adaptation to each demonstration in the set. The resulting assessments hence enable us to select suitable demonstration subsets for acquiring better imitated skills. The videos related to our experiments are available at https://sites.google.com/view/deepdj.
The problem is described as doing imitation learning from a set of demonstrations that includes useless behavior. Authors propose a method that is an extension of MAML which selects the useful demonstrations by their provided performance gains at the meta-training time.
Paper clearly demonstrates significant amount of work. Pieces from different modern method implementations (like MAML, TRPO, GAIL, multiple custom loss functions) are combined to work together. Also four custom task domains are implemented with MuJoCo. Finally decent amount of experiments are run.
Unfortunately, all that hard work can't be justified by the motivations that are very artificial in details and by the final task performance.
First of all, the setup includes small number of demonstrations where almost none of them are seemingly successful (judging by the videos). This is a very artificial setting that does not reflect the actual imitation learning problems like demonstrations provided by humans. There, normally the problem is either dealing with small number of demonstrations that are all typically successful but similarly suboptimal or dealing with small number of distinct demonstrators which are again successful but have significantly different styles. In the summary video, authors motivate the case by learning from sources like internet videos, but that setting is also very far away from the case here, because such video collections are much larger but more importantly the main problem is dealing with the third person perspective. All the experiments here is done from first person demonstrations (in one case with a slightly different body).
Biggest caveat of the paper is that it is promoted as a purely imitation learning method. Yet everything hinges on the existence of a "task heuristic" which is nothing but a reward function. If such function exists, all these first person demonstrations can be judged and selected based on that function. There would be no need for a complicated meta-learning scheme. Also the task could be trained directly on that reward by reinforcement learning. Also computation of this heuristic function is not specified. As far as I understand, it is a different quantity than the sparse "Task Success Reward".
Finally, the final performance of the imitating agents are far from accomplishing the task, though they show some resemblance to the imitation behavior. This is not all that surprising, given small number of demonstrations and high dimensional control problems.
Overall, the details of the setup makes the problem very artificial, the final performance is not impressive. Method is an amalgamation of bunch other recent work, which gives the impression of creating complexity for its own sake. I do not think that this method will be useful for moving the field forward and produce any impact.