Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging experience from learning similar tasks. In the presence of task ambiguity or unobserved dynamics, demonstrations alone may not provide enough information; an agent must also try the task to successfully infer a policy. In this work, we propose a method that can learn to learn from both demonstrations and trial-anderror experience with sparse reward feedback. In comparison to meta-imitation, this approach enables the agent to effectively and efficiently improve itself autonomously beyond the demonstration data. In comparison to meta-reinforcement learning, we can scale to substantially broader distributions of tasks, as the demonstration reduces the burden of exploration. Our experiments show that our method significantly outperforms prior approaches on a set of challenging, vision-based control tasks.
The paper introduces a meta-learning approach that can learn from demonstrations and subsequent reinforcement learning trials. The approach works in two phases. First, multiple demonstrations are collected for each task and a meta-learning policy is obtained by embedding the demonstrations of a single task in a context vector that is given to the actor policy.  Subsequently, the meta-policy is used to collect trajectories which are also evaluated with a (sparse) reward. These trajectories, along with the demonstrations, are again used by a second meta-learning policy (with a concatenated context from demonstrations and trajectories) to obtain the final policy. The algorithm is tested on a simple point reaching task as well as on a more complex 3d physics simulation that contains pick-and-place tasks, pushing tasks and button pressing.
The approach seems really interesting and I think combining demonstrations with reinforcement learning for meta learning is a very promising approach. This is also underlined by the experimental results presented in the paper. The only concern I have is the presentation of the paper. The algorithmic description is very short and many things are left unclear. I would also like to see a bit more ablation studies in the paper. More comments below:
- Just from the text it is not clear how the meta-learning actually works and you have to study the figures in detail to understand that a context is extracted from the demonstrations as well as the RL trajectories. The context should be better introduced (even though this has been shown already in different papers, I think it is a good strategy to write the  
in a self-contained way).
- It is unclear to me how the reward signal is used to learn the meta-policy is phase 2. I understand that it is concatenated to the embedding before forming the context (again this is not really described in text but only in the figure), but it is quite unclear to me how the reward is used as optimality criteria. How do you reinforce good trajectories and decrease the influence of poor trajectories? This is maybe one of the most important parts of the paper and needs to be described in much more detail.
- The reward functions are not properly explained in the paper, even not for the toy task (reaching). This should at least be done in the appendix.
- I would like to see more ablation studies. For example, it is mentioned that 40 observations are used from the demonstrations and the trajectories. How is this number picked and how does it influence the performance? Also, typically, how long is a single trajectory? It is also unclear to me, if we take 40 random observations and the reward signal is sparse, wouldn't it be quite likely that the observations do not contain any reward values?
- How does K influence the performance of the algorithm?