Humans are experts at high-fidelity imitation -closely mimicking a demonstration, often in one attempt. Humans use this ability to quickly solve a task instance, and to bootstrap learning of new tasks. Achieving these abilities in autonomous agents is an open problem. In this paper, we introduce an off-policy RL algorithm (MetaMimic) to narrow this gap. MetaMimic can learn both (i) policies for high-fidelity one-shot imitation of diverse novel skills, and (ii) policies that enable the agent to solve tasks more efficiently than the demonstrators. MetaMimic relies on the principle of storing all experiences in a memory and replaying these to learn massive deep neural network policies by off-policy RL. This paper introduces, to the best of our knowledge, the largest existing neural networks for deep RL and shows that larger networks with normalization are needed to achieve one-shot high-fidelity imitation on a challenging manipulation task. The results also show that both types of policy can be learned from vision, in spite of the task rewards being sparse, and without access to demonstrator actions.
Summary:
This paper proposes MetaMimic, an algorithm that does the following:
(i) Learn to imitate with high-fidelity with one-shot. The setting is that we have access to several demonstrations (only states, no actions) of the same task. During training, we have pixel observations plus proprioceptive measurements). At test time, the learned policy can imitate a single new demonstration (consisting of only pixel observations) of the same task.
(ii) When given access to rewards, the policy can exceed the human demonstrator by augmenting its experience replay buffer with the experience gained while learning (i). Therefore, even in a setting with sparse rewards and no access to expert actions (only states), the policy can learn to solve the task.
Overall Evaluation:
This is a good paper. In my opinion however, it does not pass the bar for ICLR.
Pros:
- The paper is well written. The contributions are clearly listed, the methods section is easy to follow and the authors explain the choices they make. The illustrations are clear and intuitive.
- The overview of hyperparameter choice and tuning / importance factor in the Appendix is useful.
- Interesting pipeline of learning policies that can use demonstrations without actions.
- The results on the simulated robot arm (block stacking task with two blocks) are good.
Cons:
- The abstracts oversells the contribution a bit when saying that MetaMimic can learn "policies for high-fidelity one-shot imitation of diverse novel skills". The setting that's considered in the paper is that of a single task, but different demonstrations (different humans from different starting points). This seems restrictive, and could have been motivated better.
- Experimental results are shown only for one task; block stacking with a robot arm in simulation.
- Might not be a good topical fit for ICLR, but more suited for a conference like CoRL or a workshop. The paper is very specific to imitation learning for a manipulation / control tasks, where we can (1) reset the environment to the exact starting position of the demonstrations, (2) the eucledian distance between states in the demonstration and visited by the policy is meaningful (3) we have access to both pixel observations and proprioceptive measurements. The proposed method is an elegant way to solve this, but it's unclear how well it would perform on different types of control problems, or when we want to transfer policies between different (but related) tasks.
Questions:
- Where does the "task stochasticity" come from? Only from the starting state, and from having different demonstrations? Or is the transition function also stochastic?
- The learned policy is able to do one-shot imitation, i.e., given a new demonstration (of the same task) the policy can follow this demonstration. Do I understand correct that this mean that there is *no* additional learning required at test time?
- It is not immediately clear to me why the setting of a single task but new demonstrations is interesting. Could the authors comment on this? One setting I could imagine is that the policy is trained in simulation, but then executed in the real-world, given a new demonstration. (If that's the main motivation though, then the experiments might have to support that this is possible - if no real-world robot is available, maybe the same simulator with a slightly different camera angle / light conditons or so.)
- The x-axis in the figures says "time (hours)" - is that computation time or simulated time?
Other Comments:
- In 3.2, I would be interested in seeing the following baseline comparison: Learn the test task from scratch using the one available demonstration, with the RL procedure (Equation 2, but possibly without the second term to make it fair). In Figure 5, we can see that the performance on the training tasks is much better when training on only 10 tasks, compared to 500. Then why not overfit to a single task, if that's what we're interested in? 
- An interesting baseline for 3.3 might be an RL algorithm with shaped rewards: using an additional reward term that is the eucledian distance to the *closest* datapoint from the demonstration. Compared to the baselines shown in the results section, this would be a fairer comparison because (1) unlike D4PG we also have access to information from the demonstrations and (2) no additional information is needed like the action information in D4PGfD and (3) we don't have the need for a curriculum.
Nitpick (no influence on score):
[1. Introduction]
- I find the first sentence, "One-shot imitation is a powerful way to show agents how to solve a task" a bit confusing. I'd say one-shot imitation is a method, not a way to show how to solve a task. Maybe an introductory sentence like "Expert demonstrations are a powerful way to show agents how to solve a task." works better?
- Second sentence, the chosen example is "manufacturing" tasks - do you mean manipulation? When reading this, I had to think of car manufacturing - a task I could certainly not imitate with just a few demonstrations.
- Add note that with "unconditional policy" you mean not conditioned on a demonstration.
[2. MetaMimic]
- [2.1] Third paragraph: write "Figure 2, Algorithm 1" or split the algorithm and figure up so you can refer to them separately.
- [2.1] Last paragraph, second line: remove second "to"