High-dimensional sparse reward tasks present major challenges for reinforcement learning agents. In this work we use imitation learning to address two of these challenges: how to learn a useful representation of the world e.g. from pixels, and how to explore efficiently given the rarity of a reward signal? We show that adversarial imitation can work well even in a high dimensional observation space. Surprisingly, the discriminator network, providing a learned reward function, can be tiny, comprising as few as 128 parameters, and can be easily trained using the most basic GAN formulation. Our approach removes limitations present in most contemporary imitation approaches: requiring no demonstrator actions (only video), no special initial conditions or warm starts, and no explicit tracking of any single demo. The proposed agent can solve a challenging robot manipulation task of block stacking from only video demonstrations and sparse reward, in which the non-imitating agents fail to learn completely. Furthermore, our agent learns much faster than competing approaches that depend on hand-crafted, staged dense reward functions, and also better compared to standard GAIL baselines. Finally, we develop a new adversarial goal recognizer that in some cases allows the agent to learn stacking without any task reward, purely from imitation.
minute differences between the agent and expert for a deep neural network discriminator to exploit. Furthermore, optimizing deep adversarial networks is notoriously difficult, although there is progress towards stabilizing this problem; see e.g. ; .
In this work, we show that if GAIL is provided the right kind of features, it can actually easily handle high-dimensional pixel observations using a single layer discriminator network. We can also improve its efficiency in terms of environment interactions by using a Deep Distributed Deterministic Policy Gradients (D4PG) agent , which is a state-of-the-art off-policy method for control, that can take advantage of a replay buffer to store past experiences.
We show that several types of features can be used successfully with a tiny, single-layer adversary, in cases where a deep adversary on pixels would fail completely. Specifically: self-supervised embeddings, e.g. Contrastive Predictive Coding (van den Oord et al., 2018); surprisingly, random projections through a deep residual network; and value network features from the D4PG agent itself. Concurrently with (Kostrikov et al., 2018), we additionally show how to modify GAIL for the off-policy case of D4PG agents with experience replay.
In our experiments, we demonstrate that our proposed approach is able to, from pixels, solve a challenging simulated robotic block stacking task using only demonstrations and a sparse binary reward indicating whether or not the stack has completed. Previous imitation approaches to this task have all used dense, staged task rewards crafted by humans and/or true state instead of pixels. In addition to reducing the dependency on hand-crafted rewards, our approach learns to stack faster than the dense staged reward baseline agent with the same amount of actor processes.
The main contributions of this paper are the following:
• A 6-DoF Jaco robot arm agent that learns block stacking from demonstration videos and sparse reward. On a simulated Jaco arm it achieves 94% success rate, compared to ∼ 29% using a behavior cloning agent with an equivalent number of demonstrations.
• An adversary-based early termination method for actor processes, that improves task performance and learning speed by creating a natural curriculum for the agent.
• An agent that learns with no task reward using an auxiliary goal recognizer adversary, achieving 55% stacking success from video imitation only.
• Ablation experiments on Jaco stacking as well as a 2D planar walker benchmark (Tassa et al., 2018), to understand the specific reasons for improvement in our agent. We find that random projections with a linear discriminator work surprisingly well in some cases, and that using value network features is even better.
The submission describes a sort of hybrid between reinforcement learning and imitation learning, where an auxiliary imitation learning objective helps to guide the RL policy given expert demonstrations.  The method consists of concurrently maximizing an RL objective--augmented with the GAIL discriminator as a reward—and minimizing the GAIL objective, which optimizes the discriminator between expert and policy-generated states.  Only expert states (not actions) are required, which allows the method to work given only videos of the expert demonstrations.  Experiments show that adding the visual imitation learning component allows RL to work with sparse rewards for complex tasks, in situations where RL without the imitation learning component fails.
Pros:
+ It is an interesting result that adding a weak visual imitation loss dramatically improves RL with sparse rewards 
+ The idea of a visual imitation signal is well-motivated and could be used to solve practical problems
+ The method enables an ‘early termination’ heuristic based on the imitation loss, which seems like a nice heuristic to speed up RL in practice
Cons:
+ It seems possible that imitation only helps RL where imitation alone works pretty well already
+ Some contributions are a bit muddled: e.g., the “learning with no task reward” section is a little confusing, because it seems to describe what is essentially a variant of normal GAIL
+ The presentation borders on hand-wavy at parts and may benefit from a clean, formal description
The submission tackles a real, well-motivated problem that would appeal to many in the ICLR community.  The setting is attractive because expert demonstrations are available for many problems, so it seems obvious that they should be leveraged to solve RL problems—especially the hardest problems, which feature very sparse reward signals.  It is an interesting observation that an imitation loss can be used as  a dense reward signal to supplement the sparse RL reward.  The experimental results also seem very promising, as the imitation loss seems to mean the difference between sparse-reward RL completely failing and succeeding.  Some architectural / feature selection details developed here seem to also be a meaningful contribution, as these factors also seem to determine the success or failure of the method.
My biggest doubt about the method is whether it really only works where imitation learning works pretty well already.  If we don’t have enough expert examples for imitation learning to work, or if the expert is not optimizing the given reward function, then it is possible that adding the imitation loss is detrimental, because it induces an undesirable bias.  If, on the other hand, we do have enough training examples for imitation learning to succeed and the expert is optimizing the given reward function, then perhaps we should just do imitation learning instead of RL.  So, it is possible that there is some sweet spot where this method makes sense, but the extent of that sweet spot is unclear to me.
The experiments are unclear on this issue for a few reasons.  First, figure 4 is confusing, as it is titled ‘comparison to standard GAIL', which makes it sound like a comparison to standard imitation learning.  However, I believe this figure is actually showing the performance of different variants of GAIL used as a subroutine in the hybrid RL-IL method.  I would like to know how much reward vanilla GAIL (without sparse rewards) achieves in this setting.  Second, figure 8 seems to confirm that some variant of vanilla imitation learning (without sparse rewards) actually does work most of the time, achieving results that are as good as some variants of the hybrid RL-IL method.  I think it would be useful to know, essentially, how much gain the hybrid method achieves over vanilla IL in different situations.
Another disappointing aspect of the paper is the ‘learning with no task reward’ section, which is a bit confusing.  The concept seems reasonable at a first glance, except that once we replace the sparse task reward with another discriminator, aren’t we firmly back in the imitation learning setting again?  So, the motivation for this section just seems a bit unclear to me.  This seems to be describing a variant of GAIL with D4PG for the outer optimization instead of TRPO, which seems like a tangent from the main idea of the paper.  I don’t think it is necessarily a bad idea to have another discriminator for the goal, but this part seems somewhat out of place.
On presentation: I think the presentation is a bit overly hand-wavy in parts.  I think the manuscript could benefit from having a concise, formal description.  Currently, the paper feels like a series of disjoint equations with unclear connections among them.  The paper is still intelligible, but not without knowing a lot of context relating to RL/IL methods that are trendy right now.  I feel that this is an unfortunate trend recently that should be corrected.  Also, I’m not sure it is really necessary to invoke “GAIL” to describe the IL component, since the discriminator is in fact linear, and the entropy component is dropped.  I think “apprenticeship learning” may be a more apt analogy.
On originality: as far as I can tell, the main idea of the work is novel.  The work consists mainly of combining existing methods (D4PG, GAIL) in a novel way.  However, some minor novel variations of GAIL are also proposed, as well as novel architectural considerations.
Overall, this is a nice idea applied to a well-motivated problem with promising results, although the exact regime in which the method succeeds could be better characterized.