We propose a general deep reinforcement learning method and apply it to robot manipulation tasks. Our approach leverages demonstration data to assist a reinforcement learning agent in learning to solve a wide range of tasks, mainly previously unsolved. We train visuomotor policies end-to-end to learn a direct mapping from RGB camera inputs to joint velocities. Our experiments indicate that our reinforcement and imitation approach can solve contact-rich robot manipulation tasks that neither the state-of-the-art reinforcement nor imitation learning method can solve alone. We also illustrate that these policies achieved zero-shot sim2real transfer by training with large visual and dynamics variations.
Paper summary: The authors propose a number of tricks to enable training policies for pick and place style tasks using a combination of GAIL-based imitation learning and hand-specified rewards, as well as use of unobserved state information during training and hand-designed curricula. The results demonstrate manipulation policies for stacking blocks and moving objects, as well as preliminary results for zero-shot transfer from simulation to a real robot for a picking task and an attempt at a stacking task.
Review summary: The paper proposes a limited but interesting contribution that will be especially of interest to practitioners, but the scope of the contribution is somewhat incremental in light of recent work, and the results, while interesting, could certainly be better. In the balance, I think the paper should be accepted, because it will be of value to practitioners, and I appreciate the detail and real-world experiments. However, some of the claims should be revised to better reflect what the paper actually accomplishes: the contribution is a bit limited in places, but that's *OK* -- the authors should just be up-front about it.
Pros:
- Interesting tasks that combine imitation and reinforcement in a logical (but somewhat heuristic) way
- Good simulated results on a variety of pick-and-place style problems
- Some initial attempt at real-world transfer that seems promising, but limited
- Related work is very detailed and I think many will find it to be a very valuable overview
Cons:
- Some of the claims (detailed below) are a bit excessive in my opinion
- The paper would be better if it was scoped more narrowly
- Contribution is a bit incremental and somewhat heuristic
- The experimental results are difficult to interpret in simulation
- The real-world experimental results are not great
- There are a couple of missing citations (but overall related work is great)
Detailed discussion of potential issues and constructive feedback:
> "Our approach leverages demonstration data to assist a reinforcement learning agent in learning to solve a wide range of tasks, mainly previously unsolved."
>> This claim is a bit peculiar. Picking up and placing objects is certainly not "unsolved," there are many examples. If you want image-based pick and place with demonstrations for example, see Chebotar '17 (not cited). If you want stacking blocks, see Nair '17. While it's true that there is a particular combination of factors that doesn't exactly appear in prior work, the statement the authors make is way too strong. Chebotar '17 shows picking and placing a real-world objective with a much higher success rate than reported here, without simulation. Nair '17 shows a much harder stacking task, but without images -- would that method have worked just as well with image-based distillation? Very likely. Rajeswaran '17 shows tasks that arguably are much harder. Maybe a more honest statement is that this paper proposes some tasks that prior methods don't show, and some prior methods show tasks that the proposed method can't solve. But as-is, this statement misrepresents prior work.
> Previous RL-based robot manipulation policies (Nair et al., 2017; Popov et al., 2017) largely rely on low-level states as input, or use severely limited action spaces that ignore the arm and instead learn Cartesian control of a simple gripper. This limits the ability of these methods to represent and solve more complex tasks (e.g., manipulating arbitrary 3D objects) and to deploy in real environments where the privileged state information is unavailable.
>> This is a funny statement. Some use images, some don't. There is a ton of prior work on RL-based robot manipulation that does use images. The current paper does use object state information during training, which some prior works manage to avoid. The comments about Cartesian control are a bit peculiar... the proposed method controls fingers, but the hand is simple. Some prior works have simpler grippers (e.g., Nair) and some have much more complex hands (e.g., Rajeswaran). So this one falls somewhere in the middle. That's fine, but again, this statement overclaims a bit.
> To sidestep the constraints of training on real hardware we embrace the sim2real paradigm which
has recently shown promising results (James et al., 2017; Rusu et al., 2016a).
>> Probably should cite Sadeghi et al. and Tobin et al. in regard to randomization, both of which precede James '17.
> we can, during training, exploit privileged information about the true system state
>> This was done also in Pinto et al. and many of the cited GPS papers
> our policies solve the tasks that the state-of-the-art reinforcement and imitation learning cannot solve
>> I don't think this statement is justified without much wider comparisons -- the authors don't attempt any comparisons to prior work, such as Chebotar '17 (which arguably is closest in terms of demonstrated behaviors), Nair '17 (which is also close but doesn't use images, though it likely could).
> An alternative strategy for dealing with the data demand is to train in simulation and transfer
>> Aside from previously mentioned citations, should probably cite Devin "Towards Adapting Deep Visuomotor Representations"
> Sec 3.2.1
>> This method seems a bit heuristic. It's logical, but can you say anything about what this will converge to? GAIL will try to match the demonstration distribution, and RL will try to maximize expected reward. What will this method do?
> Experiments
>> Would it be possible to indicate some measure of success rate for the simulated experiments? As-is, it's hard to tell how well either the proposed method or the baselines actually work.
> Transfer
>> My reading of the transfer experiments is that they are basically unsuccessful. Picking up a rectangular object with 80% success rate is not very good. The stacking success rate is too low to be useful. I do appreciate the authors trying out their method on a real robotic platform, but perhaps the more honest assessment of the outcome of these experiments is that the approach didn't work very well, and more research is needed. Again, it's *OK* to say this! Part of the purpose of publishing a paper is to stimulate future research directions. I think the transfer experiments should definitely be kept, but the authors should discuss the limitations to help future work address them, and present the transfer appropriately in the intro.
> Diverse Visuomotor Skills
>> I think this is a peculiar thing to put in the title. Is the implication that prior work is not diverse? Arguably several prior papers show substantially more diverse skills. It seems that all the skills here are essentially pick and place skills, which is fine (these are interesting skills), but the title seems like a peculiar jab at prior work not being "diverse" enough, which is simply misleading.