Hardware Conditioned Policies for Multi-Robot Transfer Learning
Deep reinforcement learning could be used to learn dexterous robotic policies but it is challenging to transfer them to new robots with vastly different hardware properties. It is also prohibitively expensive to learn a new policy from scratch for each robot hardware due to the high sample complexity of modern state-ofthe-art algorithms. We propose a novel approach called Hardware Conditioned Policies where we train a universal policy conditioned on a vector representation of robot hardware. We considered robots in simulation with varied dynamics, kinematic structure, kinematic lengths and degrees-of-freedom. First, we use the kinematic structure directly as the hardware encoding and show great zero-shot transfer to completely novel robots not seen during training. For robots with lower zero-shot success rate, we also demonstrate that fine-tuning the policy network is significantly more sample-efficient than training a model from scratch. In tasks where knowing the agent dynamics is important for success, we learn an embedding for robot hardware and show that policies conditioned on the encoding of hardware tend to generalize and transfer well. Videos of experiments are available at: https://sites.google.com/view/robot-transfer-hcp.
Disclaimer: my background is in control theory and only recently I have invested most of time in reading and doing research in the area of machine learning and reinforcement learning with specific focus on robotics and control. I went through the submitted paper carefully, including the supplementary material. Therefore I am quite confident with my assessment, especially since the problem that the addressed problem is well inside my core expertise (adaptive control). As I previously said, I am very confident with the problem, less confident with the theoretical framework (reinforcement learning) used to solve it. The math presented in the paper is relatively shallow and carefully checked. I am quite confident with the used reinforcement learning tools (PPO, DDPG, DDPG+HER).   = MAJOR COMMENTS =  1. The problem addressed in the paper is interesting. The very interesting part of the paper is the implicit encoding which proposes a solution to condition the control policy on quantities that are not directly measurable (e.g. dynamic properties). In the area of robotics, a similar problem has been solved with adaptive control (Slotine 1988, “Adaptive manipulator control: A case study”) where dynamic parameters (link masses, inertias and centers of mass, joint damping and friction) are estimated on-line using their linearity. The mathematical details of the proposed implicit encoding are confined in a relatively short subsection (Section 4.2 "Implicit encoding"). This section should be expanded and presented in depth. Details on how the cost function $L$ is constructed should be added so that the reader can appreciate what is novel with respect to (Slotine 1988).  - The paper would be very difficult to replicate due to the lack of details. Here is a list of things that should be better described/detailed in order to improve the paper repeatability. Page 3. Section 4 (first paragraph). What is the structure of the policy network? What is the state $s_t$ (e.g. joint positions and velocities)?  Page 4. Section 4.2. What is the cost function $L$? What is the vector of parameters $\theta$? This is fundamental to understand how $v_h$ is estimated from the available measurements ($\theta$?).   = MINOR COMMENTS = Appendix B, Table 2. How is the length of a link measured (e.g. how are these parameters related to the Denavit-Hartenberg parameters)? What is the meaning of the armature of a joint?   Page 2, Section 2 "Related work". A section on adaptive control for robotics should be added. A source of references can be found in (Slotine 1988) and in the book "Applied nonlinear control" by Jean-Jacques Slotine 1991.  Page 3, Section 4.1 "Explicit encoding". The adopted explicit encoding is quite subjective and information-dense. It is subjective in the sense that the position of the local coordinate systems is totally arbitrary and the proposed results might depend on the specific choice of these coordinate systems. It is information-dense because it includes all the coordinate systems across the kinematic chain. It would be good to test how the proposed approach behaves when reducing this information, possibly reducing it to the end-effector pose only.