Much of contemporary sensorimotor learning assumes that one is already given a complex agent (e.g., a robotic arm) and the goal is to learn to control it. In contrast, this paper investigates a modular co-evolution strategy: a collection of primitive agents learns to self-assemble into increasingly complex collectives in order to solve control tasks. Each primitive agent consists of a limb and a neural controller. Limbs may choose to link up to form collectives, with linking being treated as a dynamic action. When two limbs link, a joint is added between them, actuated by the 'parent' limb's controller. This forms a new 'single' agent, which may further link with other agents. In this way, complex morphologies can emerge, controlled by a policy whose architecture is in explicit correspondence with the morphology. In experiments, we demonstrate that agents with these modular and dynamic topologies generalize better to test-time environments compared to static and monolithic baselines. Project videos are available at https:// doubleblindICLR19.github.io/self-assembly/.
Summary:
--------------
The paper considers the problem of constructing compositional robotic morphologies that can solve different continuous control tasks in a (multi-agent) reinforcement learning setting. The authors created an environment where the actor consists of a number of primitive components which interface with each other via "linking" and construct a morphology of a robot. To learn in such an environment, the authors proposed a graph neural network policy architecture and showed that it is better than the baselines on the proposed tasks.
I find the idea of learning in environments with modular morphologies as well as the proposed tasks interesting. However, the major drawback of the paper is the lack of any reasonable details on the methods and experiments. It's hard to comment on the novelty of the architecture or the soundness of the method when such details are simply unavailable.
More comments and questions are below. I would not recommend publishing the paper in the current form.
Comments:
----------------
- If I understand it correctly, each component ("limb") represents an agent. Can you define precisely (ie mathematically) what the observations and actions of each agent are?
- Page 4, paragraph 2: in the inline equation, you write that a sum over actions equals policy applied to a sum over states. What does it mean? My understanding of monolithic agents is that observations and actions must be stacked together. Otherwise, the information would be lost.
- Page 4, paragraphs 3-(end of section): if I understand it correctly, the proposed method looks similar to the problem of "learning to communicate" in a cooperative multi-agent setting. This raises the question, how exactly the proposed architecture is trained? Is it joint learning and joint execution (ie there's a shared policy network, observation and action spaces are shared, etc), or not? All the details on how to apply RL to the proposed setup are completely omitted.
- Is the topology of the sub-agents restricted to a tree? Why so? How is it selected (in cases when it is not hand-specified)?
- From the videos, it looks like certain behaviors are very unphysical or unrealistic (eg parts jumping around and linking to each other). I'm wondering which kind of simulator was used? How was linking defined (on the simulator level)? It would be nice if such environments with modular morphologies were built using the standard simulators, such as MuJoCo, Bullet, etc.
All in all, despite potentially interesting ideas and setup, the paper is sloppily written, has mistakes, and lacks crucial details.