Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning
The physical design of a robot and the policy that controls its motion are inherently coupled. However, existing approaches largely ignore this coupling, instead choosing to alternate between separate design and control phases, which requires expert intuition throughout and risks convergence to suboptimal designs. In this work, we propose a method that jointly optimizes over the physical design of a robot and the corresponding control policy in a model-free fashion, without any need for expert supervision. Given an arbitrary robot morphology, our method maintains a distribution over the design parameters and uses reinforcement learning to train a neural network controller. Throughout training, we refine the robot distribution to maximize the expected reward. This results in an assignment to the robot parameters and neural network policy that are jointly optimal. We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel robot designs and walking gaits for several different morphologies, achieving performance comparable to or better than that of hand-crafted designs.
 I*m glad to see the concept of jointly learning to control and evolve pop up again! Unfortunately, this paper has a number of weak points that - I believe - make it unfit for publication in its current state. Main weak points: - No comparisons to other methods (e.g. switch between policy optimization for the controller and CMA-ES for the mechanical parameters). The basic result of the paper is that allowing PPO to optimize more parameters, achieves better results... - One can argue that this is not true joint optimization Mechanical and control parameters are still treated differently. This begs the question: How should one define mechanical "variables" in order for them to behave similarly to other optimization variables (assuming that mechanical and control parameters influence the performance in a similar way)? Additional relevant papers (slightly different approach): http://www.pnas.org/content/108/4/1234.full#sec-1 http://ai2-s2-pdfs.s3.amazonaws.com/ad27/0104325010f54d1765fdced3af925ecbfeda.pdf Minor issues: Figure 1: please add labels/captions Figure 2: please label the axes
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning
The physical design of a robot and the policy that controls its motion are inherently coupled. However, existing approaches largely ignore this coupling, instead choosing to alternate between separate design and control phases, which requires expert intuition throughout and risks convergence to suboptimal designs. In this work, we propose a method that jointly optimizes over the physical design of a robot and the corresponding control policy in a model-free fashion, without any need for expert supervision. Given an arbitrary robot morphology, our method maintains a distribution over the design parameters and uses reinforcement learning to train a neural network controller. Throughout training, we refine the robot distribution to maximize the expected reward. This results in an assignment to the robot parameters and neural network policy that are jointly optimal. We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel robot designs and walking gaits for several different morphologies, achieving performance comparable to or better than that of hand-crafted designs.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 I*m glad to see the concept of jointly learning to control and evolve pop up again! Unfortunately, this paper has a number of weak points that - I believe - make it unfit for publication in its current state. Main weak points: - No comparisons to other methods (e.g. switch between policy optimization for the controller and CMA-ES for the mechanical parameters). The basic result of the paper is that allowing PPO to optimize more parameters, achieves better results... - One can argue that this is not true joint optimization Mechanical and control parameters are still treated differently. This begs the question: How should one define mechanical "variables" in order for them to behave similarly to other optimization variables (assuming that mechanical and control parameters influence the performance in a similar way)? Additional relevant papers (slightly different approach): http://www.pnas.org/content/108/4/1234.full#sec-1 http://ai2-s2-pdfs.s3.amazonaws.com/ad27/0104325010f54d1765fdced3af925ecbfeda.pdf Minor issues: Figure 1: please add labels/captions Figure 2: please label the axes
assistant
 I*m glad to see the concept of jointly learning to control and evolve pop up again! Unfortunately, this paper has a number of weak points that - I believe - make it unfit for publication in its current state. Main weak points: - No comparisons to other methods (e.g. switch between policy optimization for the controller and CMA-ES for the mechanical parameters). The basic result of the paper is that allowing PPO to optimize more parameters, achieves better results... - One can argue that this is not true joint optimization Mechanical and control parameters are still treated differently. This begs the question: How should one define mechanical "variables" in order for them to behave similarly to other optimization variables (assuming that mechanical and control parameters influence the performance in a similar way)? Additional relevant papers (slightly different approach): http://www.pnas.org/content/108/4/1234.full#sec-1 http://ai2-s2-pdfs.s3.amazonaws.com/ad27/0104325010f54d1765fdced3af925ecbfeda.pdf Minor issues: Figure 1: please add labels/captions Figure 2: please label the axes