The Body is not a Given: Joint Agent Policy Learning and Morphology Evolution
Reinforcement learning (RL) has proven to be a powerful paradigm for deriving complex behaviors from simple reward signals in a wide range of environments. When applying RL to continuous control agents in simulated physics environments, the body is usually considered to be part of the environment. However, during evolution the physical body of biological organisms and their controlling brains are co-evolved, thus exploring a much larger space of actuator/controller configurations. Put differently, the intelligence does not reside only in the agent*s mind, but also in the design of their body. We propose a method for uncovering strong agents, consisting of a good combination of a body and policy, based on combining RL with an evolutionary procedure. Given the resulting agent, we also propose an approach for identifying the body changes that contributed the most to the agent performance. We use the Shapley value from cooperative game theory to find the fair contribution of individual components, taking into account synergies between components. We evaluate our methods in an environment similar to the the recently proposed Robo-Sumo task, where agents in a 3D environment with simulated physics compete in tipping over their opponent or pushing them out of the arena. Our results show that the proposed methods are indeed capable of generating strong agents, significantly outperforming baselines that focus on optimizing the agent policy alone. A video is available at: www.youtube.com/watch?v=eei6Rgom3YY
 The paper is generally clear and well structured, with the exception of the proposed approach which is not clearly explained. Although not very novel, the topic considered is fairly interesting as it might tap into important questions about natural evolution. Pro: - The use of multi-agents cooperative game theory is interesting and well fit with the evolution of bodies. Cons: - This cooperative game theory formulation is not really explored in details and has little practical consequences to the experimental setting. - The contribution of the paper is very limited. The idea of jointly optimizing bodies and controllers has been extensively explored in the existing literature. Compared to this literature, I do not see any significant novelty being introduced. - The actual algorithm used is unclear to me. Where is SVG being used in algorithm 1? - The experimental results are weak (more details below). - The paper is often not sufficiently rigorous in its statements (see details below). In its current form, the paper does not seem to be a significant contribution to the literature. Minor points: - "the the" in the abstract - In the paragraph at the end of the first page and the beginning of the second page, there is a number of statements about the role of the body and intelligence. In a scientific article, this type of unsupported and hand-wavy statements is unacceptable and must either be supported by previous works or clearly indicated as a hypothesis. - What is a "physically simulated environment"? I guess that you refer to a software simulator, but this wording is non-sense. A physically simulated environment to me suggests that somebody physically simulates the environment with action-figures or something similar. - In the related work, you indicate a number of previous work from the evolutionary community without comparing them to your work. Instead, you briefly conclude that "in contrast, we use an RL algorithm". Since both evolutionary strategies and RL are nothing more than optimizers (and in particular, since your approach is also based on populations and evolution), you need to explain this connection significantly more in details. - One aspect that I find worthy of further explanations in the paper it is the use of self-play and a population of agents. This could be at the same time beneficial and a limitation. Thinking about and discussing both would be valuable. - please use a spell checker: "algoithm" - The experimental results in Sec 3.1 this moment only show that adding parameters of the body increase performances -- which is a fairly well-studied phenomenon in the literature. No comparisons to previous literature are performed (e.g., evolutionary strategies). Moreover, it naturally raises the question of how would an equal-weight fix-configuration robot perform. - Fig.5A should be in the number of iterations of similar metric. Using time can be misleading and unclear (is it the computational time or equivalent real-time?). Whatever metric you decide to use, you should accurately describe how it is computed. - It would be nice to have an intuitive explanation of what elo scores correspond to behaviorally. - statements such as the "in 64% of the runs" would benefit from including a plot of the whole distribution.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The Body is not a Given: Joint Agent Policy Learning and Morphology Evolution
Reinforcement learning (RL) has proven to be a powerful paradigm for deriving complex behaviors from simple reward signals in a wide range of environments. When applying RL to continuous control agents in simulated physics environments, the body is usually considered to be part of the environment. However, during evolution the physical body of biological organisms and their controlling brains are co-evolved, thus exploring a much larger space of actuator/controller configurations. Put differently, the intelligence does not reside only in the agent*s mind, but also in the design of their body. We propose a method for uncovering strong agents, consisting of a good combination of a body and policy, based on combining RL with an evolutionary procedure. Given the resulting agent, we also propose an approach for identifying the body changes that contributed the most to the agent performance. We use the Shapley value from cooperative game theory to find the fair contribution of individual components, taking into account synergies between components. We evaluate our methods in an environment similar to the the recently proposed Robo-Sumo task, where agents in a 3D environment with simulated physics compete in tipping over their opponent or pushing them out of the arena. Our results show that the proposed methods are indeed capable of generating strong agents, significantly outperforming baselines that focus on optimizing the agent policy alone. A video is available at: www.youtube.com/watch?v=eei6Rgom3YY
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper is generally clear and well structured, with the exception of the proposed approach which is not clearly explained. Although not very novel, the topic considered is fairly interesting as it might tap into important questions about natural evolution. Pro: - The use of multi-agents cooperative game theory is interesting and well fit with the evolution of bodies. Cons: - This cooperative game theory formulation is not really explored in details and has little practical consequences to the experimental setting. - The contribution of the paper is very limited. The idea of jointly optimizing bodies and controllers has been extensively explored in the existing literature. Compared to this literature, I do not see any significant novelty being introduced. - The actual algorithm used is unclear to me. Where is SVG being used in algorithm 1? - The experimental results are weak (more details below). - The paper is often not sufficiently rigorous in its statements (see details below). In its current form, the paper does not seem to be a significant contribution to the literature. Minor points: - "the the" in the abstract - In the paragraph at the end of the first page and the beginning of the second page, there is a number of statements about the role of the body and intelligence. In a scientific article, this type of unsupported and hand-wavy statements is unacceptable and must either be supported by previous works or clearly indicated as a hypothesis. - What is a "physically simulated environment"? I guess that you refer to a software simulator, but this wording is non-sense. A physically simulated environment to me suggests that somebody physically simulates the environment with action-figures or something similar. - In the related work, you indicate a number of previous work from the evolutionary community without comparing them to your work. Instead, you briefly conclude that "in contrast, we use an RL algorithm". Since both evolutionary strategies and RL are nothing more than optimizers (and in particular, since your approach is also based on populations and evolution), you need to explain this connection significantly more in details. - One aspect that I find worthy of further explanations in the paper it is the use of self-play and a population of agents. This could be at the same time beneficial and a limitation. Thinking about and discussing both would be valuable. - please use a spell checker: "algoithm" - The experimental results in Sec 3.1 this moment only show that adding parameters of the body increase performances -- which is a fairly well-studied phenomenon in the literature. No comparisons to previous literature are performed (e.g., evolutionary strategies). Moreover, it naturally raises the question of how would an equal-weight fix-configuration robot perform. - Fig.5A should be in the number of iterations of similar metric. Using time can be misleading and unclear (is it the computational time or equivalent real-time?). Whatever metric you decide to use, you should accurately describe how it is computed. - It would be nice to have an intuitive explanation of what elo scores correspond to behaviorally. - statements such as the "in 64% of the runs" would benefit from including a plot of the whole distribution.
assistant
 The paper is generally clear and well structured, with the exception of the proposed approach which is not clearly explained. Although not very novel, the topic considered is fairly interesting as it might tap into important questions about natural evolution. Pro: - The use of multi-agents cooperative game theory is interesting and well fit with the evolution of bodies. Cons: - This cooperative game theory formulation is not really explored in details and has little practical consequences to the experimental setting. - The contribution of the paper is very limited. The idea of jointly optimizing bodies and controllers has been extensively explored in the existing literature. Compared to this literature, I do not see any significant novelty being introduced. - The actual algorithm used is unclear to me. Where is SVG being used in algorithm 1? - The experimental results are weak (more details below). - The paper is often not sufficiently rigorous in its statements (see details below). In its current form, the paper does not seem to be a significant contribution to the literature. Minor points: - "the the" in the abstract - In the paragraph at the end of the first page and the beginning of the second page, there is a number of statements about the role of the body and intelligence. In a scientific article, this type of unsupported and hand-wavy statements is unacceptable and must either be supported by previous works or clearly indicated as a hypothesis. - What is a "physically simulated environment"? I guess that you refer to a software simulator, but this wording is non-sense. A physically simulated environment to me suggests that somebody physically simulates the environment with action-figures or something similar. - In the related work, you indicate a number of previous work from the evolutionary community without comparing them to your work. Instead, you briefly conclude that "in contrast, we use an RL algorithm". Since both evolutionary strategies and RL are nothing more than optimizers (and in particular, since your approach is also based on populations and evolution), you need to explain this connection significantly more in details. - One aspect that I find worthy of further explanations in the paper it is the use of self-play and a population of agents. This could be at the same time beneficial and a limitation. Thinking about and discussing both would be valuable. - please use a spell checker: "algoithm" - The experimental results in Sec 3.1 this moment only show that adding parameters of the body increase performances -- which is a fairly well-studied phenomenon in the literature. No comparisons to previous literature are performed (e.g., evolutionary strategies). Moreover, it naturally raises the question of how would an equal-weight fix-configuration robot perform. - Fig.5A should be in the number of iterations of similar metric. Using time can be misleading and unclear (is it the computational time or equivalent real-time?). Whatever metric you decide to use, you should accurately describe how it is computed. - It would be nice to have an intuitive explanation of what elo scores correspond to behaviorally. - statements such as the "in 64% of the runs" would benefit from including a plot of the whole distribution.