Cambridge engineers build robots that learn to evolve and improve
Researchers and engineers at Cambridge University have developed a robot, which without any human intervention can improve the way it works over time. The robot has been trained to construct other smaller robots, evolving the manufacturing process each time it builds one to improve the next. The ultimate aim for the Cambridge team is to develop robots that can adapt to their surroundings.&nbsp;
Four agents were subject to further analysis of their locomotion patterns: The best agents in the last generation of experiments 1a, 1c and 2, and the second best of experiment 2. Each of these agents were equipped with three reflective markers and tracked with a motion capture system (NaturalPoint, OptiTrack). The trajectory of the midpoint of these markers was recorded while the agent moved on the same ground as in the experiment. From the tracking data, the trajectories in the horizontal plane were extracted and are shown in Fig 4B . While some candidates show stable locomotion patterns, others exhibit rather unsteady trajectories for forward locomotion.

In the course of the evolutionary process, many variations of morphologies were generated, eventually resulting in faster, i.e. more fit, individuals (see Table 2 ). For example, the mean fitness of the best three individuals in the first generation of experiment 1d was 2.8 cm/s, but it increased up to 6.7 cm/s in the last generation. The increase of performance is not only achieved by the fine tuning of design parameters, but also by “inventing” qualitatively different morphologies and gait patterns for locomotion.

In all experiments, the fitness increases relative to the initial generation (A). The lines show the upper quartile of the normalized fitnesses in each generation, with the errorbars indicating best and median fitness. The increasing fitness indicates that the evolutionary process applied to the initial population of robots improves their locomotion capabilities. The top view of the trajectories of four successful robots from experiments 1a, 1c and 2 shows that different locomotion strategies are applied (B). While most successful solutions result in a stable limit cycle, also more unsteady behavior (blue) can achieve a good performance.

In each of the five experiments, ten generations with ten robots were built and tested. For each robot of experiment 2, an image of its top-view at the beginning of the evaluation process is shown. The number on the top-left corner of each image indicates its fitness (cm/s). The lines between generations show the relations between robots, i.e. the method for generating the new genotype (solid black: elite; thin black: crossover; thin grey: mutation). Negative fitnesses and missing images indicate failure of the building process of the respective robot. The images show that various types of robots are tested, and the fitness of the robots increases in the course of the experiment.

The evolutionary process was applied to a population of ten agents over ten generations; i.e. 100 candidates were built and evaluated during an experimental run. An experiment starts with a set of agents, usually randomly generated. Variations of designs are built during evolution and an increase of performance can be observed after a few generations (see S1 Table for the genome data). The increase of fitness was quantitatively analyzed through five such experiments, in which 500 candidate agents were generated by parameter variation, and evaluated in three different environments ( Fig 3 and S2 Fig ). Except for the first two experiments, each agent was built and evaluated once. In about 96% of all trials, the construction process was successful and the performance of the agent can be analyzed. In the cases where construction fails, zero fitness is awarded to the agent, i.e. it is eliminated in the step to the next generation. In the generation maps, a negative error code is indicating the failure mode (-5: unspecified, -13: HMA connection failure, -14: collision during assembly, -16: other). The normalized fitness of the upper quartile in each generation is plotted in Fig 4A , and a fitness increase of more than 40% over ten generations was observed in all experiments as shown by the results in Table 2 . The results in Fig 4A show relatively large fitness variations over generations also for the fittest individuals, although elitism is employed. This is a result of the real-world testing, which includes uncertainties and lacks perfect repeatability. Nevertheless, the overall trends indicate a clear increase of the locomotion fitness during the optimization process.

Diversity and Physical Constraints

To achieve the observed increase of fitness through the course of evolution, a sufficient diversity of solutions is required. The variety of solutions is expressed in the various behaviors the candidate agents exhibit when interacting with the environment in the testing phase. To quantify the behavioral diversity in all five experiments, each agent’s behavior is classified according to four distinct features. Each unique combination of features is considered as a class of behaviors, and based on this classification the behavioral diversity is calculated as introduced in the Diversity Analysis section. The behavioral diversity for all experiments is listed in Table 2.

The categorization of behaviors relies on real-world testing, because the behavior only emerges through the interaction of the robot morphology and its controller with the environment. To enable for a further analysis, a second diversity measure, based on the agents’ morphologies is applied. Each agent is categorized according to its shape factor and size. These parameters can be calculated given an agent’s genome without physically assembling it. The morphological diversity calculated with these measures is also indicated in Table 2. The complete data including behavioral and morphological categorization for all real agents is listed in S1 Table. Behavioral and morphological diversities are two different measures, but both show similar trends. An increased morphological diversity is likely to result in more diverse behaviors, although there is no guarantee.

Many elements in our experiments can influence the emergence of diversity. Four main physical constraints in the automated construction process were identified that dominantly affect the degree of diversity generated in the evolutionary optimization. These are: (1) the payload limit of the mother robot which restricts the maximum weight of a candidate agent, (2) the reachable range of the mother robot that restricts the size of the candidate agent, (3) stability and fixation capabilities of the mother robot while a candidate agent is being built, and (4) the connection capabilities of the mother robot (e.g. whether the mother robot can connect vertical and horizontal surfaces, or only certain configurations).

To analyze the influence of these four physical constraints onto the diversity that can be achieved, 1.25 million genomes were randomly sampled with uniform parameter distributions and one to ten components. The genomes were then “built” in simulation to calculate the resulting morphologies. For the virtual developmental process, different sets of the abovementioned constraints were activated. First, all constraints were active as in the real experiments 1a-1d, then the stability constraint was removed as in the experiment 2, and eventually all four constraints were relaxed. The resulting morphologies were categorized according to their shape factor and size. From this categorization a theoretical morphological diversity is calculated (Table 2). With this set of randomly generated genomes, categorized based on the corresponding phenotypes and filtered according to the aforementioned four physical construction constraints, the emergence likelihood of each morphology type can be computed. The resulting probability distribution of morphologies is plotted in Fig 5A. In this case, the construction constraints are very strong, thus the distribution is biased towards morphologies with one or two elements and a high shape factor (i.e. the mother robot is less likely to be able to construct complex agents).

PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 5. Exploration of the design space. Each agent can be located in the design space given its number of elements and shape factor (A,B,C). The grey area shows the theoretical limit which can be reached using the cubic modules. The portion of this space that can be covered is restricted by the constraints which apply to the real-world construction process. The colored areas illustrate the parts of the design space that can be reached with different sets of active constraints. All constraints are active in (A), the stability condition is relaxed in (B) and no constraints are active in (C). This subsequently increases the reachable portion of the design space. The solid black markers indicate the distribution of the agents in experiment 1b (A) and experiment 2 (B). Their area is proportional to the number of agents in the bin. The right column (D-I) shows two example morphologies per experiment. https://doi.org/10.1371/journal.pone.0128444.g005

Such a restricted trend in evolutionary dynamics can also be observed in the real-world experiments. In the experiments 1a-d (see S2 Fig for the results), with the above physical constraints of the mother robot active throughout the evolutionary process, the populations are dominated by one morphology after several generations. In the experiments 1a and 1b, the resulting morphologies are very simple, and the results in Table 2 suggest that with a final fitness of only 2.9 cm/s, both experiments ended in a local maximum. The morphology distribution of experiment 1b is shown with solid markers in Fig 5A. The following experiments 1c and 1d result in more complex agents, but still their populations are taken over by one type of morphology, with few improvements in the last generations.

In contrast, the breakthroughs of morphological diversity and gait patterns were observed when the physical constraint (3) was relieved in experiment 2. The construction process was set up such that the robotic agent under construction can be manually fixed to ensure postural stability. The effect is shown in Fig 5B, in which the enhancement of morphological diversity can be observed through a wider distribution of morphologies. The real-world experimental run shown in Fig 3 employed this evolutionary configuration, and it can be seen that this experiment does not converge to a single solution, with the optimization still discovering new successful morphologies in the last generation. This experiment resulted in the highest real-world diversity and a final fitness of 7.2 cm/s.

Moreover, an additional theoretical analysis of the physical constraints on the construction process was conducted by also removing the three remaining constraints. Fig 5C shows that the probability distribution of morphologies spreads further out to lower shape factors and a larger number of elements, thus the morphological diversity can be enriched and more complex morphologies of agents theoretically be built. Even though the experimental setup does not allow to conduct such experiments, yet it theoretically demonstrates the scalability of the approach to develop richer morphological diversity with realistic modifications to the hardware setup of the mother robot.

Table 2 shows the computed diversity of all real-world and theoretical experiments introduced earlier. This analysis highlights that the physical construction constraints have a critical impact on the generation of diverse solutions, more than other elements such as environmental conditions. To maintain diverse populations throughout the optimization, which increases the chance to avoid local optima, these constraints must therefore be carefully considered.