From Nodes to Networks: Evolving Recurrent Neural Networks
Gated recurrent networks such as those composed of Long Short-Term Memory (LSTM) nodes have recently been used to improve state of the art in many sequential processing tasks such as speech recognition and machine translation. However, the basic structure of the LSTM node is essentially the same as when it was first conceived 25 years ago. Recently, evolutionary and reinforcement learning mechanisms have been employed to create new variations of this structure. This paper proposes a new method, evolution of a tree-based encoding of the gated memory nodes, and shows that it makes it possible to explore new variations more effectively than other methods. The method discovers nodes with multiple recurrent paths and multiple memory cells, which lead to significant improvement in the standard language modeling benchmark task. Remarkably, this node did not perform well in another task, music modeling, but it was possible to evolve a different node that did, demonstrating that the approach discovers customized structure for each task. The paper also shows how the search process can be speeded up by training an LSTM network to estimate performance of candidate structures, and by encouraging exploration of novel solutions. Thus, evolutionary design of complex neural network structures promises to improve performance of deep learning architectures beyond human ability to do so.
 A genetic algorithm is used to do an evolutionary architecture search to find better tree-like architectures with multiple memory cells and recurrent paths. To speed up search, an LSTM based seq2seq framework is also developed that can predict the final performance of the child model based on partial training results. The algorithms and intuitions based on novelty search are interesting and there are improvements over baseline NAS model with the same architecture search space. Although, the experiments are not compared against latest architectures and best results. For example on PTB, there are new architectures such as those created by ENAS that result in much lower perplexity than best reported in Table 1, for the same parameter size. While you have mentioned ENAS in the related work, the lack of a comparison makes it hard to evaluate the true benefit if this work compared with existing literature. There is no clear abolition study for the Meta-LSTM idea. Figure 4 provides some insights but it*d be good if some experiments were done to show clear wins over baseline methods that do not employ performance prediction. There are many typos and missing reference in the paper that needs to be fixed.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
From Nodes to Networks: Evolving Recurrent Neural Networks
Gated recurrent networks such as those composed of Long Short-Term Memory (LSTM) nodes have recently been used to improve state of the art in many sequential processing tasks such as speech recognition and machine translation. However, the basic structure of the LSTM node is essentially the same as when it was first conceived 25 years ago. Recently, evolutionary and reinforcement learning mechanisms have been employed to create new variations of this structure. This paper proposes a new method, evolution of a tree-based encoding of the gated memory nodes, and shows that it makes it possible to explore new variations more effectively than other methods. The method discovers nodes with multiple recurrent paths and multiple memory cells, which lead to significant improvement in the standard language modeling benchmark task. Remarkably, this node did not perform well in another task, music modeling, but it was possible to evolve a different node that did, demonstrating that the approach discovers customized structure for each task. The paper also shows how the search process can be speeded up by training an LSTM network to estimate performance of candidate structures, and by encouraging exploration of novel solutions. Thus, evolutionary design of complex neural network structures promises to improve performance of deep learning architectures beyond human ability to do so.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 A genetic algorithm is used to do an evolutionary architecture search to find better tree-like architectures with multiple memory cells and recurrent paths. To speed up search, an LSTM based seq2seq framework is also developed that can predict the final performance of the child model based on partial training results. The algorithms and intuitions based on novelty search are interesting and there are improvements over baseline NAS model with the same architecture search space. Although, the experiments are not compared against latest architectures and best results. For example on PTB, there are new architectures such as those created by ENAS that result in much lower perplexity than best reported in Table 1, for the same parameter size. While you have mentioned ENAS in the related work, the lack of a comparison makes it hard to evaluate the true benefit if this work compared with existing literature. There is no clear abolition study for the Meta-LSTM idea. Figure 4 provides some insights but it*d be good if some experiments were done to show clear wins over baseline methods that do not employ performance prediction. There are many typos and missing reference in the paper that needs to be fixed.
assistant
 A genetic algorithm is used to do an evolutionary architecture search to find better tree-like architectures with multiple memory cells and recurrent paths. To speed up search, an LSTM based seq2seq framework is also developed that can predict the final performance of the child model based on partial training results. The algorithms and intuitions based on novelty search are interesting and there are improvements over baseline NAS model with the same architecture search space. Although, the experiments are not compared against latest architectures and best results. For example on PTB, there are new architectures such as those created by ENAS that result in much lower perplexity than best reported in Table 1, for the same parameter size. While you have mentioned ENAS in the related work, the lack of a comparison makes it hard to evaluate the true benefit if this work compared with existing literature. There is no clear abolition study for the Meta-LSTM idea. Figure 4 provides some insights but it*d be good if some experiments were done to show clear wins over baseline methods that do not employ performance prediction. There are many typos and missing reference in the paper that needs to be fixed.