Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.
 The paper proposes a new RNN unit: ON-LSTM. The idea is to explicitly integrates the latent tree structure into recurrent models. Experiments are conducted to evaluate performances on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference. Good results on unsupervised parsing show that the model learns something close to human judgments of the sentence parses. The paper is clearly written, and the experiments seem planned well. The language modeling results are not state-of-the-art, but the unsupervised parsing results of layer 2 are quite impressive. The analyses are reasonable. Overall, the paper seems worthy of being accepted.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper proposes a new RNN unit: ON-LSTM. The idea is to explicitly integrates the latent tree structure into recurrent models. Experiments are conducted to evaluate performances on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference. Good results on unsupervised parsing show that the model learns something close to human judgments of the sentence parses. The paper is clearly written, and the experiments seem planned well. The language modeling results are not state-of-the-art, but the unsupervised parsing results of layer 2 are quite impressive. The analyses are reasonable. Overall, the paper seems worthy of being accepted.
assistant
 The paper proposes a new RNN unit: ON-LSTM. The idea is to explicitly integrates the latent tree structure into recurrent models. Experiments are conducted to evaluate performances on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference. Good results on unsupervised parsing show that the model learns something close to human judgments of the sentence parses. The paper is clearly written, and the experiments seem planned well. The language modeling results are not state-of-the-art, but the unsupervised parsing results of layer 2 are quite impressive. The analyses are reasonable. Overall, the paper seems worthy of being accepted.