ORDERED NEURONS: INTEGRATING TREE STRUCTURES INTO RECURRENT NEURAL NETWORKS
Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference 1 .
Language is hierarchically structured: smaller units (e.g., noun phrases) are nested within larger units (e.g., clauses). This is a strict hierarchy: when a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the different units of an LSTM can learn to track information at different time scales, the standard architecture does not impose this sort of strict hierarchy. This paper proposes to add this constraint to the system by ordering the units; a vector of "master" input and forget gates ensures that when a given unit is reset all of the units that follow it in the ordering are also reset.
Strengths:
* The paper introduces an elegant way of adding a hierarchical inductive bias; the intuition behind this idea is explained clearly.
* The evaluation tasks are very sensible. It's good that the model is shown to obtain good perplexity and slightly improve over an LSTM baseline; it's not the state of the art, but that's not the point of the paper (in fact, I would emphasize that even more than the authors do). The unsupervised parse evaluation (Table 2) is the heart of the paper, in my opinion (and should probably be emphasized more) -- the results from the second layer are quite impressive.
* The (mildly) better performance than LSTMs on long-distance dependencies, and (mildly) worse performance on local dependencies, in the Marvin & Linzen dataset, is interesting (and merits additional analysis).
Weaknesses:
* The discussion of the motivation for unsupervised structure induction in the introduction is somewhat confused. I am not sure that neural networks with latent syntactic structures can really address the seemingly very fundamental question mentioned in the first paragraph (whether syntax is related to "an underlying mechanism of human cognition") - I would suggest eliminating this part. At the same time, the authors might want to add another motivation for studying architectures that discover latent structure (as opposed to being given that structure) - this setting corresponds more closely to human language acquisition, where children aren't given annotated parse trees.
* The authors discuss hierarchy in terms of syntactic structure alone, but it would seem to me that the hierarchy that the LSTM is inducing could just as well include topic shifts, speech acts and others, especially if the network is trained across sentences.
* There is limited analysis of the model. Why does the second layer show better unsupervised parsing performance than the third layer? (Could this be related to syntactic vs. semantic/discourse units I mention in the previous bullet?) Why is the model better at ADJP boundaries than NP boundaries? It would have been more useful to report less experiments but analyze the results of each experiment in greater depth.
* In this vein, I am not sure it's useful to include WSJ10 in Table 2, which is busy as it is. These sentences are clearly too easy, as the right branching baseline shows, and require additional POS tagging.
* I found it difficult to read Figure A.2: could you help us understand what we should take away from it? 
* It is not entirely clear why the model needs both unit-specific forget/input gates and the "master" forget/input gates, and there is no discussion of this issue. Have you tried using only the "master" gates?
Minor notes:
* RNNGs are described as having an explicit bias to model syntactic structure; this is an arguably confusing use of the word "bias", in that the architecture has a hard constraint enforcing syntactic structures (bias implies a soft constraint).
* There are some language issues: agreement errors (e.g. "have" in the sentence that starts with "Developing" in the introduction), typos ("A order should exist", "co-occurance"), determiner issues ("values in [the] master forget gate", "when the overlap exists") - I would suggest going through and copy editing the paper.
* "cummax" seems like a better choice of name for cumulative maximum than "cumax".
* It may be helpful to remind the reader of the update equation for c_t in a standard LSTM.
* Did the language model have 1150 units in each layer or in total? Why did you use exactly three layers? Did you try one, two and four?
* It's not clear if the results in Table 2 reflect the best seed out of five (as the title of the column "max" indicates) or the average (as the caption says).