Deep Recursive Neural Networks for Compositionality in Language
Recursive neural networks comprise a class of architecture that can operate on structured input. They have been previously successfully applied to model compositionality in natural language using parse-tree-based structural representations. Even though these architectures are deep in structure, they lack the capacity for hierarchical representation that exists in conventional deep feed-forward networks as well as in recently investigated deep recurrent neural networks. In this work we introduce a new architecture -a deep recursive neural network (deep RNN) -constructed by stacking multiple recursive layers. We evaluate the proposed model on the task of fine-grained sentiment classification. Our results show that deep RNNs outperform associated shallow counterparts that employ the same number of parameters. Furthermore, our approach outperforms previous baselines on the sentiment analysis task, including a multiplicative RNN variant as well as the recently introduced paragraph vectors, achieving new state-of-the-art results. We provide exploratory analyses of the effect of multiple layers and show that they capture different aspects of compositionality in language.
This paper proposes the use of 'deep' recursive networks by adding depth across space, which is motivated by the success of doing the same thing with recurrent networks. Each layer has its own parameters that are shared across different heights within that layer. The authors also propose the use of dropout and ReLUs, while fixing the word representations to pre-trained embeddings. Experimentation is performed on the sentiment treebank of Socher et al.The proposed method is novel although essentially a generalization of the same approach used with recurrent nets. The authors achieve very strong performance on the sentiment treebank dataset. The performance gains obtained with dropout and ReLUs are interesting in itself. That being said, some of the model and experimental design choices are questionable (or not well justified) and I believe some additional experimentation is necessary before I can wholeheartedly recommend acceptance.The paper itself is clearly written for the most part and sufficient detail is given for the reader to reproduce the results on their own.Detailed comments / questions:- line 138: What is W^eta and W^xh? These haven't been defined. Are these just to refer to either W^{eta}_L and W^{eta}_R? It would be clearer to have two equations: one for the inputs and another for the hidden layers. It would also help to summarize the parameter space i.e. {W_L, W_R, U, b, c}. Including the dimensionalities of each of your matricies when you define them would also be helpful.- lines 148-154: This seems very speculative. Is this based off of your own empirical results? If so, you should mention this. I'm not convinced that this is actually an issue. - baseline: The authors should include the results of "A convolutional neural network for modelling sentences" (Kalchbrenner et al) from ACL this year.- How come you didn't fine-tune the word vectors? Finetuning them on a sentiment task allows the words embeddings to become reflective of their sentiment. It seems strange to have phrases and sentence vectors that are discriminative of sentiment but where individual word vectors are not. (I realize this is OK with a deep RNN since you could just forward-pass the word vectors across layers). Is it possible that the improvements you're getting with deep RNNs happening because you are not fine-tuning the words? Conceivably, a single layer RNN with fine-tuned embeddings should do better. I think this needs to be controlled for.- line 251 (shared dropout units): What was the motivation for this choice?- lines 255-256 (stability): Alternatively, you could try constraining the norms of the weights or truncating the gradients, as is sometimes done with recurrent nets.- binary classification: This should be done separately. The issue is that at training time, you are also including the neutral class examples which deviates from the experiment protocol. Consequently, these results are not directly comparable to the existing approaches.- The analogy with deep recurrent nets is reasonable when you are making predictions at each node in the tree, as is the case on the treebank experiments. It would have been useful to see whether or not there is an advantage to using deep RNNs when only a single, root label exists. Would you still expect to see an improvement over a single layer RNN?