Global Belief Recursive Neural Networks
Recursive Neural Networks have recently obtained state of the art performance on several natural language processing tasks. However, because of their feedforward architecture they cannot correctly predict phrase or word labels that are determined by context. This is a problem in tasks such as aspect-specific sentiment classification which tries to, for instance, predict that the word Android is positive in the sentence Android beats iOS. We introduce global belief recursive neural networks (GB-RNNs) which are based on the idea of extending purely feedforward neural networks to include one feedbackward step during inference. This allows phrase level predictions and representations to give feedback to words. We show the effectiveness of this model on the task of contextual sentiment analysis. We also show that dropout can improve RNN training and that a combination of unsupervised and supervised word vector representations performs better than either alone. The feedbackward step improves F1 performance by 3% over the standard RNN on this task, obtains state-of-the-art performance on the SemEval 2013 challenge and can accurately predict the sentiment of specific entities.
This paper introduces "belief propagation recursive neural	networks". This kind of networks extends the "feedforward" (bottom-up)	recursive networks with a backward (top-down) step during	inference. This allows phrase level predictions and embeddings to	give feedback to word embeddings and labels.  	This paper is overall well written. The model is well	motivated and described. This extends the idea of the	Bidirectional Recursive Neural Networks introduced by Irsoy	and Cardie 2008 (note that the reference to that paper is	incomplete). For instance one contribution is the introduction of hybrid word vectors.           Maybe this is the most important difference with previous work ?          Moreover, experimental comparisons are not completely fair: were the RNN and B-RNN also use hybrid vectors ? how the "best" models are tuned and selected ? How the single best is selected ? It is worth noticing that the others methods do not use an ensemble. These points must be clarified.         The term "belief propagation" is misleading and	maybe the term forward-backward could be well suited. 	Section 3.5 (Training) is a little bit too short and could	provide more details. For instance, I guess that the training	use the back-propagation through structure algorithm (the paper	of Goller 1996 could be cited). This algorithm implies that	the recursive model is unfolded. In this case, this yields a	very deep network to "reach" the word embedding part that is	updated. If I understand correctly, I think that the authors	could provide more details on this very important step.  	Section 3.6 is a little bit confused and the difference	between this model and the wok of Irsoy and Cardie must be	clarified. 	Section 4.1 must be improved to be understood by readers	outside of the NLP community. In the task 2 of Semeval 2013	there are 2 subtasks: contextual and message polarity.  I	guess that you address the contextual polarity task. With a	better presentation of the task, one can even understand the	parser constraints. 	Maybe, sections  4.3 and 4.4 could be merged.