In this paper we study the problem of answering cloze-style questions over documents. Our model, the Gated-Attention (GA) Reader, integrates a multi-hop architecture with a novel attention mechanism, which is based on multiplicative interactions between the query embedding and the intermediate states of a recurrent neural network document reader. This enables the reader to build query-specific representations of tokens in the document for accurate answer selection. The GA Reader obtains state-of-the-art results on three benchmarks for this task–the CNN & Daily Mail news stories and the Who Did What dataset. The effectiveness of multiplicative interaction is demonstrated by an ablation study, and by comparing to alternative compositional operators for implementing the gated-attention.
1 INTRODUCTION
A recent trend to measure progress towards machine reading is to test a system’s ability to answer questions about a document it has to comprehend. Towards this end, several large-scale datasets of cloze-style questions over a context document have been introduced recently, which allow the training of supervised machine learning systems (Hermann et al., 2015; Hill et al., 2015; Onishi et al., 2016). Such datasets can be easily constructed automatically and the unambiguous nature of their queries provides an objective benchmark to measure a system’s performance at text comprehension.
Deep learning models have recently been shown to outperform traditional shallow approaches on text comprehension tasks (Hermann et al., 2015). The success of many recent models can be attributed primarily to two factors: (1) Multi-hop architectures allow a (Weston et al., 2014; Sordoni et al., 2016; Shen et al., 2016), model to scan the document and the question iteratively for multiple passes. (2) Attention mechanisms, (Weston et al., 2014; Chen et al., 2016; Hermann et al., 2015) borrowed from the machine translation literature (Bahdanau et al., 2014), allow the model to focus on appropriate subparts of the context document. Intuitively, the multi-hop architecture allows the reader to incrementally refine token representations, and the attention mechanism re-weights different parts in the document according to their relevance to the query.
The effectiveness of multi-hop reasoning and attentions have been explored orthogonally so far in the literature. In this paper, we focus on combining both in a complementary manner, by designing a novel attention mechanism which gates the evolving token representations across hops. More specifically, unlike existing models where the query attention is applied either token-wise (Hermann et al., 2015; Kadlec et al., 2016; Chen et al., 2016; Hill et al., 2015) or sentence-wise (Weston et al., 2014; Sukhbaatar et al., 2015) to allow weighted aggregation, the Gated-Attention (GA) module proposed in this work allows the query to directly interact with each dimension of the token embeddings at the semantic-level, and is applied layer-wise as information filters during the multi-hop representation learning process. Such a fine-grained attention enables our model to learn conditional token representations with respect to the given question, leading to accurate answer selections.
We show in our experiments that the proposed GA reader, despite its relative simplicity, consistently improves over a variety of strong baselines on three benchmark datasets1. Our key contribution, ∗ BD and HL contributed equally to this work. 1Code for GA reader is available on github: https://github.com/bdhingra/ga-reader
the GA module, provides a significant improvement when the dataset size is large. Qualitatively, visualization of the attentions at intermediate layers of the GA reader shows that in each layer the GA reader attends to distinct salient aspects of the query which help in determining the answer.
2 RELATED WORK
The cloze-style QA task involves tuples of the form (d, q, a, C), where d is a document (context), q is a query over the contents of d, in which a phrase is replaced with a placeholder, and a is the answer to q, which comes from a set of candidates C. In this work we consider datasets where each candidate c ∈ C has at least one token which also appears in the document. The task can then be described as: given a document-query pair (d, q), find a ∈ C which answers q. Below we provide an overview of representative neural network architectures which have been applied to this problem.
LSTMs with Attention: Several architectures introduced in (Hermann et al., 2015) employ LSTM units to compute a combined document-query representation g(d, q), which is used to rank the candidate answers. Their techniques include the DeepLSTM Reader which performs a single forward pass through the concatenated (document, query) pair to obtain g(d, q); the Attentive Reader which first computes a document vector d(q) by a weighted aggregation of words according to attentions based on q, and then combines d(q) and q to obtain their joint representation g(d(q), q); and the Impatient Reader where the document representation is built incrementally. The architecture of the Attentive Reader has been simplified recently in Stanford Attentive Reader, where shallower recurrent units were used with a bilinear form for the query-document attention (Chen et al., 2016).
Attention Sum: The Attention-Sum (AS) Reader (Kadlec et al., 2016) uses two bi-directional GRU networks (Cho et al., 2014) to encode both d and q into vectors, similar to Stanford AR. A probability distribution over the entities in d is obtained by computing dot products between q and the entity embeddings and taking a softmax. An aggregation scheme named pointer-sum attention is further applied to sum the probabilities of the same entity, so that frequent entities the document will be favored compared to rare ones. Building on the AS Reader, the Attention-over-Attention (AoA) Reader (Cui et al., 2016) introduces a two-way attention mechanism where the query and the document are mutually attentive to each other.
Mulit-hop Architectures: Memory Networks (MemNets) were proposed in (Weston et al., 2014), where each sentence in the document is encoded to a memory by aggregating nearby words. Attention over the memory slots given the query is used to compute an overall memory and to renew the query representation over multiple iterations, allowing certain types of reasoning over the salient facts in the memory and the query. Neural Semantic Encoders (NSE) (Munkhdalai & Yu, 2016a) extended MemNets by introducing a write operation which can evolve the memory over time during the course of reading. Iterative reasoning has been found effective in several more recent models, including the Iterative Attentive Reader (Sordoni et al., 2016) and ReasoNet (Shen et al., 2016). The latter allows a dynamic number of reasoning steps and is trained with reinforcement learning.
Other related works include Dynamic Entity Representation network (DER) (Kobayashi et al., 2016), which builds dynamic representations of the candidate answers while reading the document, and accumulates the information about an entity by max-pooling. EpiReader (Trischler et al., 2016) consists of two networks, where one proposes a small set of candidate answers, and the other reranks the proposed candidates conditioned on the query and the context. (Bajgar et al., 2016) showed a 10% improvement on the CBT corpus (Hill et al., 2015) by training the AS Reader on an augmented training set of about 14 million examples, making a case for community to exploit data abundance. The focus of this paper, however, is on designing models which exploit the available data efficiently.
3 GATED-ATTENTION READER
3.1 MOTIVATION
Our proposed GA readers perform multiple hops over the document (context), similar to the Memory Networks architecture (Sukhbaatar et al., 2015). Multi-hop architectures mimic the multi-step comprehension process of human readers, and have shown promising results in several recent models for text comprehension (Sordoni et al., 2016; Kumar et al., 2015; Shen et al., 2016). The contextual representations in GA readers, namely the embeddings of words in the document, are iteratively
refined across hops until reaching a final attention-sum module (Kadlec et al., 2016) which maps the contextual representations in the last hop to a probability distribution over candidate answers.
The attention mechanism has been introduced recently to model human focus, leading to significant improvement in machine translation and image captioning (Bahdanau et al., 2014; Mnih et al., 2014). In reading comprehension tasks, ideally, the semantic meanings carried by the contextual embeddings should be aware of the query across hops. As an example, human readers are able to keep the question in mind during multiple passes of reading, to successively mask away information irrelevant to the query. However, existing neural network readers are restricted to either attend to tokens (Hermann et al., 2015; Chen et al., 2016) or entire sentences (Weston et al., 2014), with the assumption that certain sub-parts of the document are more important than others. In contrast, we propose a finer-grained model which attends to components of the semantic representation being built up by the GRU. The new attention mechanism, called gated-attention, is implemented based on multiplicative interactions between the query and the contextual embeddings, and is applied per hop to act as fine-grained information filters during the multi-step reasoning. The filters weigh individual components of the vector representation of each token in the document separately.
The design of gated-attention layers is motivated by the effectiveness of multiplicative interaction among vector-space representations, e.g., in various types of recurrent units (Hochreiter & Schmidhuber, 1997; Wu et al., 2016) and in relational learning (Yang et al., 2014; Kiros et al., 2014). While other types of compositional operators are possible, such as concatenation or addition (Mitchell & Lapata, 2008), we find that multiplication has strong empirical performance (section 4.4). Intuitively, multiplicative interaction e q between two word embeddings e and q adjusts the semantic meaning of e towards q, keeping the compositionality of the original embeddings preserved.2
3.2 MODEL DETAILS
Several components of the model use a Gated Recurrent Unit (GRU) (Cho et al., 2014) which maps an input sequence X = [x1, x2, . . . , xT ] to an ouput sequence H = [h1, h2, . . . , hT ] as follows:
rt = σ(Wrxt + Urht−1 + br),
zt = σ(Wzxt + Uzht−1 + bz),
h̃t = tanh(Whxt + Uh(rt ht−1) + bh), ht = (1− zt) ht−1 + zt h̃t.
where denotes the Hadamard product or the element-wise multiplication. rt and zt are called the reset and update gates respectively, and h̃t the candidate output. A Bi-directional GRU (BiGRU) processes the sequence in both forward and backward directions to produce two sequences [hf1 , h f 2 , . . . , h f T ] and [h b 1, h b 2, . . . , h
b T ], which are concatenated at the output ←→ GRU(X) = [hf1‖hbT , . . . , h f T ‖h b 1] (1)
where ←→ GRU(X) denotes the full output of the Bi-GRU obtained by concatenating each forward
state hfi and backward state h b T−i+1 at time-step i given the input X . Note
←→ GRU(X) is a matrix in
R2nh×T where nh stands for the number of hidden units in GRU.
LetX(0) = [x(0)1 , x (0) 2 , . . . x (0) |D|] denote the token embeddings of the document, which are also inputs at layer 1 for the document reader below, and Y = [y1, y2, . . . y|Q|] denote the token embeddings of the query. Here |D| and |Q| denote the document and query lengths respectively.
3.2.1 MULTI-HOP ARCHITECTURE
Figure 1 illustrates the Gated-Attention (GA) reader. The model reads the document and the query overK horizontal layers, where layer k receives the contextual embeddingsX(k−1) of the document from the previous layer. The document embeddings are transformed by taking the full output of a document Bi-GRU (indicated in blue in Figure 1):
D(k) = ←→ GRU (k)
D (X (k−1)) (2)
2e1 q + e2 q = (e1 + e2) q,∀e1, e2.
At the same time, a layer-specific query representation is computed as the full output of a separate query Bi-GRU (indicated in green in Figure 1):
Q(k) = ←→ GRU (k)
Q (Y ) (3)
Next, Gated-Attention is applied to D(k) and Q(k) to compute inputs for the next layer X(k).
X(k) = GA(D(k), Q(k)) (4)
where GA is defined in the following subsection.
3.2.2 GATED-ATTENTION MODULE
For brevity, let us drop the superscript k in this subsection as we are focusing on a particular layer. For each token di in D, the GA module forms a token-specific representation of the query q̃i using soft attention, and then multiplies the query representation element-wise with the document token representation. Specifically, for i = 1, . . . , |D|:
αi = softmax(Q>di) (5) q̃i = Qαi
xi = di q̃i (6) In equation (6) we use the multiplication operator to model the interactions between di and q̃i. In the experiments section, we also report results for other choices of gating functions, including addition xi = di + q̃i and concatenation xi = di‖q̃i.
3.2.3 ANSWER PREDICTION
Let q(K)` = q f ` ‖qbT−`+1 be an intermediate output of the final layer query Bi-GRU at the location ` of the cloze token in the query, and D(K) = ←→ GRU (K)
D (X (K−1)) be the full output of final layer
document Bi-GRU. To obtain the probability that a particular token in the document answers the query, we take an inner-product between these two, and pass through a softmax layer:
s = softmax((q(K)` ) TD(K)) (7)
where vector s defines a probability distribution over the |D| tokens in the document. The probability of a particular candidate c ∈ C as being the answer is then computed by aggregating the probabilities of all document tokens which appear in c and renormalizing over the candidates:
Pr(c|d, q) ∝ ∑
i∈I(c,d)
si (8)
where I(c, d) is the set of positions where a token in c appears in the document d. This aggregation operation is the same as the pointer sum attention applied in the AS Reader (Kadlec et al., 2016).
Finally, the candidate with maximum probability is selected as the predicted answer:
a∗ = argmaxc∈C Pr(c|d, q). (9)
During the training phase, model parameters of the GA reader are updated w.r.t. a cross-entropy loss between the predicted probabilities and the true answers.
3.2.4 FURTHER ENHANCEMENTS
Character-level Embeddings: Given a token w from the document or query, its vector space representation is computed as x = L(w)||C(w). L(w) retrieves the word-embedding for w from a lookup table L ∈ R|V |×nl , whose rows hold a vector for each unique token in the vocabulary. We also utilize a character composition model C(w) which generates an orthographic embedding of the token. Such embeddings have been previously shown to be helpful for tasks like Named Entity Recognition (Yang et al., 2016) and dealing with OOV tokens at test time (Dhingra et al., 2016). The embedding C(w) is generated by taking the final outputs zfnc and z b nc of a Bi-GRU applied to embeddings from a lookup table of characters in the token, and applying a linear transformation:
z = zfnc ||z b nc
C(w) =Wz + b
Question Evidence Common Word Feature (qe-comm): (Li et al., 2016) recently proposed a simple token level indicator feature which significantly boosts reading comprehension performance in some cases. For each token in the document we construct a one-hot vector fi ∈ {0, 1}2 indicating whether that token is present in the query or not. It can be incorporated into the GA reader by assigning a feature lookup table F ∈ RnF×2 (we use nF = 2), taking the feature embedding ei = fTi F and appending it to the inputs of the last layer document BiGRU as, x(K)i ‖fi for all i. We conducted several experiments both with and without this feature and observed some interesting trends, which are discussed below. Henceforth, we refer to this feature as the qe-comm feature or just feature.
4 EXPERIMENTS AND RESULTS
4.1 DATASETS
We evaluate the GA reader on five large-scale datasets recently proposed in the literature. The first two, CNN and Daily Mail news stories3 consist of articles from the popular CNN and Daily Mail websites (Hermann et al., 2015). A query over each article is formed by removing an entity from the short summary which follows the article. Further, entities within each article were anonymized to make the task purely a comprehension one. N-gram statistics, for instance, computed over the entire corpus are no longer useful in such an anonymized corpus.
The next two datasets are formed from two different subsets of the Children’s Book Test (CBT)4 (Hill et al., 2015). Documents consist of 20 contiguous sentences from the body of a popular children’s book, and queries are formed by deleting a token from the 21st sentence. We only focus on
3 https://github.com/deepmind/rc-data 4 http://www.thespermwhale.com/jaseweston/babi/CBTest.tgz
subsets where the deleted token is either a common noun (CN) or named entity (NE) since simple language models already give human-level performance on the other types (cf. (Hill et al., 2015)).
The final dataset we evaluate on is Who Did What5 (WDW) (Onishi et al., 2016), constructed from the LDC English Gigaword newswire corpus. First, article pairs which appeared around the same time and with overlapping entities are chosen, and then one article forms the document and a cloze query is constructed from the other. Missing tokens are always person named entities. Questions which are easily answered by simple baselines are filtered out, to make the task more challenging. There are two versions of the training set—a small but focused “Strict” version and a large but noisy “Relaxed” version. We report results on both settings which share the same validation and test sets. Statistics of all the datasets used in our experiments are summarized in Table 1.
4.2 IMPLEMENTATION DETAILS
Our model was implemented using the Theano (Theano Development Team, 2016) and Lasagne6 Python libraries. We used stochastic gradient descent with ADAM updates for optimization, which combines classical momentum and adaptive gradients (Kingma & Ba, 2014). The batch size was 32 and the initial learning rate was 5×10−4 which was halved every epoch after the second epoch. The same setting is applied to all models and datasets. We also used gradient clipping with a threshold of 10 to stabilize GRU training (Pascanu et al., 2012). We set the number of layers K to be 3 for all experiments, and provide further analysis below. The number of hidden units for the character GRU was set to 50. The remaining two hyperparameters—size of document and query GRUs, and dropout rate—were tuned on the validation set, and their optimal values are shown in Table 2. In general, the optimal GRU size increases and the dropout rate decreases as the corpus size increases.
The word lookup table was initialized with 100d GloVe vectors7 (Pennington et al., 2014) and OOV tokens at test time were assigned unique random vectors. We empirically observed that initializing with pre-trained embeddings gives higher performance compared to random initialization for all datasets. Furthermore, for smaller datasets (WDW and CBT) we found that fixing these embeddings to their pretrained values led to higher test performance, possibly since it avoids overfitting. We do not use the character composition model for CNN and Daily Mail, since entities (and hence candidate answers) are anonymized to generic tokens in these datasets. For other datasets the character lookup table was randomly initialized with 25d vectors. All other parameters were initialized to their default values as specified in the Lasagne library.
4.3 PERFORMANCE COMPARISON
Tables 3 and 5 show a comparison of the performance of GA Reader with previously published results on WDW and CNN, Daily Mail, CBT datasets respectively. The numbers reported for GA Reader are for single best models, though we compare to both ensembles and single models from prior work. GA Reader-- refers to an earlier version of the model, unpublished but described in a preprint, with the following differences—(1) it does not utilize token-specific attentions within the GA module, as described in equation (5), (2) it does not use a character composition model, (3) it is initialized with word embeddings pretrained on the corpus itself rather than GloVe. A detailed analysis of these differences is studied in the next section. Here we present 4 variants of the latest GA Reader, using combinations of whether the qe-comm feature is used (+feature) or not, and whether the word lookup table L(w) is updated during training or fixed to its initial value.
Interestingly, we observe that feature engineering leads to significant improvements for WDW and CBT datasets, but not for CNN and Daily Mail datasets. We note that anonymization of the latter
5 https://tticnlp.github.io/who_did_what/ 6 https://lasagne.readthedocs.io/en/latest/ 7 http://nlp.stanford.edu/projects/glove/
Table 3: Validation/Test accuracy (%) on WDW dataset for both “Strict” and “Relaxed” settings. Results marked with † are cf previously published works.
Model Strict Relaxed Val Test Val Test
Human † – 84 – – Attentive Reader † – 53 – 55 AS Reader † – 57 – 59 Stanford AR † – 64 – 65 NSE † 66.5 66.2 67.0 66.7 GA-- † – 57 – 60.0 GA (update L(w)) 67.8 67.0 67.0 66.6 GA (fix L(w)) 68.3 68.0 69.6 69.1 GA (+feature, update L(w)) 70.1 69.5 70.9 71.0 GA (+feature, fix L(w)) 71.6 71.2 72.6 72.6
Table 4: Top: Performance of different gating functions. Bottom: Effect of varying the number of hops K. Results on WDW dataset without using the qe-comm feature and with fixed L(w).
Gating Function Accuracy Val Test
Sum 64.9 64.5 Concatenate 64.4 63.7 Multiply 68.3 68.0 K
1 (AS) † – 57 2 65.6 65.6 3 68.3 68.0 4 68.3 68.2
datasets means that there is already some feature engineering (it adds hints about whether a token is an entity), and these are much larger than the other four. In machine learning it is common to see the effect of feature engineering diminish with increasing data size. Similarly, fixing the word
embeddings provides an improvement for the WDW and CBT, but not for CNN and Daily Mail. This is not surprising given that the latter datasets are larger and less prone to overfitting.
Comparing with prior work, on the WDW dataset the basic version of the GA Reader outperforms all previously published models when trained on the Strict setting. By adding the qe-comm feature the performance increases by 3.2% and 3.5% on the Strict and Relaxed settings respectively to set a new state of the art on this dataset. On the CNN and Daily Mail datasets the GA Reader leads to an improvement of 3.2% and 4.3% respectively over the best previous single models. They also outperform previous ensemble models, setting a new state of that art for both datasets. For CBT-NE, GA Reader with the qe-comm feature outperforms all previous single and ensemble models except the AS Reader trained on the much larger BookTest Corpus (Bajgar et al., 2016). Lastly, on CBTCN the GA Reader with the qe-comm feature outperforms all previously published single models except the NSE, and AS Reader trained on a larger corpus.
4.4 GA READER ANALYSIS
In this section we do an ablation study to see the effect of Gated Attention. We compare the GA Reader as described here to a model which is exactly the same in all aspects, except that it passes document embeddings D(k) in each layer directly to the inputs of the next layer without using the GA module. In other words X(k) = D(k) for all k > 0. This model ends up using only one query GRU at the output layer for selecting the answer from the document. We compare these two variants both with and without the qe-comm feature on CNN and WDW datasets for three subsets of the training data - 50%, 75% and 100%. Test set accuracies for these settings are shown in Figure 2. On CNN when tested without feature engineering, we observe that GA provides a significant boost in performance compared to without GA. When tested with the feature it still gives an improvement, but the improvement is significant only with 100% training data. On WDW-Strict, which is a third of the size of CNN, without the feature we see an improvement when using GA versus without using GA, which becomes significant as the training set size increases. When tested with the feature on WDW, for a small data size without GA does better than with GA, but as the dataset size increases they become equivalent. We conclude that Gated Attention provides a boost in the absence of feature engineering, or as the training set size increases.
Next we look at the question of how to gate intermediate document reader states from the query, i.e. what operation to use in equation 6. Table 4 (top) shows the performance on WDW dataset for three common choices – sum (x = d+ q), concatenate (x = d‖q) and multiply (x = d q). Empirically we find that element-wise multiplication does significantly better than the other two, which justifies our motivation to “filter” out document features which are irrelevant to the query.
At the bottom of Table 4 we show the effect of varying the number of hops K of the GA Reader on the final performance. We note that for K = 1, our model is equivalent to the AS Reader without any GA modules. We see a steep and steady rise in accuracy as the number of hops is increased from K = 1 to K = 3, which remains constant beyond that. This is a fairly common trend in machine learning as model complexity is increased, however we note that a multi-hop architecture is important to achieve a high performance for this task, and provide further evidence for this in the next section.
Lastly, we perform an ablation study for the three components of the GA Reader which were absent in the preprint version (GA Reader--). Table 6 shows accuracy on WDW by removing one component at a time. The steepest reduction is observed when we replace pretrained GloVe vectors with those pretrained on the corpus itself. GloVe vectors were trained on a large corpus of about 6 billion tokens (Pennington et al., 2014), and provide an important source of prior knowledge for the model. We note here that the strongest baseline on WDW, NSE (Munkhdalai & Yu, 2016b), also uses pretrained GloVe vectors, hence the comparison is fair in that respect. Next, we observe a substantial drop when removing token-specific attentions over the query in the GA module, which allow gating indi-
vidual tokens in the document only by parts of the query relevant to that token rather than the overall query representation. Finally, removing the character embeddings, which were only used for WDW and CBT datasets, leads to a reduction of about 1% in the performance.
4.5 ATTENTION VISUALIZATION
To gain an insight into the reading process employed by the model we analyzed the attention distributions at intermediate layers of the reader. Figure 3 shows an example from the validation set of WDW dataset (several more are in the Appendix). In each figure, the left and middle plots visualize attention over the query (equation 5) for candidates in the document after layers 1 & 2 respectively. The right plot shows attention over candidates in the document of cloze placeholder (XXX) in the query at the final layer. The full document, query and correct answer are shown at the bottom.
A generic pattern observed in these examples is that in intermediate layers, candidates in the document (shown along rows) tend to pick out salient tokens in the query which provide clues about the cloze, and in the final layer the candidate with the highest match with these tokens is selected as the answer. In Figure 3 there is a high attention of the correct answer on financial regulatory standards in the first layer, and on us president in the second layer. The incorrect answer, in contrast, only attends to one of these aspects, and hence receives a lower score in the final layer despite the n-gram overlap it has with the cloze token in the query. Importantly, different layers tend to focus on different tokens in the query, which supports the hypothesis that the multi-hop architecture of GA Reader is able to combine distinct pieces of information to answer the query.
5 CONCLUSION
We presented the Gated-Attention reader for answering cloze-style questions over documents. The GA reader features a novel multiplicative gating mechanism, combined with a multi-hop architecture. Our model achieves state-of-the-art performance on several large-scale benchmark datasets with more than 4% improvements over competitive baselines. Our model design is backed up by an ablation study showing statistically significant improvements of using Gated Attention as information filters. We also showed empirically that multiplicative gating is superior to addition and concatenation operations for implementing gated-attentions, though a theoretical justification remains part of future research goals. Analysis of document and query attentions in intermediate layers of the reader further reveals that the model iteratively attends to different aspects of the query to arrive at the final answer. In this paper we have focused on text comprehension, but we believe that the Gated-Attention mechanism may benefit other tasks as well where multiple sources of information interact. Concurrent to our work (Chu et al., 2016) have also shown the effectiveness of GA Readers on the LAMBADA dataset (Paperno et al., 2016) for language modeling.
A ATTENTION PLOTS

Summary:
The authors propose a multi-hop "gated attention" model, which models the interactions between query and document representations, for answering cloze-style questions. The document representation is attended to sequentially over multiple-hops using similarity with the query representation (using a dot-product) as the scoring/attention function.
The proposed method improves upon (CNN, Daily Mail, Who-Did-What datasets) or is comparable to (CBT dataset) the state-of-the-art results.
Pros:
1. Nice idea on heirarchical attention for modulating the context (document) representation by the task-specific (query) representation.
2. The presentation is clear with thorough experimental comparison with the latest results.
Comments:
1. The overall system presents a number of architectural elements: (1) attention at multiple layers (multi-hop), (2) query based attention for the context (or gated attention), (3) encoding the query vector at each layer independently.
It is important to breakdown the gain in performance due to the above factors: the ablation study presented in section 4.4 helps establish the importance of Gated Attention (#2 above). However, it is not clear:
  (1) how much multiple-hops of gated-attention contribute to the performance.
  (2) how important is it to have a specialized query encoder for each layer.
Understanding the above better, will help simplify the architecture.
2. The tokens are represented using L(w) and C(w). It is not clear if C(w) is crucial for the performance of the proposed method.
There is a significant performance drop when C(w) is absent (e.g. in "GA Reader--"; although there are other changes in "GA Reader--" which could affect the performance). Hence, it is not clear how much does the main idea, i.e., gated attention contributes towards the superior performance of the proposed method.


We use reinforcement learning to learn tree-structured neural networks for computing representations of natural language sentences. In contrast with prior work on tree-structured models, in which the trees are either provided as input or predicted using supervision from explicit treebank annotations, the tree structures in this work are optimized to improve performance on a downstream task. Experiments demonstrate the benefit of learning task-specific composition orders, outperforming both sequential encoders and recursive encoders based on treebank annotations. We analyze the induced trees and show that while they discover some linguistically intuitive structures (e.g., noun phrases, simple verb phrases), they are different than conventional English syntactic structures.
1 INTRODUCTION
Languages encode meaning in terms of hierarchical, nested structures on sequences of words (Chomsky, 1957). However, the degree to which neural network architectures that compute representations of the meaning of sentences for practical applications should explicitly reflect such structures is a matter for debate. In this work, we use reinforcement learning to learn to construct trees for computing sentence representations, guided by feedback from downstream tasks that depend on these representations. The space of structures that are considered by the learner includes both fully sequential structures (corresponding to traditional recurrent neural network “encoders”), as well as all projective binary trees. Thus, although we take seriously the notion that good compositional architectures might be tree-structured, we specify neither the form of the tree nor whether a tree is necessary at all, and instead leave those decisions up to the learner (and the data).
To place this work in context, there are three predominant approaches for constructing vector representations of sentences from a sequence of words. The first composes words sequentially using a recurrent neural network, treating the RNN’s final hidden state as the representation of the sentence (Cho et al., 2014; Sutskever et al., 2014; Kiros et al., 2015). In such models, there is no explicit hierarchical organization imposed on the words, and the RNN’s dynamics must learn to simulate it. The second approach uses tree-structured networks to recursively compose representations of words and phrases to form representations of larger phrases and, finally, the complete sentence. In contrast to sequential models, these models’ architectures are organized according to each sentence’s syntactic structure, that is, the hierarchical organization of words into nested phrases that characterizes human intuitions about how words combine to form grammatical sentences. Prior work on tree-structured models has assumed that trees are either provided together with the input sentences (Clark et al., 2008; Grefenstette & Sadrzadeh, 2011; Socher et al., 2012; 2013; Tai et al., 2015) or that they are predicted based on explicit treebank annotations jointly with the downstream task (Bowman et al., 2016; Dyer et al., 2016). The last approach for constructing sentence representations uses convolutional neural networks to produce the representation in a bottom up manner, either with syntactic information (Ma et al., 2015) or without (Kim, 2014; Kalchbrenner et al., 2014).
Our work can be understood as a compromise between the first two approaches. Rather than using explicit supervision of tree structure, we use reinforcement learning to learn tree structures (and thus, sentence-specific compositional architectures), taking performance on a downstream task that uses the computed sentence representation as the reward signal. In contrast to sequential RNNs, which ignore tree structure, our model still generates a latent tree for each sentence and uses it to
structure the composition. Our hypothesis is that encouraging the model to learn tree-structured compositions will bias the model toward better generalizations about how words compose to form sentence meanings, leading to better performance on downstream tasks.
This work is related to unsupervised grammar induction (Klein & Manning, 2004; Blunsom & Cohn, 2010; Spitkovsky et al., 2011, inter alia), which seeks to infer a generative grammar of an infinite language from a finite sample of strings from the language—but without any semantic feedback. Previous work on unsupervised grammar induction that incorporates semantic supervision involves designing complex models for Combinatory Categorial Grammars (Zettlemoyer & Collins, 2005) or marginalizing over latent syntactic structures (Naradowsky et al., 2012). Since semantic feedback has been proposed as crucial for the acquisition of syntax (Pinker, 1984), our model offers a simpler alternative.1 However, our primary focus is on improving performance on the downstream model, so the learner may settle on a different solution than conventional English syntax. We thus also explore what kind of syntactic structures are derivable from shallow semantics.
Experiments on various tasks (i.e., sentiment analysis, semantic relatedness, natural language inference, and sentence generation) show that reinforcement learning is a promising direction to discover hierarchical structures of sentences. Notably, representations learned this way outperformed both conventional left-to-right models and tree-structured models based on linguistic syntax in downstream applications. This is in line with prior work showing the value of learning tree structures in statistical machine translation models (Chiang, 2007). Although the induced tree structures manifested a number of linguistically intuitive structures (e.g., noun phrases, simple verb phrases), there are a number of marked differences to conventional analyses of English sentences (e.g., an overall left-branching structure).
2 MODEL
Our model consists of two components: a sentence representation model and a reinforcement learning algorithm to learn the tree structure that is used by the sentence representation model.
2.1 TREE LSTM
Our sentence representation model follows the Stack-augmented Parser-Interpreter Neural Network (SPINN; Bowman et al., 2016), SPINN is a shift-reduce parser that uses Long Short-Term Memory (LSTM; Hochreiter and Schmidhuber, 1997) as its composition function. Given an input sentence of N words x = {x1, x2, . . . , xN}, we represent each word by its embedding vector xi ∈ RD. The parser maintains an index pointer p starting from the leftmost word (p = 1) and a stack. To parse the sentence, it performs a sequence of operations a = {a1, a2, . . . , a2N−1}, where at ∈ {SHIFT, REDUCE}. A SHIFT operation pushes xp to the stack and moves the pointer to the next word (p++); while a REDUCE operation pops two elements from the stack, composes them to a single element, and pushes it back to the stack. SPINN uses Tree LSTM (Tai et al., 2015; Zhu et al., 2015) as the REDUCE composition function, which we follow. In Tree LSTM, each element of the stack is represented by two vectors, a hidden state representation h and a memory representation c. Two elements of the stack (hi, ci) and (hj , cj) are composed as:
i = σ(WI [hi,hj ] + bI) o = σ(WO[hi,hj ] + bI)
fL = σ(WFL [hi,hj ] + bFL) fR = σ(WFR [hi,hj ] + bFR)
g = tanh(WG[hi,hj ] + bG) c = fL ci + fR cj + i g h = o c
(1)
where [hi,hj ] denotes concatenation of hi and hj , and σ is the sigmoid activation function.
A unique sequence of {SHIFT, REDUCE} operations corresponds to a unique binary parse tree of the sentence. A SHIFT operation introduces a new leaf node in the parse tree, while a REDUCE operation combines two nodes by merging them into a constituent. See Figure 1 for an example. We note that for a sentence of lengthN , there are exactlyN SHIFT operations andN−1 REDUCE operations that are needed to produce a binary parse tree of the sentence. The final sentence representation produced
1Our model only produces an interpretation grammar that parses language instead of a generative grammar.
by the Tree LSTM is the hidden state of the final element of the stack hN−1 (i.e., the topmost node of the tree).
Tracking LSTM. SPINN optionally augments Tree LSTM with another LSTM that incorporates contextual information in sequential order called tracking LSTM, which has been shown to improve performance for textual entailment. It is a standard recurrent LSTM network that takes as input the hidden states of the top two elements of the stack and the embedding vector of the word indexed by the pointer at timestep t. Every time a REDUCE operation is performed, the output of the tracking LSTM e is included as an additional input in Eq. 1 (i.e., the input to the REDUCE composition function is [hi,hj , e] instead of [hi,hj ]).
2.2 REINFORCEMENT LEARNING
In previous work (Tai et al., 2015; Bowman et al., 2016), the tree structures that guided composition orders of Tree LSTM models are given directly as input (i.e., a is observed and provided as an input). Formally, each training data is a triplet {x,a,y}. Tai et al. (2015) consider models where a is also given at test time, whereas Bowman et al. (2016) explore models where a can be either observed or not at test time. When it is only observed during training, a policy is trained to predict a at test time. Note that in this case the policy is trained to match explicit human annotations (i.e., Penn TreeBank annotations), so the model learns to optimize representations according to structures that follows human intuitions. They found that models that observe a at both training and test time are better than models that only observe a during training.
Our main idea is to use reinforcement learning (policy gradient methods) to discover the best tree structures for the task that we are interested in. We do not place any kind of restrictions when learning these structures other than that they have to be valid binary parse trees, so it may result in tree structures that match human linguistic intuition, heavily right or left branching, or other solutions if they improve performance on the downstream task.
We parameterize each action a ∈ {SHIFT, REDUCE} by a policy network π(a | s;WR), where s is a representation of the current state and WR is the parameter of the network. Specifically, we use a two-layer feedforward network that takes the hidden states of the top two elements of the stack hi and hj and the embedding vector of the word indexed by the pointer xp as its input:
s = ReLU(W1R[hi,hj,xp] + b 1 R) such that π(a | s;WR) ∝ exp(w2>R s+ b2R)
where [hi,hj ,xp] denotes concatenation of vectors inside the brackets.
If a is given as part of the training data, the policy network can be trained—in a supervised training regime—to predict actions that result in trees that match human intuitions. Our training data, on the other hand, is a tuple {x,y}. We use REINFORCE (Williams, 1992), which is an instance of a broader class of algorithms called policy gradient methods, to learn WR such that the sequence of actions a = {a1, . . . , aT } maximizes:
R(W) = Eπ(a,s;WR) [ T∑ t=1 rtat ] ,
where rt is the reward at timestep t. We use performance on a downstream task as the reward function. For example, if we are interested in using the learned sentence representations in a classification task, our reward function is the probability of predicting the correct label using a sentence representation composed in the order given by the sequence of actions sampled from the policy network, so R(W) = log p(y | T-LSTM(x);W), where we use W to denote all model parameters (Tree LSTM, policy network, and classifier parameters), y is the correct label for input sentence x, and x is represented by the Tree LSTM structure in §2.1. For a natural language generation task where the goal is to predict the next sentence given the current sentence, we can use the probability of predicting words in the next sentence as the reward function, so R(W) = log p(xs+1 | T-LSTM(xs);W). Note that in our setup, we do not immediately receive a reward after performing an action at timestep t. The reward is only observed at the end after we finish creating a representation for the current sentence with Tree LSTM and use the resulting representation for the downstream task. At each timestep t, we sample a valid action according to π(a | s;WR). We add two simple constraints to make the sequence of actions result in a valid tree: REDUCE is forbidden if there are fewer than two elements on the stack, and SHIFT is forbidden if there are no more words to read from the sentence. After reaching timestep 2N − 1, we construct the final representation and receive a reward that is used to update our model parameters.
We experiment with two learning methods: unsupervised structures and semi-supervised structures. Suppose that we are interested in a classification task. In the unsupervised case, the objective function that we maximize is log p(y | T-LSTM(x);W). In the semi-supervised case, the objective function for the first E epochs also includes a reward term for predicting the correct SHIFT or REDUCE actions obtained from an external parser—in addition to performance on the downstream task, so we maximize log p(y | T-LSTM(x);W) + log π(a | s;WR). The motivation behind this model is to first guide the model to discover tree structures that match human intuitions, before letting it explore other structures close to these ones. After epochE, we remove the second term from our objective function and continue maximizing the first term. Note that unsupervised and semi-supervised here refer to the tree structures, not the nature of the downstream task.
3 EXPERIMENTS
3.1 BASELINES
The goal of our experiments is to evaluate our hypothesis that we can discover useful task-specific tree structures (composition orders) with reinforcement learning. We compare the following composition methods (the last two are unique to our work):
• Right to left: words are composed from right to left.2
• Left to right: words are composed from left to right. This is the standard recurrent neural network composition order.
• Bidirectional: A bidirectional right to left and left to right models, where the final sentence embedding is an average of sentence embeddings produced by each of these models.
• Balanced binary tree: words are composed according to a balanced binary parse tree of the sentence.
• Supervised syntax: words are composed according to a predefined parse tree of the sentence. When parse tree information is not included in the dataset, we use Stanford parser (Klein & Manning, 2003) to parse the corpus.
• Semi-supervised syntax: a variant of our reinforcement learning method, where for the first E epochs we include rewards for predicting predefined parse trees given in the supervised model, before letting the model explore other kind of tree structures at later epochs (i.e., semi-supervised structures in §2.2).
• Latent syntax: another variant of our reinforcement learning method where there is no predefined structures given to the model at all (i.e., unsupervised structures in §2.2).
2We choose to include right to left as a baseline since a right-branching tree structure—which is the output of a right to left composition order—has been shown to be a reliable baseline for unsupervised grammar induction (Klein & Manning, 2004).
For learning, we use stochastic gradient descent with minibatches of size 1 and `2 regularization constant tune on development data from {10−4, 10−5, 10−6, 0}. We use performance on development data to choose the best model and decide when to stop training.
3.2 TASKS
We evaluate our method on four sentence representation tasks: sentiment classification, semantic relatedness, natural language inference (entailment), and sentence generation. We show statistics of the datasets in Table 1 and describe each task in detail in this subsection.
Stanford Sentiment Treebank. We evaluate our model on a sentiment classification task from the Stanford Sentiment Treebank (Socher et al., 2013). We use the binary classification task where the goal is to predict whether a sentence is a positive or a negative movie review.
We set the word embedding size to 100 and initialize them with Glove vectors (Pennington et al., 2014)3. For each sentence, we create a 100-dimensional sentence representation s ∈ R100 with Tree LSTM, project it to a 200-dimensional vector and apply ReLU: q = ReLU(Wps + bp), and compute p(ŷ = c | q;wq) ∝ exp(wq,cq+ bq). We run each model 3 times (corresponding to 3 different initialization points) and use the development data to pick the best model. We show the results in Table 2. Our results agree with prior work that have shown the benefits of using syntactic parse tree information on this dataset (i.e., supervised recursive model is generally better than sequential models). The best model is the latent syntax model, which is also competitive with results from other work on this dataset. Both the latent and semi-supervised syntax models outperform models with predefined structures, demonstrating the benefit of learning task-specific composition orders.
Semantic relatedness. The second task is to predict the degree of relatedness of two sentences from the Sentences Involving Compositional Knowledge corpus (SICK; Marelli et al., 2014) . In this dataset, each pair of sentences are given a relatedness score on a 5-point rating scale. For each sentence, we use Tree LSTM to create its representations. We denote the final representations by {s1, s2} ∈ R100. We construct our prediction by computing: u = (s2 − s1)2, v = s1 s2, q = ReLU(Wp[u,v] + bp), and ŷ = w>q q + bq , where Wp ∈ R200×200,bp ∈ R200,wq ∈ R200, bq ∈ R1 are model parameters, and [u,v] denotes concatenation of vectors inside the brackets. We learn the model to minimize mean squared error.
We run each model 5 times and use the development data to pick the best model. Our results are shown in Table 3. Similarly to the previous task, they clearly demonstrate that learning the tree structures yields better performance.
We also provide results from other work on this dataset for comparisons. Some of these models (Lai & Hockenmaier, 2014; Jimenez et al., 2014; Bjerva et al., 2014) rely on feature engineering and are designed specifically for this task. Our Tree LSTM implementation performs competitively with most models in terms of mean squared error. Our best model—semi-supervised syntax—is better than most models except LSTM models of Tai et al. (2015) which were trained with a different objective function.4 Nonetheless, we observe the same trends with their results that show the benefit of using syntactic information on this dataset.
Stanford Natural Language Inference. We next evaluate our model for natural language inference (i.e., recognizing textual entailment) using the Stanford Natural Language Inference corpus (SNLI; Bowman et al., 2015) . Natural language inference aims to predict whether two sentences are entailment, contradiction, or neutral, which can be formulated as a three-way classification problem. Given a pair of sentences, similar to the previous task, we use Tree LSTM to create sentence representations {s1, s2} ∈ R100 for each of the sentences. Following Bowman et al. (2016), we construct our prediction by computing: u = (s2−s1)2, v = s1 s2, q = ReLU(Wp[u,v, s1, s2]+bp), and p(ŷ = c | q;wq) ∝ exp(wq,cq+ bq), where Wp ∈ R200×400,bp ∈ R200,wq ∈ R200, bq ∈ R1 are model parameters. The objective function that we maximize is the log likelihood of the correct label under the models.
We show the results in Table 4. The latent syntax method performs the best. Interestingly, the sequential left to right model is better than the supervised recursive model in our experiments, which contradicts results from Bowman et al. (2016) that show 300D-LSTM is worse than 300D-SPINN. A possible explanation is that our left to right model has identical number of parameters with the supervised model due to the inclusion of the tracking LSTM even in the left to right model (the only difference is in the composition order), whereas the models in Bowman et al. (2016) have
4Our experiments with the regularized KL-divergence objective function (Tai et al., 2015) do not result in significant improvements, so we choose to report results with the simpler mean squared error objective function.
different number of parameters. Due to the poor performance of the supervised model relative to the unsupervised model, semi-supervised training can only mitigate the loss in accuracy, rather than improve over unsupervised learning. Our models underperform state-of-the-art models on this dataset that have almost four times the number of parameters. We only experiment with smaller models since tree-based models with dynamic structures (e.g., our semi-supervised and latent syntax models) take longer to train. See §4 for details and discussions about training time.
Sentence generation. The last task that we consider is natural language generation. Given a sentence, the goal is to maximize the probability of generating words in the following sentence. This is a similar setup to the Skip Thought objective (Kiros et al., 2015), except that we do not generate the previous sentence as well. Given a sentence, we encode it with Tree LSTM to obtain s ∈ R100. We use a bag-of-words model as our decoder, so p(wi | s;V) ∝ exp(v>i s), where V ∈ R100×29,209 and vi ∈ R100 is the i-th column of V. Using a bag-of-words decoder as opposed to a recurrent neural network decoder increases the importance of producing a better representation of the current sentence, since the model cannot rely on a sophisticated decoder with a language model component to predict better. This also greatly speeds up our training time.
We use IMDB movie review corpus (Diao et al., 2014) for this experiment, The corpus consists of 280,593, 33,793, and 34,029 reviews in training, development, and test sets respectively. We construct our data using the development and test sets of this corpus. For training, we process 33,793 reviews from the original development set to get 441,617 pairs of sentences. For testing, we use 34,029 reviews in the test set (446,471 pairs of sentences). Half of these pairs is used as our development set to tune hyperparamaters, and the remaining half is used as our final test set. Our results in Table 5 further demonstrate that methods that learn tree structures perform better than methods that have fixed structures.
4 DISCUSSION
Learned Structures. Our results in §3 show that our proposed method outperforms competing methods with predefined composition order on all tasks. The right to left model tends to perform worse than the left to right model. This suggests that the left to right composition order, similar to how human reads in practice, is better for neural network models. Our latent syntax method is able to discover tree structures that work reasonably well on all tasks, regardless of whether the task is better suited for a left to right or supervised syntax composition order.
We inspect what kind of structures the latent syntax model learned and how closely they match human intuitions. We first compute unlabeled bracketing F1 scores5 for the learned structures and parses given by Stanford parser on SNLI and Stanford Sentiment Treebank. In the SNLI dataset, there are 10,000 pairs of test sentences (20,000 sentences in total), while the Stanford Sentiment Treebank test set contains 1,821 test sentences. The F1 scores for the two datasets are 41.73 and 40.51 respectively. For comparisons, F1 scores of a right (left) branching tree are 19.94 (41.37) for SNLI and 12.96 (38.56) for SST.
We also manually inspect the learned structures. We observe that in SNLI, the trees exhibit overall left-branching structure, which explains why the F1 scores are closer to a left branching tree structure. Note that in our experiments on this corpus, the supervised syntax model does not perform as well as the left-to-right model, which suggests why the latent syntax model tends to converge towards the left-to-right model. We handpicked two examples of trees learned by our model and show them in Figure 2. We can see that in some cases the model is able to discover concepts such as noun phrases (e.g., a boy, his sleds) and simple verb phrases (e.g., wearing sunglasses, is frowning). Of course, the model sometimes settles on structures that make little sense to humans. We show two such examples in Figure 3, where the model chooses to compose playing frisbee in and outside a as phrases.
Training Time. A major limitation of our proposed model is that it takes much longer to train compared to models with predefined structures. We observe that our models only outperforms models with fixed structures after several training epochs; and on some datasets such as SNLI or IMDB, an epoch could take a 5-7 hours (we use batch size 1 since the computation graph needs to be reconstructed for every example at every iteration depending on the samples from the policy network). This is also the main reason that we could only use smaller 100-dimensional Tree LSTM models in
5We use evalb toolkit from http://nlp.cs.nyu.edu/evalb/.
all our experiments. While for smaller datasets such as SICK the overall training time is approximately 6 hours, for SNLI or IMDB it takes 3-4 days for the model to reach convergence. In general, the latent syntax model and semi-supervised syntax models take about two or three times longer to converge compared to models with predefined structures.
5 CONCLUSION
We presented a reinforcement learning method to learn hierarchical structures of natural language sentences. We demonstrated the benefit of learning task-specific composition order on four tasks: sentiment analysis, semantic relatedness, natural language inference, and sentence generation. We qualitatively and quantitatively analyzed the induced trees and showed that they both incorporate some linguistically intuitive structures (e.g., noun phrases, simple verb phrases) and are different than conventional English syntactic structures.

In this paper, the authors propose a new method to learn hierarchical representations of sentences, based on reinforcement learning. They propose to learn a neural shift-reduce parser, such that the induced tree structures lead to good performance on a downstream task. They use reinforcement learning (more specifically, the policy gradient method REINFORCE) to learn their model. The reward of the algorithm is the evaluation metric of the downstream task. The authors compare two settings, (1) no structure information is given (hence, the only supervision comes from the downstream task) and (2) actions from an external parser is used as supervision to train the policy network, in addition to the supervision from the downstream task. The proposed approach is evaluated on four tasks: sentiment analysis, semantic relatedness, textual entailment and sentence generation.
I like the idea of learning tree representations of text which are useful for a downstream task. The paper is clear and well written. However, I am not convinced by the experimental results presented in the paper. Indeed, on most tasks, the proposed model is far from state-of-the-art models:
 - sentiment analysis, 86.5 v.s. 89.7 (accuracy);
 - semantic relatedness, 0.32 v.s. 0.25 (MSE);
 - textual entailment, 80.5 v.s. 84.6 (accuracy).
From the results presented in the paper, it is hard to know if these results are due to the model, or because of the reinforcement learning algorithm.
PROS:
 - interesting idea: learning structures of sentences adapted for a downstream task.
 - well written paper.
CONS:
 - weak experimental results (do not really support the claim of the authors).
Minor comments:
In the second paragraph of the introduction, one might argue that bag-of-words is also a predominant approach to represent sentences.
Paragraph titles (e.g. in section 3.2) should have a period at the end.
----------------------------------------------------------------------------------------------------------------------
UPDATE
I am still not convinced by the results presented in the paper, and in particular by the fact that one must combine the words in a different way than left-to-right to obtain state of the art results.
However, I do agree that this is an interesting research direction, and that the results presented in the paper are promising. I am thus updating my score from 5 to 6.