UNROLLED ITERATIVE ESTIMATION
The past year saw the introduction of new architectures such as Highway networks (Srivastava et al., 2015a) and Residual networks (He et al., 2015) which, for the first time, enabled the training of feedforward networks with dozens to hundreds of layers using simple gradient descent. While depth of representation has been posited as a primary reason for their success, there are indications that these architectures defy a popular view of deep learning as a hierarchical computation of increasingly abstract features at each layer. In this report, we argue that this view is incomplete and does not adequately explain several recent findings. We propose an alternative viewpoint based on unrolled iterative estimation—a group of successive layers iteratively refine their estimates of the same features instead of computing an entirely new representation. We demonstrate that this viewpoint directly leads to the construction of Highway and Residual networks. Finally we provide preliminary experiments to discuss the similarities and differences between the two architectures.
The past year saw the introduction of new architectures such as Highway networks (Srivastava et al., 2015a) and Residual networks (He et al., 2015) which, for the first time, enabled the training of feedforward networks with dozens to hundreds of layers using simple gradient descent. While depth of representation has been posited as a primary reason for their success, there are indications that these architectures defy a popular view of deep learning as a hierarchical computation of increasingly abstract features at each layer.
In this report, we argue that this view is incomplete and does not adequately explain several recent findings. We propose an alternative viewpoint based on unrolled iterative estimation—a group of successive layers iteratively refine their estimates of the same features instead of computing an entirely new representation. We demonstrate that this viewpoint directly leads to the construction of Highway and Residual networks. Finally we provide preliminary experiments to discuss the similarities and differences between the two architectures.
1 INTRODUCTION
Deep learning can be thought of as learning many levels of representation of the input which form a hierarchy of concepts (Deng & Yu, 2014; Goodfellow et al., 2016; LeCun et al., 2015) (but note that this is not the only view: cf. Schmidhuber (2015)). With fixed computational budget, deeper architectures are believed to possess greater representational power and, consequently, higher performance than shallower models. Intuitively, each layer of a deep neural network computes a new level of representation. For convolutional networks, Zeiler & Fergus (2014) visualized the features computed by each layer, and demonstrated that they in fact become increasingly abstract with depth. We refer to this way of thinking about neural networks as the representation view, which probably dates back to Hubel & Wiesel (1962). The representation view links the layers in a network to the abstraction levels of their representations, and as such represents a pervasive assumption in many recent publications including He et al. (2015) who describe the success of their Residual networks
like this: “Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset.”
Surprisingly, increasing the depth of a network beyond a certain point often leads to a decline in performance even on the training set (Srivastava et al., 2015a). Since adding more layers cannot decrease representational power, this phenomenon is usually attributed to the vanishing gradient problem (Hochreiter, 1991). Therefore, even though deeper models are more powerful in principle, they often fall short in practice.
Recently, training feedforward networks with hundreds of layers has become feasible through the invention of Highway networks Srivastava et al. (2015a) and Residual networks (ResNets; He et al. 2015). The latter have been widely successful in computer vision, advancing the state of the art on many benchmarks and winning several pattern recognition competitions (He et al., 2015), while Highway networks have been used to improve language modeling (Kim et al., 2015; Jozefowicz et al., 2016; Zilly et al., 2016) and translation (Lee et al., 2016). Both architectures have been introduced with the explicit goal of training deeper models.
There are, however, some surprising findings that seem to contradict the applicability of the representation view to these very deep networks. For example, it has been reported that removing almost any layer from a trained Highway or Residual network has only minimal effect on its overall performance (Srivastava et al., 2015b; Veit et al., 2016). This idea has been extended to a layerwise dropout as a regularizer for ResNets (Huang et al., 2016b). But if each layer supposedly builds a new level of representation from the previous one, then removing any layer should critically disrupt the input for the following layer. So how is it possible that doing so seems to have only a negligible effect on the network output? Veit et al. (2016) even demonstrated that shuffling some of the layers in a trained ResNet barely affects performance.
It has been argued that ResNets are better understood as ensembles of shallow networks (Huang et al., 2016b; Veit et al., 2016; Abdi & Nahavandi, 2016). According to this interpretation, ResNets implicitly average exponentially many subnetworks, each of which only use a subset of the layers. But the question remains open as to how a layer in such a subnetwork can successfully operate with changing input representations. This, along with other findings, begs the question as to whether the representation view is appropriate for understanding these new architectures.
In this paper, we propose a new interpretation that reconciles the representation view with the operation of Highway and Residual networks: functional blocks1 in these networks do not compute entirely new representations; instead, they engage in an unrolled iterative estimation of representations that refine/improve upon their input representation, thus preserving feature identity. The transition to a new level of representation occurs when a dimensionality change—through projection—separates two groups of blocks which we refer to as a stage (Figure 1). Taking this perspective, we are able to explain previously elusive findings such as the effects of lesioning and shuffling. Furthermore, we formalize this notion and use it to directly derive Residual and Highway networks. Finally, we present some preliminary experiments to compare these two architectures and investigate some of their relative advantages and disadvantages.
2 CHALLENGING THE REPRESENTATION VIEW
This section provides a brief survey of some the findings and points of contention that seem to contradict a representation view of Highway and Residual networks.
Staying Close to the Inputs. The success of ResNets has been partly attributed to the fact that they obviate the need to learn the identity mapping, which is difficult. However, learning the negative identity (so that a feature can replaced by a higher level one) should be at least as difficult. The fact that the residual form is useful indicates that Residual blocks typically stay close to the input representation, rather than replacing it.
The analysis by Srivastava et al. (2015a) shows that in trained Highway networks, the activity of the transform gates is often sparse for each individual sample, while their average activity over all training samples is non-sparse. Most units learn to copy their inputs and only replace features selectively. Again, this means that most of the features are propagated unchanged rather than being combined
1We refer to the building blocks of a ResNet—a few layers with an identity skip connection—as a Residual block (He et al., 2015). Analogously, in a Highway network, we refer to a collection of layers with a gated skip connection as a Highway block. See Figure 1 for an illustration.
Figure 2: (a) A single neural network layer that directly computes the desired representation. (b) The unrolled iterative estimation stage (e.g. from a Residual network) stretches the computation over three layers by first providing a noisy estimate of that representation, but then iteratively refines it over the next to layers. (c) A classic group of three layers can also distribute the computation, but they would produce a new representation at each layer. The iterative estimation stage in (b) can be seen as a middle ground between a single classic neural network layer, (a), and multiple classic layers, (c).
and changed between layers—an observation that contradicts the idea of building a new level of abstraction at each layer.
Lesioning. If it were true that each layer computes a completely new set of features, then removing a layer from a trained network would completely change the input distribution for the next layer. We would then expect to see the overall performance drop to almost chance level. This is in fact what Veit et al. (2016) find for the 15-layer VGG network on CIFAR-10: removing any layer from the trained network sets the classification error to around 90%. But the lesioning studies conducted on Highway networks (Srivastava et al., 2015a) and ResNets (Veit et al., 2016) paint an entirely different picture: only a minor drop in performance is observed for any removed layer. This drop is more pronounced for the early layers and the layers that change dimensionality (i.e. number of filter maps and map sizes), but performance is always still far superior to random guessing.
Huang et al. (2016b) take lesioning one step further and drop out entire ResNet layers as a regularizer during training. They describe their method as “[...] a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time”. The regularization effect of this procedure is explained as inducing an implicit ensemble of many shallow networks akin to normal dropout. Note that this explanation requires a departure from the representation view in that each layer has to cope with the possibility of having its entire input layer removed. Otherwise, most shallow networks in the ensemble would perform no better than chance level, just like the lesioned VGG net.
Reshuffling. The link between layers and representation levels may be most clearly challenged by an experiment in Veit et al. (2016) where the layers of a trained 110-layer ResNet are reshuffled. Remarkably, error increases smoothly with the amount of reshuffling, and many re-orderings result only in a small increase in error. Note, however, that only layers within a stage are reshuffled, since the dimensionality of the swapped layers must match. Veit et al. (2016) take these results as evidence that ResNets behave as ensembles of exponentially many shallow networks.
3 UNROLLED ITERATIVE ESTIMATION VIEW
The representation view has guided neural networks research by providing intuitions about the “meaning” of their computations. In this section we will augment the representation view to deal with the incongruities and hopefully enable future research on these very deep architectures to reap the same benefits. The target of our modification is the mapping of layers/blocks of the network to levels of abstraction.
At this point it is interesting to note that the one-to-one mapping of neural network layers to levels of abstraction is an implicit assumption rather than a stated part of the representation view. A recent deep learning textbook (Goodfellow et al., 2016) explicitly states: “[. . . ] the depth flowchart of the computations needed to compute the representation of each concept may be much deeper than the graph of the concepts themselves.” So in a strict sense the evidence from Section 2 does not in fact contradict a representation view of Residual and Highway networks. It only conflicts with the idea
that each layer forms a new level of representation. We can therefore reconcile very deep networks with the representation view by explicitly giving up this assumption.
Unrolled Iterative Estimation. We propose to think of blocks in Highway and Residual networks as performing unrolled iterative estimation of representations. By that we mean that the blocks in a stage work together to estimate and iteratively refine a single level of representation. The first layer in that stage already provides a (rough) estimate for the final representation. Subsequent layer in the stage then refine that estimate without changing the level of representation. So if the first layer in a stage detects simple shapes, then the rest of the layers in that stage will work at that level too.
A good initial estimate for a representation should on average be correct even though it might have high variance. We can thus formalize the notion of "preserving feature identity" as being an unbiased estimator for the target representation. This means the units aki in different layers k ∈ {1 . . . L} are all estimators for the same latent feature Ai, where Ai refers to the (unknown) value towards which the i-th feature is converging. The unbiased estimator condition can then be written as the expected difference between the estimator and the final feature:
E x∈X [aki −Ai] = 0. (1)
Note that both the aki s and Ai depend on the samples x of the data-generating distribution X and are thus random variables. The fact that they both depend on the same x is also the reason we need to keep them within the same expectation and cannot just write E[aki ] = Ai.
Feature Identity. A stage that performs iterative estimation is different from one that computes a new level of representation at each block because it preserves the feature identity. They operate differently even if their structure and their final representations are equivalent, because of the way they treat intermediate representations. This is illustrated in Figure 2, where the iterative estimation stage, (b), is contrasted with a single classic block (a), and multiple classic blocks, (c). In the iterative estimation case (middle), all the blocks within the stage produce estimates of the same representation (indicated by having different shades of blue). Whereas, in a classical stage, (c), the intermediate representations would all be different (represented by different colors).
3.1 HIGHWAY AND RESIDUAL NETWORKS
Both Highway and Residual networks address the problem of training very deep architectures by improving the error flow via identity skip connections that allow units to copy their inputs on to the next layer unchanged. This design principle was originally introduced in Long Short-Term Memory (LSTM) recurrent networks (Hochreiter & Schmidhuber, 1997) and mathematically these architectures correspond to a simplified LSTM network, "unrolled" over time.
In Highway Networks, for each unit there are two additional gating units, which control how much (typically non-linear) transformation is applied (transform gate T ) and how much to just copy of the activation from the corresponding unit in the previous layer (carry gate C). Let H(x) be a nonlinear parametric function of the inputs, x, (typically an affine projection followed by pointwise non-linearity). Then a traditional feed-forward network layer can be written as:
y(x) = H(x). (2)
By adding two additional units, T (x) and C(x) a Highway layer can be written as:
y(x) = H(x) · T (x) + x · C(x). (3)
Usually this is further simplified by coupling the gates, i.e. setting C(x) = 1− T (x):
y(x) = H(x) · T (x) + x · (1− T (x)). (4)
ResNets simplify the Highway networks approach by reformulating the desired transformation as the input plus a residual F (x). The rationale behind this is that it is easier to optimize the residual form than the original function. For the extreme case where the desired function is the identity, this amounts to the trivial task of pushing the residual to zero:
y(x) = F (x) + x. (5)
As with Highway networks, Residual networks can be viewed as unfolded recurrent neural networks of the particular mathematical form (one with an identity self-connection) of an LSTM cell. This has been explicitly pointed out by Liao & Poggio (2016), who also argue that this could allow Residual networks to emulate recurrent processing in the visual cortex and thus adds to their biological plausibility. Setting F (x) = T (x)[H(x)− x] converts Equation 5 to Equation 4 showing that both formulations differ only in the precise functional form for F . Alternatively, Residual networks can be seen as a particular case of Highway networks where C(x) = T (x) = 1 and are not learned.
3.2 DERIVING RESIDUAL NETWORKS
Equation 1 can be used to directly derive the ResNet equation (Equation 5). First, it follows that the expected difference between outputs of two consecutive blocks in a stage is zero:
E[aki −Ai]− E[ak−1i −Ai] = 0 (6) E[aki − ak−1i ] = 0. (7)
If we write feature aki as a combination of a k−1 i and a residual Fi, it follows from Equation 7 that the residual has to be zero-mean:
aki = a k−1 i + Fi (8)
=⇒ E[Fi] = 0. (9)
Therefore, if the residual block F has a zero mean over the training set, then Equation 1 holds and it can be said to maintain feature identity. Note that this is a reasonable assumption, especially when using batch normalization.
3.3 DERIVING HIGHWAY NETWORKS
The coupled Highway formula (Equation 4) can be directly derived as an alternative way of ensuring Equation 1 if we assume a Hi to be a new estimate of Ai. Highway layers then result from the optimal way to linearly combine the former estimate ak−1i with Hi such that the resulting a k i is a minimum variance estimate of Ai, i.e. requiring E[aki −Ai] = 0 and that Var[aki −Ai] is minimal.
Let α1 = Var[aki −Ai]−Cov[aki −Ai, aki −Hi] and α2 = Var[Hi−Ai]−Cov[aki −Ai, aki −Hi], then the optimal linear way of combining them is then given by the following estimator (see Section A.1 for derivation):
ak+1i = α2
α1 + α2 aki + α1 α1 + α2 Hi. (10)
If we use a neural network to compute Hi and another one to compute Ti = α1α1+α2 , then we recover the Highway formula: aki = Hi · Ti + ak−1i · (1− Ti), (11) where Hi and Ti are both functions of the previous layer activations ak−1.
4 DISCUSSION
4.1 IMPLICATIONS FOR HIGHWAY NETWORKS
In Highway networks with coupled gates the mixing coefficients always sum to one. This ensures that the expectation of the new estimate will always be correct (cf. Equation 14). The precise value of mixing will only determine the variance of the new estimate. We can bound this variance to be less or equal to the variance of the previous layer by restricting both mixing coefficients to be positive. In Highway networks this is done by using the logistic sigmoid activation function for the transform gate Ti. This restriction is equivalent to the assumption of α1 and α2 having the same sign. This assumption holds, for example, if the error of the new estimate Hi − Ai is independent of the old ak−1i −Ai. Because in that case their covariance is zero and thus both alphas are positive. Using the logistic sigmoid as activation function for the transform gate further means that the preactivation of Ti implicitly estimates log(α2α1 ). This is easy to see because the logistic sigmoid of that
term is 1
1 + elog( α2 α1
) =
1
1 + α2α1 = α1 α1 + α2 . (12)
For the simple case of independent estimates (Cov[aki −Ai, aki −Hi] = 0), this gives us another way of understanding the transform gate bias: It controls our initial belief in the variance of the layers estimate as compared to the previous one. A low bias means that the layers on average produce a high variance estimate, and should thus only contribute little, which seems a reasonable assumption for initialization.
4.2 EXPERIMENTAL CORROBORATION OF ITERATIVE ESTIMATION VIEW
The primary prediction of the iterative estimation view is that the estimation error for Highway or Residual blocks within the same stage should be zero in expectation. To empirically test this claim, we extract the intermediate layer outputs for 5000 validation set images using the 50-layer ResNet trained on the ILSVRC-2015 dataset from He et al. (2015). These are then used to compute the empirical mean and standard deviation of the estimation error over the validation subset, for all blocks in the four Residual stages in the network. Finally the mean of the empirical mean and standard deviation is computed over the three spatial dimensions.
Figure 3 shows that for the first three stages, the mean estimation error is indeed close to zero. This indicates that it is valid to interpret the role of Residual blocks in this network as that of iteratively refining a representation. Moreover, in each stage the standard deviation of the estimation error decreases over successive blocks, indicating the convergence of the refinement procedure. We note that stage four (with three blocks) appears to be underestimating the representation values, indicating a probable weak link in the architecture.
4.3 VISUAL EVIDENCE & STAGE-WISE ESTIMATION OF FEATURES
ResNets (He et al., 2015) and many other derived architectures share some common characteristics: They are divided into stages of Residual blocks that share the same dimensionality. In between these stages the input dimensionality changes, typically by down-sampling and an increase in the number of channels. These stages typically also increase in length: the early stages consist of fewer layers compared to later ones.
We can now interpret these design choices from an iterative estimation point of view. From this perspective the level of representation stays the same within each stage, through the use of identity shortcut connections. Between stages, the level of representation is changed by the use of a projection to change dimensionality. This means that we expect the type of features that are detected to be very similar within a stage and jump in abstraction between stages. This view also suggests that the first few stages can be shorter, since low level representations tend to be relatively simple and need little
iterative refinement. The features of later stages on the other hand are likely complex with numerous inter-dependencies and therefore benefit more from iterative refinement.
Many visualization studies (such as those by Zeiler & Fergus (2014)) have examined the activities in trained convolutional networks and found evidence supporting the representation view. However, these studies were conducted on networks not designed for iterative estimation. The interpretation above paints a different picture for networks which learn unrolled iterative estimation. In these networks, we should observe stages and not layers corresponding to levels of representation.
Indeed, visualization of Residual network features supports the iterative estimation view. In Figure 4 we reproduce visualizations from a study by Chu et al. (2017) who observe: “[. . . ] residual layers of the same dimensionality learn features that get refined and sharpened”. These visualizations show how the response of a single filter changes over three Residual blocks within the same stage of a 50-layer Residual network trained for image classification. Note that the filter appears to refine its response by including surrounding context, rather than changing it across blocks in the same stage. In the first block, the top nine activating patches for the filter include three light sources and six specular highlights. In later blocks, through the incorporation of spatial context, eight out of nine maximally activating patches are specular highlights. Similar refinement behavior is observed throughout the different stages of the network.
Another finding in line with this implication of the iterative estimation view is that in some cases sharing weights of the Residual blocks within a stage doesn’t deteriorate performance much (Liao & Poggio, 2016). Similarly Lu & Renals (2015) shared the weights of the transform and carry gates of a thin and deep highway network, while still achieving better performance than both normal deep neural networks and Residual networks.
4.4 REVISITING EVIDENCE AGAINST THE REPRESENTATION VIEW
Staying Close to the Inputs. When iteratively re-estimating a variable, staying close to the old value should be a more common operation than changing it significantly. This is the reason why the ResNet formulation makes sense: learning the identity is hard and it is needed frequently. It also explains sparse transform gate activity in trained Highway networks: These networks learn to dynamically and selectively update individual features, while keeping most of the representation intact.
Lesioning. Another implication of the iteration view is that processing in layers is incremental and somewhat interchangeable. Each layer (apart from the first) refines an already reasonable estimate of the representation. It follows that removing layers, like in the lesioning experiments, should have only a mild effect on the final result because doing so does not change the overall representation the next layer receives, only its quality. The following layer can still perform mostly the same operation, even with a somewhat noisy input. Layer dropout (Huang et al., 2016b) amplifies this effect by explicitly training the network to work with a variable number of iterations. By dropping random layers it further penalizes iterations relying on each other, which could be another explanation for the regularization effect of the technique.
Shuffling. The layers within a stage should also be interchangeable to a certain degree, because they all work with the same input and output representations. Of course, this interchangeability is
not without limitations. The network could learn to depend on a specific order of refinements, which would be disturbed by shuffling and lesioning. But we can expect these effects to be moderate in many cases, which is indeed what has been reported in the literature.
5 COMPARATIVE CASE STUDIES
The preceding sections show that we can construct both Highway and Residual architectures mathematically grounded in learning unrolled iterative estimation. The common feature between these architectures is that they preserve feature identities, and the primary difference is that they have different biases towards switching feature identities. Unfortunately, since our current understanding of the computations required to solve complex problems is limited, it is extremely hard to say a priori which architecture may be more suitable for which type of problems. Therefore, in this section we perform two case studies comparing and contrasting their behavior experimentally. The studies are each based on applications for which Residual and Highway layers respectively have been effective.
5.1 IMAGE CLASSIFICATION
Deep Residual networks outperformed all other entries at the 2016 ImageNet classification challenge. In this study we compare the performance of 50-layer convolutional Highway and Residual networks for ImageNet classification. Our aim is not to examine the importance of depth for this task— shallower networks have already outperformed deep Residual networks on all original Residual network benchmarks (Huang et al., 2016a; Szegedy et al., 2016). Instead, our goal is to fairly compare the two architectures, and test the following claims regarding deep convolutional Highway networks (He et al., 2015; 2016; Veit et al., 2016):
1. They are harder to train, leading to stalled training or poor results. 2. They require extensive tuning of the initial bias, and even then produce much worse results
compared to Residual networks. 3. They are wasteful in terms of parameters since they utilize extra learned gates, doubling the
total parameters for the same number of units compared to a Residual layer.
We train a 50-layer convolutional Highway network based on the 50-layer Residual network from He et al. (2015). The design of the two networks are identical (including use of batch normalization (BN) after every convolution operation), except that unlike Residual blocks, the Highway blocks use two sets of layers to learn H and T and then combine them using the coupled Highway formulation. We train two slight variations of the Highway network: Highway, in which H has the same design as in a Residual block before addition i.e. Conv-BN-ReLU-Conv-BN-ReLU-Conv-BN, and Highway-Full, in which an additional third ReLU operation is added. The design of T is Conv-BN-ReLU-Conv-BN-ReLU-Conv-BN-Sigmoid. As proposed initially for Highway layers, both H and T are learned using the same receptive fields and number of parameters. The transform gate biases are set to −1 at the start of training. For fair comparison, the number of feature maps throughout the Highway network is reduced such that the total number of parameters is close to the Residual network. The training algorithm and learning rate schedule are kept the same as those used for the Residual network.
The plots in Figure 5a show that the Residual network fits the data better—its final training loss is lower than the Highway network. The final performance of both networks on the validation set (see Table 1b) is very similar, with the Residual network producing a slightly better top-5 classification error of 7.17% vs. 7.53% for the Highway network. The Highway-Full network produces even closer results with a mean error of 7.29%. These results contradict claims 1 and 2 above, since the Highway networks are easy to train without requiring any bias tuning. However, there is some support for claim 3 since the Highway network appears to slightly underfit compared to the Residual network, suggesting lower capacity for the same number of parameters.
Importance of Expressive Gating. The mismatch between the results above and claims 1 and 2 made by He et al. (2016) can be explained based on the importance of having sufficiently expressive transform gates. For experiments with Highway networks (which they refer to as Residual networks with exclusive gating), He et al. (2016) used 1 × 1 convolutions for the transform gate, instead of having the same receptive fields for the gates as the primary transformation (H), as done by Srivastava et al. (2015a). This change in design appears to be the primary cause of instabilities in learning since the gates can no longer function effectively. Therefore, it is important to use equally expressive transformations for H and T in Highway networks.
Role of Batch Normalization. Since both architectures have built-in ease of optimization compared to plain networks, it is interesting to investigate the necessity of batch normalization for training these networks. Our derivation in Section 3.2 suggest that BN in Residual networks could take the role of an inductive bias towards iterative estimation by keeping the expected mean of the residual zero (cf. Equation 9). To investigate its role we train the networks above without any batch normalization. The resulting training curves are shown in Figure 5b of the supplementary.
We find that without BN both networks reach an even lower training error than before while performing worse on the validation set indicating increased overfitting for both. This shows that BN is not necessary for training these networks and does not speed up learning. Interestingly, the effect is more pronounced for the Highway network, which now fits the data better than the ResNet. This contradicts claim 3, since a Highway network with the same number of parameters as a Residual network demonstrates slightly higher capacity. On the other hand both networks produce a higher validation error—10.03% and 9.40% for the Highway and Residual network respectively—indicating a clear case of overfitting. This means that batch normalization provides regularization benefits that can’t easily be explained by either improved optimization nor by the inductive bias for Residual networks.
5.2 LANGUAGE MODELING
Next we compare different functional forms (or variants) of the Highway network formulation for the case of character-aware language modeling. Kim et al. (2015) have shown that utilizing a few Highway fully connected layers instead of conventional plain layers improves model performance for a variety of languages. The architecture consists of a stack of convolutional layers followed by Highway layers and then an LSTM layer which predicts the next word based on the history. Similar architectures have since been utilized for obtaining substantial improvements for large-scale language modeling (Jozefowicz et al., 2016) and character level machine translation (Lee et al., 2016). Highway layers with coupled gates have been used in all these studies.
Only two to four Highway layers were necessary to obtain significant modeling improvements in the studies above. Thus, it is reasonable to assume that the central advantage of using Highway layers for this task is not easing of credit assignment over depth, but an improved modeling bias. To test how well Residual and other variants of Highway networks perform, we compare several language models trained on the Penn Treebank dataset using the same setup and code provided by Kim et al. (2015). We use the LSTM-Char-Large model, only changing the two Highway layers to different variants. The following variants are tested:
Full The original Highway formulation based on the LSTM cell. We note that this variant uses more parameters than the others, since changing the layer size to reduce parameters would affect the rest of the network architecture as well.
Coupled The most commonly used Highway variant, derived in Section 3.3.
C-Only A Highway variant with a carry gate but no transform gate (always set to one). T-Only A Highway variant with a transform gate but no carry gate (always set to one). Residual The Residual form from He et al. (2015), in which both transform and carry gate are
always one. For this variant we use four layers instead of two, to match the amount of computation/parameters of the other variants.
The test set perplexity of each model is shown in Table 1a. We find that the the Full, Coupled and C-Only variants have similar performance, better than the T-Only variant and substantially better than the Residual variant. The Residual variant results in performance close to that obtained by using a single plain layer, even though four Residual layers are used. Learned gating of the identity connection is crucial for improving performance for this task.
Recall that the Highway layers transform character-aware representations before feeding them into an LSTM layer. Thus the non-contextual word-level representations resulting from the convolutional layers are transformed into representations better suited for contextual language modeling. Since it is unlikely that the entire representation needs to change completely, this setting fits well with the iterative estimation perspective.
Interestingly, Table 1a shows a significant advantage for all variants with a multiplicative gate on the inputs. These results suggest that in this setting it is crucial to dynamically replace parts of the input representation. Some features need to be changed drastically conditioned on other detected features such as word type while other features need to be retained. As a result, even though Residual networks are compatible with iterative estimation, they may not be the best choice for tasks where mixing adaptive feature transform/replacement and reuse is required.
6 CONCLUSION
This paper offers a new perspective on Highway and Residual networks as performing unrolled iterative estimation. As an extension of the popular representation view, it stands in contrast to the optimization perspective from which these architectures have originally been introduced. According to the new view, successive layers (within a stage) cooperate to compute a single level of representation. Therefore, the first layer already computes a rough estimate of that representation, which is then iteratively refined by the successive layers. Unlike layers in a conventional neural network, which each compute a new representation, these layers therefore preserve feature identity.
We have further shown that both Residual and Highway networks can be directly derived from this new perspective. This offers a unified theory from which these architectures can be understood as two approaches to the same problem. This view further provides a framework from which to understand several surprising recent findings like resilience to lesioning, benefits of layer dropout, and the mild negative effects of layer reshuffling. Together with the derivations these results serve as compelling evidence for the validity of our new perspective.
Motivated by their conceptual similarities we set out to compare Highway and Residual networks. In preliminary experiments we found that they give very similar results for networks of equal size, thus refuting some claims that Highway networks would need more parameters, or that any form of gating impairs the performance of Residual networks. In another example, we found non-gated identity skip-connections to perform significantly worse, and offered a possible explanation: If the task requires dynamically replacing individual features, then the use of gating is beneficial.
The preliminary evidence presented in this report is meant as a starting point for further investigation. We hope that the unrolled iterative estimation perspective will provide valuable intuitions to help guide research into understanding, improving and possibly combining these exciting techniques.
ACKNOWLEDGEMENTS
The authors wish to thank Faustino Gomez, Bas Steunebrink, Jonathan Masci, Sjoerd van Steenkiste and Christian Osendorfer for their feedback and support. We are grateful to NVIDIA Corporation for providing us a DGX-1 as part of the Pioneers of AI Research award. This research was supported by the EU project “INPUT” (H2020-ICT-2015 grant no. 687795).
A DERIVATION
A.1 OPTIMAL LINEAR ESTIMATOR
Assume two random variables A and B that are both noisy measurements of a third (latent) random variable C: E[A− C] = E[B − C] = 0 (13) We call the corresponding variances Var[A − C] = σ2A and Var[B − C] = σ2B and covariance Cov[A,B] = σ2AB .
We are looking for the linear estimator q(A,B) = q0+q1A+q2B of C with E[q−C] = 0 (unbiased) that has minimum variance.
E[q(A,B)− C] = 0 E[q0 + q1A+ q2B − C] = 0
E[q0 + q1A− q1C + q2B − q2C + (q1 + q2 − 1)C] = 0 E[q0 + q1(A− C) + q2(B − C) + (q1 + q2 − 1)C] = 0
q0 + (q1 + q2 − 1)E[C] = 0 E[C](1− q1 − q2) = q0
for all E[C] which is possible iff: q0 = 0 and q1 + q2 = 1. (14)
The second condition about minimal variance thus reduces to: minimize q1,q2 Var[q1A+ q2B − C]
subject to q1 + q2 = 1
We can solve this using Lagrangian multipliers. For that we need to take the derivative of the following term w.r.t. q1, q2 and λ and set them to zero:
Var[q1A+ q2B − C]− λ(q1 + q2 − 1))
The first equation is therefore:
d
dq1 (Var[q1A+ q2B − C]− λ(q1 + q2 − 1)) = 0
d
dq1 Var[q1A+ q2B − C]− λ = 0
d
dq1 Var[q1(A− C) + q2(B − C)]− λ = 0
d
dq1 (q21 Var[A− C] + 2q1q2 Cov[A− C,B − C])− λ = 0
2q1σ 2 A + 2q2σ 2 AB − λ = 0
Analogously we get: 2q2σ 2 B + 2q1σ 2 AB − λ = 0
and: q1 + q2 = 1
Solving these equations gives us:
q1 = σ2B − σ2AB
σ2A − 2σ2AB + σ2B (15)
q2 = σ2A − σ2AB
σ2A − 2σ2AB + σ2B (16)
(17)
We can write our estimator in terms of α1 = σ2B − σ2AB and α2 = σ2A − σ2AB :
q = α1
α1 + α2 A+ α2 α1 + α2 B

Thank you for an interesting angle on highway and residual networks. This paper shows a new angle to how and what kind of representations are learnt at each layer in the aforementioned models. Due to residual information being provided at a periodic number of steps, each of the layers preserve feature identity which prevents lesioning unlike convolutional neural nets.

Pros
- the iterative unrolling view was extremely simple and intuitive, which was supported by theoretical results and reasonable assumptions.
- Figure 3 gave a clear visualization for the iterative unrolling view

Cons
- Even though, the perspective is interesting few empirical results were shown to support the argument. The major experiments are image classification and language models trained on mutations of character-aware neural language models.
- Figure 4 and 5 could be combined and enlarged to show the effects of batch normalization.


IMPROVING NEURAL LANGUAGE MODELS WITH A CONTINUOUS CACHE
We propose an extension to neural network language models to adapt their prediction to the recent history. Our model is a simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation. This mechanism is very efficient and scales to very large memory sizes. We also draw a link between the use of external memory in neural network and cache models used with count based language models. We demonstrate on several language model datasets that our approach performs significantly better than recent memory augmented networks.
1 INTRODUCTION
Language models, which are probability distributions over sequences of words, have many applications such as machine translation (Brown et al., 1993), speech recognition (Bahl et al., 1983) or dialogue agents (Stolcke et al., 2000). While traditional neural networks language models have obtained state-of-the-art performance in this domain (Jozefowicz et al., 2016; Mikolov et al., 2010), they lack the capacity to adapt to their recent history, limiting their application to dynamic environments (Dodge et al., 2015). A recent approach to solve this problem is to augment these networks with an external memory (Graves et al., 2014; Grefenstette et al., 2015; Joulin & Mikolov, 2015; Sukhbaatar et al., 2015). These models can potentially use their external memory to store new information and adapt to a changing environment.
While these networks have obtained promising results on language modeling datasets (Sukhbaatar et al., 2015), they are quite computationally expensive. Typically, they have to learn a parametrizable mechanism to read or write to memory cells (Graves et al., 2014; Joulin & Mikolov, 2015). This may limit both the size of their usable memory as well as the quantity of data they can be trained on. In this work, we propose a very light-weight alternative that shares some of the properties of memory augmented networks, notably the capability to dynamically adapt over time. By minimizing the computation burden of the memory, we are able to use larger memory and scale to bigger datasets. We observe in practice that this allows us to surpass the perfomance of memory augmented networks on different language modeling tasks.
Our model share some similarities with a model proposed by Kuhn (1988), called the cache model. A cache model stores a simple representation of the recent past, often in the form of unigrams, and uses them for prediction (Kuhn & De Mori, 1990). This contextual information is quite cheap to store and can be accessed efficiently. It also does not need any training and can be appplied on top of any model. This makes this model particularly interesting for domain adaptation (Kneser & Steinbiss, 1993).
Our main contribution is to propose a continuous version of the cache model, called Neural Cache Model, that can be adapted to any neural network language model. We store recent hidden activations and use them as representation for the context. Using simply a dot-product with the current hidden activations, they turn out to be extremely informative for prediction. Our model requires no training and can be used on any pre-trained neural networks. It also scales effortlessly to thousands of memory cells. We demonstrate the quality of the Neural Cache models on several language model tasks and the LAMBADA dataset (Paperno et al., 2016).
2 LANGUAGE MODELING
A language model is a probability distribution over sequences of words. Let V be the size of the vocabulary; each word is represented by a one-hot encoding vector x in RV = V , corresponding to its index in the vocabulary. Using the chain rule, the probability assigned to a sequence of words x1, . . . , xT can be factorized as
p(x1, ..., xT ) = T∏ t=1 p(xt | xt−1, ..., x1).
Language modeling is often framed as learning the conditional probability over words, given the history (Bahl et al., 1983).
This conditional probability is traditionally approximated with non-parameteric models based on counting statistics (Goodman, 2001). In particular, smoothed N-gram models (Katz, 1987; Kneser & Ney, 1995) achieve good performance in practice (Mikolov et al., 2011). Parametrized alternatives are either maximum entropy language models (Rosenfeld, 1996), feedforward networks (Bengio et al., 2003) or recurrent networks (Mikolov et al., 2010). In particular, recurrent networks are currently the best solution to approximate this conditional probability, achieving state-of-the-arts performance on standard language modeling benchmarks (Jozefowicz et al., 2016; Zilly et al., 2016).
Recurrent networks. Assuming that we have a vector ht ∈ Rd encoding the history xt, ..., x1, the conditional probability of a word w can be parametrized as
pvocab(w | xt, ..., x1) ∝ exp(h>t ow).
The history vector ht is computed by a recurrent network by recursively applying an equation of the form
ht = Φ (xt, ht−1) ,
where Φ is a function depending on the architecture of the network. Several architecture for recurrent networks have been proposed, such as the Elman network (Elman, 1990), the long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) or the gated recurrent unit (GRU) (Chung et al., 2014). One of the simplest recurrent networks is the Elman network (Elman, 1990), where
ht = σ (Lxt +Rht−1) ,
where σ is a non-linearity such as the logistic or tanh functions, L ∈ Rd×V is a word embedding matrix and R ∈ Rd×d is the recurrent matrix. The LSTM architecture is particularly interesting in the context of language modelling (Jozefowicz et al., 2016) and we refer the reader to Graves et al. (2013) for details on this architecture.
The parameters of recurrent neural network language models are learned by minimizing the negative log-likelihood of the training data. This objective function is usually minimized by using the stochastic gradient descent algorithm, or variants such as Adagrad (Duchi et al., 2011). The gradient is computed using the truncated backpropagation through time algorithm (Werbos, 1990; Williams & Peng, 1990).
Cache model. After a word appears once in a document, it is much more likely to appear again. As an example, the frequency of the word tiger on the Wikipedia page of the same name is 2.8%, compared to 0.0037% over the whole Wikipedia. Cache models exploit this simple observation to improve n-gram language models by capturing long-range dependencies in documents. More precisely, these models have a cache component, which contains the words that appeared in the recent history (either the document or a fixed number of words). A simple language model, such as a unigram or smoothed bigram model, is fitted on the words of the cache and interpolated with the static language model (trained over a larger dataset). This technique has many advantages. First, this is a very efficient way to adapt a language model to a new domain. Second, such models can predict out-of-vocabulary words (OOV words), after seeing them once. Finally, this helps capture long-range dependencies in documents, in order to generate more coherent text.
h1 h2 h3 h4
x1 x2 x3 x4
(h1, x2) (h2, x3) (h3, x4) x5
L L L L
R R R
Id Id Id O
Id
Figure 1: The neural cache stores the previous hidden states in memory cells. They are then used as keys to retrieve their corresponding word, that is the next word. There is no transformation applied to the storage during writing and reading.
3 NEURAL CACHE MODEL
The Neural Cache Model adds a cache-like memory to neural network language models. It exploits the hidden representations ht to define a probability distribution over the words in the cache. As illustrated Figure 1, the cache stores pairs (hi, xi+1) of a hidden representation, and the word which was generated based on this representation (we remind the reader that the vector hi encodes the history xi, ..., x1). At time t, we then define a probability distribution over words stored in the cache based on the stored hidden representations and the current one ht as
pcache(w | h1..t, x1..t) ∝ t−1∑ i=1 1{w=xi+1} exp(θh > t hi)
where the scalar θ is a parameter which controls the flatness of the distribution. When θ is equal to zero, the probability distribution over the history is uniform, and our model is equivalent to a unigram cache model (Kuhn & De Mori, 1990).
From the point of view of memory-augmented neural networks, the probability pcache(w | h1..t, x1..t) given by the neural cache model can be interpreted as the probability to retrieve the word w from the memory given the query ht, where the desired answer is the next word xt+1. Using previous hidden states as keys for the words in the memory, the memory lookup operator can be implemented with simple dot products between the keys and the query. In contrast to existing memory-augmented neural networks, the neural cache model avoids the need to learn the memory lookup operator. Such a cache can thus be added to a pre-trained recurrent neural language model without fine tuning of the parameters, and large cache size can be used with negligible impact on the computational cost of a prediction.
Neural cache language model. Following the standard practice in n-gram cache-based language models, the final probability of a word is given by the linear interpolation of the cache language model with the regular language model, obtaining:
p(w | h1..t, x1..t) = (1− λ)pvocab(w | ht) + λpcache(w | h1..t, x1..t) .
Instead of taking a linear interpolation between the two distribution with a fixed λ, we also consider a global normalization over the two distribution:
p(w | h1..t, x1..t) ∝ ( exp(h>t ow) + t−1∑ i=1 1{w=xi+1} exp(θh > t hi + α) ) .
This corresponds to taking a softmax over the vocabulary and the words in the cache. The parameter α controls the weight of the cache component, and is the counterpart of the λ parameter for linear interpolation.
The addition of the neural cache to a recurrent neural language model inherits the advantages of ngram caches in usual cache-based models: The probability distribution over words is updated online depending on the context, and out-of-vocabulary words can be predicted as soon as they have been seen at least once in the recent history. The neural cache also inherits the ability of the hidden states of recurrent neural networks to model longer-term contexts than small n-grams, and thus allows for a finer modeling of the current context than e.g., unigram caches.
Training procedure. For now, we first train the (recurrent) neural network language model, without the cache component. We only apply the cache model at test time, and choose the hyperparameters θ and λ (or α) on the validation set. A big advantage of our method is that it is very easy and cheap to apply, with already trained neural models. There is no need to perform backpropagation over large contexts, and we can thus apply our method with large cache sizes (larger than one thousand).
4 RELATED WORK
Cache model. Adding a cache to a language model was intoducted in the context of speech recognition(Kuhn, 1988; Kupiec, 1989; Kuhn & De Mori, 1990). These models were further extended by Jelinek et al. (1991) into a smoothed trigram language model, reporting reduction in both perplexity and word error rates. Della Pietra et al. (1992) adapt the cache to a general n-gram model such that it satisfies marginal constraints obtained from the current document.
Adaptive language models. Other adaptive language models have been proposed in the past: Kneser & Steinbiss (1993) and Iyer & Ostendorf (1999) dynamically adapt the parameters of their model to the recent history using different weight interpolation schemes. Bellegarda (2000) and Coccaro & Jurafsky (1998) use latent semantic analysis to adapt their models to the current context. Similarly, topic features have been used with either maximum entropy models (Khudanpur & Wu, 2000) or recurrent networks (Mikolov & Zweig, 2012; Wang & Cho, 2015). Finally, Lau et al. (1993) proposes to use pairs of distant of words to capture long-range dependencies.
Memory augmented neural networks. In the context of sequence prediction, several memory augmented neural networks have obtained promising results (Sukhbaatar et al., 2015; Graves et al., 2014; Grefenstette et al., 2015; Joulin & Mikolov, 2015). In particular, Sukhbaatar et al. (2015) stores a representation of the recent past and accesses it using an attention mechanism Bahdanau et al. (2014). Sukhbaatar et al. (2015) shows that this reduces the perplexity for language modeling.
This approach has been successfully applied to question answering, when the answer is contained in a given paragraph (Chen et al., 2016; Hermann et al., 2015; Kadlec et al., 2016; Sukhbaatar et al., 2015). Similarly, Vinyals et al. (2015) explores the use of this mechanism to reorder sequences of tokens. Their network uses an attention (or “pointer”) over the input sequence to predict which element should be selected as the next output. Gulcehre et al. (2016) have shown that a similar mechanism called pointer softmax could be used in the context of machine translation, to decide which word to copy from the source to target.
Independently of our work, Merity et al. (2016) apply the same mechanism to recurrent network. Unlike our work, they uses the current hidden activation as a representation of the current input (while we use it to represent the output). This requires additional learning of a transformation between the current representation and those in the past. The advantage of our approach is that we can scale to very large caches effortlessly.
5 EXPERIMENTS
In this section, we evaluate our method on various language modeling datasets, which have different sizes and characteristics. On all datasets, we train a static recurrent neural network language model with LSTM units. We then use the hidden representations from this model to obtain our cache, which is interpolated with the static LSTM model. We also evaluate a unigram cache model interpolated with the static model as another baseline.
5.1 SMALL SCALE EXPERIMENTS
Datasets. In this section, we describe experiments performed on two small datasets: the Penn Tree Bank (Marcus et al., 1993) and the wikitext2 (Merity et al., 2016) datasets. The Penn Tree Bank dataset is made of articles from the Wall Street Journal, contains 929k training tokens and has a vocabulary size of 10k. The wikitext2 dataset is derived from Wikipedia articles, contains 2M training tokens and has a vocabulary size of 33k. These datasets contain non-shuffled documents, therefore requiring models to capture inter-sentences dependencies to perform well.
Implementation details. We train recurrent neural network language models with 1024 LSTM units, regularized with dropout (probability of dropping out units equals to 0.65). We use the Adagrad algorithm, with a learning rate of 0.2, a batchsize of 20 and initial weight uniformly sampled in the range [−0.05, 0.05]. We clip the norm of the gradient to 0.1 and unroll the network for 30 steps. We consider cache sizes on a logarithmic scale, from 50 to 10, 000, and fit the cache hyperparameters on the validation set.
Results. We report the perplexity on the validation sets in Figures 2 and 3, for various values of hyperparameters, for linear interpolation and global normalization. First, we observe that on both datasets, the linear interpolation method performs slightly better than the global normalization approach. It is also easier to apply in practice, and we thus use this method in the remainder of this paper. In Tables 1 and 2, we report the test perplexity of our approach and state-of-the-art models. Our approach is competitive with previous models, in particular with the pointer sentinel LSTM model of Merity et al. (2016). On Penn Tree Bank, we note that the improvement over the base model is similar for both methods. On the wikitext2 dataset, both methods obtain similar results when using the same cache size (100 words). Since our method is computationally cheap, it is easy to increase the cache to larger values (2, 000 words), leading to dramatic improvements (30% over the baseline, 12% over a small cache of 100 words).
5.2 MEDIUM SCALE EXPERIMENTS
Datasets and implementation details. In this section, we describe experiments performed over two medium scale datasets: text8 and wikitext103. Both datasets are derived from Wikipedia, but different pre-processing were applied. The text8 dataset contains 17M training tokens and has a vocabulary size of 44k words, while the wikitext103 dataset has a training set of size 103M, and a vocabulary size of 267k words. We use the same setting as in the previous section, except for the batchsize (we use 128) and dropout parameters (we use 0.45 for text8 and 0.25 for wikitext103). Since both datasets have large vocabularies, we use the adaptive softmax (Grave et al., 2016) for faster training.
Results. We report the test perplexity as a function of the cache size in Figure 4, for the neural cache model and a unigram cache baseline. We observe that our approach can exploits larger cache sizes, compared to the baseline. In Table 2, we observe that the improvement in perplexity of our method over the LSTM baseline on wikitext103 is smaller than for wikitext2 (approx. 16% v.s. 30%). The fact that improvements obtained with more advanced techniques decrease when the size of training data increases has already been observed by Goodman (2001). Both wikitext datasets sharing the same test set, we also observe that the LSTM baseline, trained on 103M tokens (wikitext103), strongly outperforms more sophisticated methods, trained on 2M tokens (wikitext2). For these two reasons, we believe that it is important to evaluate and compare methods on relatively large datasets.
5.3 EXPERIMENTS ON THE LAMBADA DATASET
Finally, we report experiments carried on the lambada dataset, introduced by Paperno et al. (2016). This is a dataset of short passages extracted from novels. The goal is to predict the last word of the excerpt. This dataset was built so that human subjects solve the task perfectly when given the full context (approx. 4.6 sentences), but fail to do so when only given the sentence with the target word. Thus, most state-of-the-art language models fail on this dataset. The lambada training set contains approximately 200M tokens and has a vocabulary size of 93, 215. We report results for our method in Table 3, as well the performance of baselines from Paperno et al. (2016). Adding a neural cache model to the LSTM baseline strongly improves the performance on the lambada dataset. We also observe in Figure 5 that the best interpolation parameter between the static model and the cache is not the same for the development and control sets. This is due to the fact that more than 83% of passages of the development set include the target word, while this is true for only 14% of the control set. Ideally, a model should have strong results on both sets. One possible generalization of our model would be to adapt the interpolation parameter based on the current vector representation of the history ht.
6 CONCLUSION
We presented the neural cache model to augment neural language models with a longer-term memory that dynamically updates the word probablilities based on the long-term context. A neural cache can be added on top of a pre-trained language model at negligible cost. Our experiments on both language modeling tasks and the challenging LAMBADA dataset shows that significant performance gains can be expected by adding this external memory component.
Technically, the neural cache models is similar to some recent memory-augmented neural networks such as pointer networks. However, its specific design makes it possible to avoid learning the memory lookup component. This makes the neural cache appealing since it can use larger cache sizes than memory-augment networks and can be applied as easily as traditional count-based caches.

The authors present a simple method to affix a cache to neural language models, which provides in effect a copying mechanism from recently used words. Unlike much related work in neural networks with copying mechanisms, this mechanism need not be trained with long-term backpropagation, which makes it efficient and scalable to much larger cache sizes. They demonstrate good improvements on language modeling by adding this cache to RNN baselines.
The main contribution of this paper is the observation that simply using the hidden states h_i as keys for words x_i, and h_t as the query vector, naturally gives a lookup mechanism that works fine without tuning by backprop. This is a simple observation and might already exist as folk knowledge among some people, but it has nice implications for scalability and the experiments are convincing.
The basic idea of repurposing locally-learned representations for large-scale attention where backprop would normally be prohibitively expensive is an interesting one, and could probably be used to improve other types of memory networks.
My main criticism of this work is its simplicity and incrementality when compared to previously existing literature. As a simple modification of existing NLP models, but with good empirical success, simplicity and practicality, it is probably more suitable for an NLP-specific conference. However, I think that approaches that distill recent work into a simple, efficient, applicable form should be rewarded and that this tool will be useful to a large enough portion of the ICLR community to recommend its publication.


This paper proposes dynamic chunk reader (DCR), an end-to-end neural reading comprehension (RC) model that is able to extract and rank a set of answer candidates from a given document to answer questions. DCR is able to predict answers of variable lengths, whereas previous neural RC models primarily focused on predicting single tokens or entities. DCR encodes a document and an input question with recurrent neural networks, and then applies a word-by-word attention mechanism to acquire question-aware representations for the document, followed by the generation of chunk representations and a ranking module to propose the topranked chunk as the answer. Experimental results show that DCR could achieve a 66.3% Exact match and 74.7% F1 score on the Stanford Question Answering Dataset (Rajpurkar et al., 2016).
1 INTRODUCTION
Reading comprehension-based question answering (RCQA) is the task of answering a question with a chunk of text taken from related document(s). A variety of neural models have been proposed recently either for extracting a single entity or a single token as an answer from a given text (Hermann et al., 2015; Kadlec et al., 2016; Trischler et al., 2016b; Dhingra et al., 2016; Chen et al., 2016; Sordoni et al., 2016; Cui et al., 2016a); or for selecting the correct answer by ranking a small set of human-provided candidates (Yin et al., 2016; Trischler et al., 2016a). In both cases, an answer boundary is either easy to determine or already given.
Different from the above two assumptions for RCQA, in the real-world QA scenario, people may ask questions about both entities (factoid) and non-entities such as explanations and reasons (nonfactoid) (see Table 1 for examples).
In this regard, RCQA has the potential to complement other QA approaches that leverage structured data (e.g., knowledge bases) for both the above question types. This is because RCQA can exploit the textual evidences to ensure increased answer coverage, which is particularly helpful for nonfactoid answers. However, it is also challenging for RCQA to identify answer in arbitrary position in the passage with arbitrary length, especially for non-factoid answers which might be clauses or sentences.
As a result, apart from a few exceptions (Rajpurkar et al., 2016; Wang & Jiang, 2016), this research direction has not been fully explored yet.
Compared to the relatively easier RC task of predicting single tokens/entities1, predicting answers of arbitrary lengths and positions significantly increase the search space complexity:
the number of possible candidates to consider is in the order of O(n2), where n is the number of passage words. In contrast, for previous works in which answers are single tokens/entities or from candidate lists, the complexity is in O(n) or the size of candidate lists l (usually l ≤5), respectively. To address the above complexity, Rajpurkar et al. (Rajpurkar et al., 2016) used a two-step chunkand-rank approach that employs a rule-based algorithm to extract answer candidates from a passage,
∗Both authors contribute equally 1State-of-the-art RC models have a decent accuracy of ∼70% on the widely used CNN/DailyMail dataset
(Hermann et al., 2015).
followed by a ranking approach with hand-crafted features to select the best answer. The rule-based chunking approach suffered from low coverage (≈ 70% recall of answer chunks) that cannot be improved during training; and candidate ranking performance depends greatly on the quality of the hand-crafted features. More recently, Wang and Jiang (Wang & Jiang, 2016) proposed two end-toend neural network models, one of which chunks a candidate answer by predicting the answer’s two boundary indices and the other classifies each passage word into answer/not-answer. Both models improved significantly over the method proposed by Rajpurkar et al. (Rajpurkar et al., 2016).
Our proposed model, called dynamic chunk reader (DCR), not only significantly differs from both the above systems in the way that answer candidates are generated and ranked, but also shares merits with both works. First, our model uses deep networks to learn better representations for candidate answer chunks, instead of using fixed feature representations as in (Rajpurkar et al., 2016). Second, it represents answer candidates as chunks, as in (Rajpurkar et al., 2016), instead of wordlevel representations (Wang & Jiang, 2016), to make the model aware of the subtle differences among candidates (importantly, overlapping candidates).
The contributions of this paper are three-fold. (1) We propose a novel neural network model for joint candidate answer chunking and ranking, where the candidate answer chunks are dynamically constructed and ranked in an end-to-end manner. (2) we propose a new question-attention mechanism to enhance passage word representation, which is subsequently used to construct chunk representations. (3) We also propose several simple but effective features to strengthen the attention mechanism, which fundamentally improves candidate ranking, with the by-product of higher exact boundary match accuracy.
The experiments on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), which contains a variety of human-generated factoid and non-factoid questions, have shown the effectiveness of above three contributions.
Our paper is organized as follows. We formally define the RCQA problem first. Next, we describe our baseline with a neural network component. We present the end-to-end dynamic chunk reader model next. Finally, we analyze our experimental results and discuss the related work. In appendix, we show formal equations and details of the model.
2 PROBLEM DEFINITION
Table 1 shows an example of our RC setting where the goal is to answer a question Qi, factoid (Q1) or non-factoid (Q2 and Q3), based on a supporting passage Pi, by selecting a continuous sequence of text Ai ⊆ Pi as answer. Qi, Pi, and Ai are all word sequences, where each word is drawn from a vocabulary, V . The i-th instance in the training set is a triple in the form of (Pi, Qi, Ai), where Pi = (pi1, . . . , pi|Pi|), Qi = (qi1, . . . , qi|Qi|), and Ai = (ai1, . . . , ai|Ai|) (pi·, qi·, ai· ∈ V ). Owing to the disagreement among annotators, there could be more than one correct answer for the same question; and the k-th answer to Qi is denoted by Aki = {aki1, . . . , aki|Aki |}. An answer candidate for the i-th training example is defined as cm,ni , a sub-sequence in Pi, that spans from position m to n (1 ≤ m ≤ n ≤ |Pi|). The ground truth answer Ai could be included in the set of all candidates
Ci = {cm,ni |∀m,n ∈ N+, subj(m,n, Pi) and 1 ≤ m ≤ n ≤ |Pi|}, where subj(m,n, Pi) is the constraint put on the candidate chunk for Pi, such as, “c m,n i can have at most 10 tokens”, or “cm,ni must have a pre-defined POS pattern”. To evaluate a system’s performance, its top answer to a question is matched against the corresponding gold standard answer(s).
Remark: Categories of RC Tasks Other simpler variants of the aforementioned RC task were explored in the past. For example, quiz-style datasets (e.g., MCTest (Richardson et al., 2013), MovieQA (Tapaswi et al., 2015)) have multiple-choice questions with answer options. Cloze-style datesets(Hermann et al., 2015; Hill et al., 2015; Onishi et al., 2016), usually automatically generated, have factoid “question”s created by replacing the answer in a sentence from the text with blank. For the answer selection task this paper focuses on, several datasets exist, e.g. TREC-QA for factoid answer extraction from multiple given passages, bAbI (Weston et al., 2014) designed for inference purpose, and the SQuAD dataset (Rajpurkar et al., 2016) used in this paper. To the best of our knowledge, the SQuAD dataset is the only one for both factoid and non-factoid answer extraction with a question distribution more close to real-world applications.
3 BASELINE: CHUNK-AND-RANK PIPELINE WITH NEURAL RC
In this section we modified a state-of-the-art RC system for cloze-style tasks for our answer extraction purpose, to see how much gap we have for the two type of tasks, and to inspire our end-to-end system in the next section. In order to make the cloze-style RC system to make chunk-level decision, we use the RC model to generate features for chunks, which are further used in a feature-based ranker like in (Rajpurkar et al., 2016). As a result, this baseline can be viewed as a deep learning based counterpart of the system in (Rajpurkar et al., 2016). It has two main components: 1) a standalone answer chunker, which is trained to produce overlapping candidate chunks, and 2) a neural RC model, which is used to score each word in a given passage to be used thereafter for generating chunk scores.
Answer Chunking To reduce the errors generated by the rule-based chunker in (Rajpurkar et al., 2016), first, we capture the part-of-speech (POS) pattern of all answer sub-sequences in the training dataset to form a POS pattern trie tree, and then apply the answer POS patterns to passage Pi to acquire a collection of all subsequences (chunk candidates) Ci whose POS patterns can be matched to the POS pattern trie. This is equivalent to putting an constraint subj(m,n, Pi) to candidate answer chunk generation process that only choose the chunk with a POS pattern seen for answers in the training data. Then the sub-sequences Ci are used as answer candidates for Pi. Note that overlapping chunks could be generated for a passage, and we rely on the ranker to choose the best candidate based on features from the cloze-style RC system. Experiments showed that for > 90% of the questions on the development set, the ground truth answer is included in the candidate set constructed in such manner.
Feature Extraction and Ranking For chunk ranking, we (1) use neural RCQA model to annotate each pij in passage Pi to get score sij , then (2) for every chunk c m,n i in passage i, collect scores (sim, . . . , sin) for all the (pim, ..., pin) contained within c m,n i , and (3) extract features on the sequence of scores (sim, . . . , sin) to characterize its scale and distribution information, which serves as the feature representation of cm,ni . In step (1) to acquire sij we train and apply a word-level single-layer Gated Attention Reader 2 (Dhingra et al., 2016), which has state-of-the-art performance on CNN/DailyMail cloze-style RC task. In step (3) for chunk cm,ni , we designed 5 features, including 4 statistics on (sim, . . . , sin): maximum, minimum, average and sum; as well as the count of matched POS pattern within the chunk, which serves as an answer prior. We use these 5 features in a state-of-the-art ranker (Ganjisaffar et al., 2011).
4 DYNAMIC CHUNK READER
The dynamic chunk reader (DCR) model is presented in Figure 1. Inspired by the baseline we built, DCR is deemed to be superior to the baseline for 3 reasons. First, each chunk has a representation constructed dynamically, instead of having a set of pre-defined feature values. Second, each passage
2We tried using more than one layers in Gated Attention Reader, but no improvement was observed.
word’s representation is enhanced by word-by-word attention that evaluates the relevance of the passage word to the question. Third, these components are all within a single, end-to-end model that can be trained in a joint manner.
DCR works in four steps. First, the encoder layer encodes passage and question separately, by using bidirectional recurrent neural networks (RNN).
Second, the attention layer calculates the relevance of each passage word to the question.
Third, the convolution layer generates unigram, bigram and trigram representation for each word. bigram and trigram of a word ends with the same word, and proper padding is applied on the first word to make sure the output is the same length as input to CNN layer.
Fourth, the chunk representation layer dynamically extracts the candidate chunks from the given passage, and create chunk representation that encodes the contextual information of each chunk.
Fifth, the ranker layer scores the relevance between the representations of a chunk and the given question, and ranks all candidate chunks using a softmax layer.
We describe each step below.
Encoder Layer We use bi-directional RNN encoder to encode Pi and Qi of example i, and get hidden state for each word position pij and qik.3 As RNN input, a word is represented by a row vector x ∈ Rn. x can be the concatenation of word embedding and word features (see Fig. 1). The word vector for the t-th word is xt. A word sequence is processed using an RNN encoder with gated recurrent units (GRU) (Cho et al., 2014), which was proved to be effective in RC and neural machine translation tasks (Bahdanau et al., 2015; Kadlec et al., 2016; Dhingra et al., 2016). For each position t, GRU computes ht with input xt and previous state ht−1, as:
3We can have separated parameters for question and passage encoders but a single shared encoder for both works better in the experiments.
rt = σ(Wrxt + Urht−1) (1) ut = σ(Wuxt + Uuht−1) (2) h̄t = tanh(Wxt + U(rt ht−1)) (3) ht = (1− ut) · ht−1 + ut · h̄t (4)
where ht, rt, and ut ∈ Rd are d-dimensional hidden state, reset gate, and update gate, respectively; W{r,u}, W ∈ Rn×d and U{r,u}, U ∈ Rd×d are the parameters of the GRU; σ is the sigmoid function, and denotes element-wise production. For a word at t, we use the hidden state −→h t from the forward RNN as a representation of the preceding context, and the←−h t from a backward RNN that encodes text reversely, to incorporate the context after t. Next, ht = [ −→ ht ; ←− ht ], the bi-directional contextual encoding of xt, is formed. [·; ·] is the concatenation operator. To distinguish hidden states from different sources, we denote the hj of j-th word in P and the hk of k-th word in Q as h p j and hqk respectively.
Attention Layer Attention mechanism in previous RC tasks (Kadlec et al., 2016; Hermann et al., 2015; Sordoni et al., 2016; Dhingra et al., 2016; Cui et al., 2016a;b) enables question-aware passage representations. We propose a novel attention mechanism inspired by word-by-word style attention methods (Rocktäschel et al., 2015; Wang & Jiang, 2015; Santos et al., 2016). For each pj , a questionattended representation vj is computed as follows (example index i is omitted for simplicity):
αjk = h p j · h q k, (5)
βj = |Q|∑ k=1 αjkh q k (6) vj = [h p j ;βj ] (7)
where hpj and h q k are hidden states from the bi-directional RNN encoders (see Figure 1). An inner product, αjk, is calculated between h p j and every question word h q k. It indicates how well the passage word pj matches with every question word qk. βj is a weighted pooling of |Q| question hidden states, which serves as a pj-aware question representation. The concatenation of h p j and βj leads to a passage-question joint representation, vj ∈ R4d.4 Next, we apply a second bi-GRU layer taking the vjs as inputs, and obtain forward and backward representations −→γj and←−γj ∈ Rd, and in turn their concatenation, γj = [−→γj ;←−γj ]. Convolution Layer Every word is encoded with complete passage context through attention layer RNN. We would like to model more complex representation of the words, by introducing unigram, bigram and trigram representations. There are two benefits for this enhanced representation: 1) each word could be enhanced with local context information to help identify the boundary of the answer chunk. Using previous words has been a common feature used in POS tagging and Named entity recognition; and 2) The information brought in by the ngram into the word representation could enhance the semantic match between the answer chunk internal and the question. Imagine scenario of a three word candidate, where the last word representation includes the two previous words through the convolution layer. Matching to the last word could also lead to the match to the semantics of the internal of the chunk. Specifically, we create for every word position j three representations, by using ngrams ending with the hidden state j:
γ̃j1 = γj ·Wc1 (8) γ̃j2 = [γj−1; γj ] ·Wc2 (9) γ̃j3 = [γj−2; γj−1; γj ] ·Wc3 (10)
4We tried another word-by-word attention methods as in (Santos et al., 2016), which has similar passage representation input to question side. However, this does not lead to improvement due to the confusion caused by long passages in RC. Consequently, we used the proposed simplified version of word-by-word attention on passage side only.
The details shown in equations above. We used three different convolution kernels for different n-grams.
Chunk Representation Layer A candidate answer chunk representation is dynamically created given convolution layer output. We first decide the text boundary for the candidate chunk, and then form a chunk representation using all or part of those γj outputs inside the chunk. To decide a candidate chunk (boundary): we tried two ways: (1) adopt the POS trie-based approach used in our baseline, and (2) enumerate all possible chunks up to a maximum number of tokens. For (2), we create up to N (max chunk length) chunks starting from any position j in Pj . Approach (1) can generate candidates with arbitrary lengths, but fails to recall candidates whose POS pattern is unseen in training set; whereas approach (2) considers all possible candidates within a window and is more flexible, but over-generates invalid candidates.
For a candidate answer chunk cm,n spanning from position m to n inclusively, we construct chunk representation γlm,n ∈ R2d using every γ̃jl within range [m,n], with a function g(·), and l ∈ {1, 2, 3}. Formally,
γlm,n = g(γ̃ml, . . . , γ̃nl)
Each γ̃jl is a convolution output over concatenated forward and backward RNN hidden states from attention layer. So the first half in γ̃jl encodes information in forward RNN hidden states and the second half encodes information in backward RNN hidden states. We experimented with several pooling functions (e.g., max, average) for g(·), and found out that, instead of pooling, the best g(·) function is to concatenate the first half of convolution output of the chunk’s first word and the second half of convolution output of the chunk’s last word. Formally,
γlm,n = g(γ̃ml, . . . , γ̃nl) = [ −→ γ̃ml; ←− γ̃nl] (11)
where −→γ̃ml is half of the hidden state for l-gram word representation corresponding to forward attention RNN output. We hypothesize that the hidden states at that two ends can better represent the chunk’s contexts, which is critical for this task, than the states within the chunk. This observation also agrees with (Kobayashi et al., 2016).
Ranker Layer A score slm,n for each l-gram chunk representation γlm,n denoting the probability of that chunk to be the true answer is calculated by dot product with question representation. The question representation is the concatenation of the last hidden state in forward RNN and the first hidden state in backward RNN. Formally for the chunk cm,ni we have
sl(cm,ni |Pi, Qi) = γ l m,n · [ −−→ hQi|Qi|; ←−− hQi1 ] (12)
where sl denotes the score generated from l-gram representation. −−→ hQik or ←−− hQik is the k-th hidden state output from question Qi’s forward and backward RNN encoder, respectively.
After that, the final score for cm,ni is evaluated as the linear combination of three scores, followed by a softmax:
s(cm,ni |Pi, Qi) = softmax(W · [s 1; s2; s3]) (13)
where sl is the shorthand notation for sl(cm,ni |Pi, Qi); W ∈ R3. In runtime, the chunk with the highest probability is taken as the answer. In training, the following negative log likelihood is minimized:
L = − N∑ i=1 logP(Ai|Pi, Qi) (14)
Note that the i-th training instance is only used when Ai is included in the corresponding candidate chunk set Ci, i.e. ∃m,nAi = cm,ni . The softmax in the final layer serves as the list-wise ranking module similar in spirit to (Cao et al., 2007).
5 EXPERIMENTS
Dataset We used the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) for the experiment. SQuAD came into our sight because it is a mix of factoid and non-factoid
questions, a real-world data (crowd-sourced), and of large scale (over 100K question-answer pairs collected from 536 Wikipedia articles). Answers range from single words to long, variable-length phrase/clauses. It is a relaxation of assumptions by the cloze-style and quiz-style RC datasets in the Problem Definition section.
Features The input vector representation of each word w to encoder RNNs has six parts including a pre-trained 300-dimensional GloVe embedding (Pennington et al., 2014) and five features (see Figure 1): (1) a one-hot encoding (46 dimensions) for the part-of-speech (POS) tag of w; (2) a one-hot encoding (14 dimensions) for named entity (NE) tag of w; (3) a binary value indicating whether w’s surface form is the same to any word in the quesiton; (4) if the lemma form of w is the same to any word in the question; and (5) if w is caplitalized. Feature (3) and (4) are designed to help the model align the passage text with question. Note that some types of questions (e.g., “who”, “when” questions) have answers that have a specific POS/NE tag pattern. For instance, “who” questions mostly have proper nouns/persons as answers and “when” questions may frequently have numbers/dates (e.g., a year) as answers. Thus, we believe that the model could exploit the co-relation between question types and answer POS/NE patterns easier with POS and NE tag features. Implementation Details We pre-processed the SQuAD dataset using Stanford CoreNLP tool5 (Manning et al., 2014) with its default setting to tokenize the text and obtain the POS and NE annotations. To train our model, we used stochastic gradient descent with the ADAM optimizer (Kingma & Ba, 2014), with an initial learning rate of 0.001. All GRU weights were initialized from a uniform distribution between (-0.01, 0.01). The hidden state size, d, was set to 300 for all GRUs. The question bi-GRU shared parameters with the passage bi-GRU, while the attention-based passage bi-GRU had its own parameters. We shuffled all training examples at the beginning of each epoch and adopted a curriculum learning approach (Bengio et al., 2009), by sorting training instances by length in every 10 batches, to enable the model start learning from relatively easier instances and to harder ones. We also applied dropout of rate 0.2 to the embedding layer of input bi-GRU encoder, and gradient clipping when the norm of gradients exceeded 10. We trained in mini-batch style (mini-batch size is 180) and applied zero-padding to the passage and question inputs in each batch. We also set the maximum passage length to be 300 tokens, and pruned all the tokens after the 300-th token in the training set to save memory and speed up the training process. This step reduced the training set size by about 1.6%. During test, we test on the full length of passage, so that we don’t prune out the potential candidates. We trained the model for at most 30 epochs, and in case the accuracy did not improve for 10 epochs, we stopped training.
For the feature ranking-based system, we used jforest ranker (Ganjisaffar et al., 2011) with LambdaMART-RegressionTree algorithm and the ranking metric was NDCG@10. For the Gated Attention Reader in baseline system, we replicated the method and use the same configurations as in (Dhingra et al., 2016).
Results
Table 2 shows our main results on the SQuAD dataset. Compared to the scores reported in (Wang & Jiang, 2016), our exact match (EM) and F1 on the development set and EM score on the test set are better, and F1 on the test set is comparable. We also studied how each component in our model contributes to the overall performance. Table 3 shows the details as well as the results of the baseline ranker. As the first row of Table 3 shows, our baseline system improves 10% (EM) over Rajpurkar et al. (Rajpurkar et al., 2016) (Table 2, row 1), the feature-based ranking system. However when compared to our DCR model (Table 3, row 2), the baseline (row 1) is more than 12% (EM) behind
5 stanfordnlp.github.io/CoreNLP/
We also did ablation tests on our DCR model. First, replacing the word-by-word attention with Attentive Reader style attention (Hermann et al., 2015) decreases the EM score by about 4.5%, showing the strength of our proposed attention mechanism.
Second, we remove the features in input to see the contribution of each feature. The result shows that POS feature (1) and question-word feature (3) are the two most important features.
Finally, combining the DCR model with the proposed POS-trie constraints yields a score similar to the one obtained using the DCR model with all possible n-gram chunks. The result shows that (1) our chunk representations are powerful enough to differentiate even a huge amount of chunks when no constraints are applied; and (2) the proposed POS-trie reduces the search space at the cost of a small drop in performance.
Analysis To better understand our system, we calculated the accuracy of the attention mechanism of the gated attention reader used in our deep learning-based baseline. We found that it is 72% accurate i.e., 72% of the times a word with the highest attention score is inside the correct answer span. This means that, if we could accurately detect the boundary around the word with the highest attention score to form the answer span, we could achieve an accuracy close to 72%. In addition, we checked the answer recall of our candidate chunking approach. When we use a window size of 10, 92% of the time, the ground truth answer will be included in the extracted Candidate chunk set. Thus the upper bound of the exact match score of our baseline system is around 66% (92% (the answer recall) × 72%). From the results, we see our DCR system’s exact match score is at 62%. This shows that DCR is proficient at differentiating answer spans dynamically.
To further analyze the system’s performance while predicting answers of different lengths, we show the exact match (EM) and F1 scores for answers with lengths up to 10 tokens in Figure 2(a). From the graph, we can see that, with the increase of answer length, both EM and F1 drops, but in different speed. The gap between F1 and exact match also widens as answer length increases. However, the model still yields a decent accuracy when the answer is longer than a single word. Additionally, Figure 2(b) shows that the system is better at “when” and “who” questions, but performs poorly
on “why” questions. The large gap between exact match and F1 on “why” questions means that perfectly identifying the span is harder than locating the core of the answer span.
Since “what”, “which”, and “how” questions contain a broad range of question types, we split them further based on the bigram a question starts with, and Figure 3 shows the breakdown for “what” questions. We can see that “what” questions asking for explanations such as “what happens” and “what happened” have lower EM and F1 scores. In contrast, “what” questions asking for year and numbers have much higher scores and, for these questions, exact match scores are close to F1 scores, which means chunking for these questions are easier for DCR.
6 RELATED WORK
Attentive Reader was the first neural model for factoid RCQA (Hermann et al., 2015). It uses Bidirectional RNN (Cho et al., 2014; Chung et al.,2014) to encode document and query respectively, and use query representation to match with every token from the document. Attention Sum Reader (Kadlec et al., 2016) simplifies the model to just predicting positions of correct answer in the document and the training speed and test accuracy are both greatly improved on the CNN/Daily Mail dataset. (Chen et al., 2016) also simplified Attentive Reader and reported higher accuracy. Windowbased Memory Networks (MemN2N) is introduced along with the CBT dataset (Hill et al., 2015), which does not use RNN encoders, but embeds contexts as memory and matches questions with embedded contexts. Those models’ mechanism is to learn the match between answer context with question/query representation. In contrast, memory enhanced neural networks like Neural Turing Machines (Graves et al., 2014) and its variants (Zhang et al., 2015; Gulcehre et al., 2016; Zaremba & Sutskever, 2015; Chandar et al., 2016; Grefenstette et al., 2015) were also potential candidates for the task, and Gulcehre et al. (Gulcehre et al., 2016) reported results on the bAbI task, which is worse than memory networks. Similarly, sequence-to-sequence models were also used (Yu et al., 2015; Hermann et al., 2015), but they did not yield better results either.
Recently, several models have been proposed to enable more complex inference for RC task. For instance, gated attention model (Dhingra et al., 2016) employs a multi-layer architecture, where each layer encodes the same document, but the attention is updated from layer to layer. EpiReader (Trischler et al., 2016b) adopted a joint training model for answer extractor and reasoner, where the extractor proposes top candidates, and the reasoner weighs each candidate by examining entailment relationship between question-answer representation and the document. An iterative alternating attention mechanism and gating strategies were proposed in (Sordoni et al., 2016) to optimize the attention through several hops. In contrast, Cui et al. (Cui et al., 2016a;b) introduced fine-grained document attention from each question word and then aggregated those attentions from each question token by summation with or without weights. This system achieved the state-of-the-art score on the CNN dataset. Those different variations all result in roughly 3-5% improvement over attention sum reader, but none of those could achieve higher than that. Other methods include using dynamic entity representation with max-pooling (Kobayashi et al., 2016) that aims to change entity representation with context, and Weissenborn’s (Weissenborn, 2016) system, which tries to separate entity from the context and then matches the question to context, scoring an accuracy around 70% on the CNN dataset.
However, all of those models assume that the answers are single tokens. This limits the type of questions the models can answer. Wang and Jiang (Wang & Jiang, 2016) proposed a match-lstm and achieved good results on SQuAD. However, this approach predicts a chunk boundary or whether a word is part of a chunk or not. In contrast, our approach explicitly constructs the chunk representations and similar chunks are compared directly to determine correct answer boundaries.
7 CONCLUSION
In this paper we proposed a novel neural reading comprehension model for question answering. Different from the previously proposed models for factoid RCQA, the proposed model, dynamic chunk reader, is not restricted to predicting a single named entity as an answer or selecting an answer from a small, pre-defined candidate list. Instead, it is capable of answering both factoid and nonfactoid questions as it learns to select answer chunks that are suitable for an input question. DCR achieves this goal with a joint deep learning model enhanced with a novel attention mechanism and five simple yet effective features. Error analysis shows that the DCR model achieves good performance, but still needs to improve on predicting longer answers, which are usually non-factoid in nature.

SUMMARY.
The paper propose a reading-comprehension question answering system for the recent QA task where answers of a question can be either single tokens or spans in the given text passage.
The model first encodes the passage and the query using a recurrent neural network.
With an attention mechanism the model calculates the importance of each word on the passage with respect to each word in the question.
The encoded words in the passage are concatenated with the attention; the resulting vector is re-encoded with a further RNN.
Three convolutional neural networks with different filter size (1,2,3-gram) are used to further capture local features.
Candidate answers are selected either matching POS patterns of answers in the training set or choosing all possible text span until a certain length.
Each candidate answer has three representations, one for each n-gram representation. The compatibility of these representation with the question representation is then calculated.
The scores are combined linearly and used for calculating the probability of the candidate answer being the right answer for the question.
The method is tested on the SQUAD dataset and outperforms the proposed baselines.
----------
OVERALL JUDGMENT
The method presented in this paper is interesting but not very motivated in some points.
For example, it is not explained why in the attention mechanism it is beneficial to concatenate the original passage encoding with the attention-weighted ones.
The contributions of the paper are moderately novel proposing mainly the attention mechanism and the convolutional re-encoding.
In fact, combining questions and passages and score their compatibility has became a fairly standard procedure in all QA models.
----------
DETAILED COMMENTS
Equation (13) i should be s, not s^l.
I still do not understand the sentence " the best function is to concatenate the hidden stat of the fist word in a chunk in forward RNN and that of the last word in backward RNN". The RNN is over what all the words in the chunk? in the passage?
The answer the authors gave in the response does not clarify this point.