Prior work has demonstrated that exploiting the sparsity can dramatically improve the energy efficiency and reduce the memory footprint of Convolutional Neural Networks (CNNs). However, these sparsity-centric optimization techniques might be less effective for Long Short-Term Memory (LSTM) based Recurrent Neural Networks (RNNs), especially for the training phase, because of the significant structural difference between the neurons. To investigate if there is possible sparsity-centric optimization for training LSTM-based RNNs, we studied several applications and observed that there is potential sparsity in the gradients generated in the backward propagation. In this paper, we investigate why the sparsity exists and propose a simple yet effective thresholding technique to induce further more sparsity during the LSTM-based RNN training. The experimental results show that the proposed technique can increase the sparsity of linear gate gradients to more than 80% without loss of performance, which makes more than 50% multiply-accumulate (MAC) operations redundant for the entire LSTM training process. These redundant MAC operations can be eliminated by hardware techniques to improve the energy efficiency and the training speed of LSTM-based RNNs.
Prior work has demonstrated that exploiting the sparsity can dramatically improve the energy efficiency and reduce the memory footprint of Convolutional Neural Networks (CNNs). However, these sparsity-centric optimization techniques might be less effective for Long Short-Term Memory (LSTM) based Recurrent Neural Networks (RNNs), especially for the training phase, because of the significant structural difference between the neurons. To investigate if there is possible sparsity-centric optimization for training LSTM-based RNNs, we studied several applications and observed that there is potential sparsity in the gradients generated in the backward propagation. In this paper, we investigate why the sparsity exists and propose a simple yet effective thresholding technique to induce further more sparsity during the LSTM-based RNN training. The experimental results show that the proposed technique can increase the sparsity of linear gate gradients to more than 80% without loss of performance, which makes more than 50% multiply-accumulate (MAC) operations redundant for the entire LSTM training process. These redundant MAC operations can be eliminated by hardware techniques to improve the energy efficiency and the training speed of LSTM-based RNNs.
1 INTRODUCTION
Deep neural networks have achieved state-of-the-art performance in many different tasks, such as computer vision (Krizhevsky et al., 2012) (Simonyan & Zisserman, 2015), speech recognition, and natural language processing (Karpathy et al., 2016). The underlying representational power of these neural networks comes from the huge parameter space, which results in an extremely large amount of computation operations and memory footprint. To reduce the memory usage and accelerate the training process, the research community has strived to eliminate the redundancy in the deep neural networks (Han et al., 2016b). Exploiting the sparsity in both weights and activations of Convolutional Neural Networks (CNNs), sparsity-centric optimization techniques (Han et al., 2016a) (Albericio et al., 2016) have been proposed to improve the speed and energy efficiency of CNN accelerators.
These sparsity-centric approaches can be classified into two categories: (1) pruning unimportant weight parameters and (2) skipping zero values in activations to eliminate multiply-accumulate (MAC) operations with zero operands. Although both categories have achieved promising results for CNNs, it remains unclear if they are applicable to training other neural networks, such as LSTMbased RNNs. The network pruning approach is not suitable for training because it only benefits the inference phase of neural networks by iteratively pruning and re-training. The approach that exploits the sparsity in the activations can be used for training because the activations are involved in both
the forward propagation and the backward propagation. But there are still some issues if we directly apply it to LSTM-based RNNs.
The sparsity in CNN activations mostly comes from the Rectified Linear Unit (ReLU) activation function, which sets all negative values to zero. However, Long Short-Term Memory, one of the most popular RNN cells, does not adopt the ReLU function. Therefore, LSTM should exhibit much less sparsity in activations than CNNs, intuitively. Furthermore, the structure of an LSTM cell is much more complicated than neurons in convolutional layers or fully connected layers of a CNN.
To explore additional opportunities to apply sparsity-centric optimization to LSTM-based RNNs, we conducted an application characterization on several LSTM-based RNN applications, including character-based language model, image captioning, and machine translation. Although the experimental results of the application characterization show that there is little sparsity in the activations, we observed potential sparsity in backward propagation of the LSTM training process. The activation values of the gates (input gate, forget gate, and output gate) and the new cell state exhibit a skewed distribution due to their functionality. That is, a large fraction of the activation values of these Sigmoid-based gates are either close to 1 or close to 0 (for the Tanh-based new cell activations, values are close to -1 or 1). This skewed distribution will lead to a considerable amount of very small values in the LSTM backward propagation since there is a term σ(x)(1− σ(x)) in the gradients of the Sigmoid-based gates (tanh(x)(1− tanh(x)) for the gradients of the new cell gradients), which will be zero given σ(x) = 0 or σ(x) = 1 (tanh(x) = −1 or tanh(x) = 1 for the new cell gradients). In real-world implementations, these very small values might be clamped to zero as they are in the form of floating-point numbers, of which the precision is limited. Therefore, there is potential sparsity in the gradients of the backward propagation of LSTM training.
To ensure that there is non-trivial amount of sparsity for hardware designers to exploit, we propose “sparsified” SGD, a rounding to zero technique to induce more sparsity in the gradients. This approach can be seen as a stochastic gradient descent (SGD) learning algorithm with sparsifying, which strips the precision of floating point numbers for unimportant small gradients. Experiment results show that with proper thresholds, we can make 80% of the gradients of the gate inputs to zero without performance loss for all applications and datasets we tested so far. As the sparse gradients of the gate inputs are involved in 67% matrix multiplications, more than 50% MAC operations are redundant in the entire LSTM training process. Eliminating these ineffectual MAC operations with hardware techniques, the energy efficiency and training speed of LSTM-based RNNs will be improved significantly.
2 BACKGROUND AND MOTIVATION
In this section, we first review some of the prior work on sparsity-centric optimization techniques for neural networks, and then illustrate the application characterization example as the motivation for our research.
2.1 SPARSITY-CENTRIC OPTIMIZATION FOR NEURAL NETWORKS
It has been demonstrated that there is significant redundancy in the parameterization of deep neural networks (Denil et al., 2013). Consequently, the over-sized parameter space results in sparsity in the weight parameters of a neural network. Besides the parameters, there is also sparsity in the activations of each layer in a network, which comes from two sources: (1) the sparsity in weight parameters and (2) the activation function of neurons, such as ReLU.
As the sparsity in weight parameters do not depend on the input data, it is often referred to as static sparsity. On the other hand, the sparsity in the activations depend on not only the weight values but also the input data. Therefore, we refer to the sparsity in the activations as dynamic sparsity.
Exploiting sparsity can dramatically reduce the network size and thus improve the computing performance and energy efficiency. For example, Deep Compression (Han et al., 2016b) applied network pruning to CNNs to significantly reduce the footprint of the weights, which enables us to store all the weights on SRAM. However, the static sparsity can only help the inference phase but not training because weight parameters are adjusted during training. Fortunately, leveraging the dynamic sparsity can benefit both inference and training of neural networks. Recent publications (Han et al.,
2016a) (Albericio et al., 2016) have proposed various approaches to eliminate ineffectual MAC operations with zero operands. Although these sparsity-centric optimization approaches have achieved promising results on CNNs, much less attention has been paid to LSTM-based RNNs, because there is a common belief that the major source of sparsity is the ReLU function, which is widely used in the convolutional layers but not in LSTM-based RNNs. To accelerate LSTM-based RNNs and improve the energy efficiency, we investigate opportunities to exploit sparsity in the LSTM-based RNN training process. As an initial step, in this paper we focus on the basic LSTM cell without peephole or other advanced features, as shown in Figure 1.
2.2 APPLICATION CHARACTERIZATION
To reveal if there is sparsity in LSTM training, we conduct an application characterization study. We start with a character-based language model as described in (Karpathy et al., 2016). This characterbased language model takes a sequence of characters as input and predicts the next character of this sequence. The characters are represented in one-hot vectors, which are transformed into distributed vectors by a word2vec layer. Then the distributed vectors feed into an RNN model based on LSTM cells, followed by a linear classifier.
The LSTM cells used in this character-based language model are all basic LSTM cells. For each cell, the forward propagation flow is as below:
it = σ(W ixt + U iht−1 + b i)
ft = σ(W fxt + U fht−1 + b f )
ot = σ(W oxt + U oht−1 + b o)
gt = tanh(W gxt + U ght−1 + b g)
ct = ft ◦ ct−1 + it ◦ gt ht = ot ◦ tanh(ct)
As shown in Figure 1, it, ft, and ot stand for input gate, forget gate, and output gate, respectively. These sigmoid-based gates (σ stands for sigmoid) are used to prevent irrelevant input from affecting the memory cell (ct). The new cell state (gt) is a preliminary summary of the current input from the previous layer and the previous status of current layer. The final hidden status ht is the output of the LSTM cell if it is seen as a black box.
Since the gates are introduced to prevent irrelevant inputs from affecting the memory cell ct, we have a hypothesis that a large fraction of the activations of these gates should be either close to 1 or close to 0, representing the control signal on or off, respectively. Similarly, the tanh-based new cell status is active if its activation is 1 or inactive if it is -1. There should also be a considerable portion of the activations close to 1 or -1.
To validate our hypothesis, we extracted the activations of the sigmoid-based gates and tanh-based new cell state from several model snapshots during training the character-based language model. Figure 2 shows the histogram of the activation values of the gates and the new cell. The red curves represent the activation values generated by a snapshot model which is 0.5% trained (in terms of total number of iterations) while the bars represent the activation values generated by a fully trained
model. We can observe skewed distributions from each gate (and new cell) for both the 0.5% trained snapshot model and the fully trained model. Furthermore, the fully trained model shows a distribution that is more skewed to the leftmost and the rightmost. Additionally, other un-shown snapshots demonstrate that the distribution becomes consistently more skewed as the training process goes on. We also observed that after 10% of the training process, the distribution becomes steady, almost the same as the fully trained model.
Besides the character-based language model, we also conducted the same characterization to the image captioning task described in (Karpathy & Li, 2015). The activation values of the RNN layer in the image captioning task exhibit the skewed distribution too. Even though we did not observe sparsity in the gate activations, the skewed distribution indicates potential sparsity in the LSTMbased RNN backward propagation, which will be shown in the next section.
3 SPARSIFIED STOCHASTIC GRADIENT DESCENT FOR LSTM
In this section, we first show how the skewed distribution of gate values leads to potential sparsity in the LSTM backward propagation, and then we propose the “sparsified” SGD to induce more sparsity in LSTM training.
3.1 POTENTIAL SPARSITY IN LSTM BACKWARD PROPAGATION
To show how the skewed distribution in the gate activations results in potential sparsity in the LSTMbased RNN backward propagation, we need to review the forward and backward propagation at first. We can re-write the forward propagation equations as
net(i)t =W ixt + U iht−1 + b i
net(f)t =W fxt + U fht−1 + b f
net(o)t =W oxt + U oht−1 + b o
net(g)t =W gxt + U ght−1 + b g
it = σ(net(i)t)
ft = σ(net(f)t)
ot = σ(net(o)t)
gt = tanh(net(g)t)
ct = ft ◦ ct−1 + it ◦ gt ht = ot ◦ tanh(ct)
Here we introduce variables net(i), net(f), net(o) and net(g) to represent the linear part of the gates and the new cell state. In GPU implementations such as cuDNN v5 (Appleyard et al., 2016), these linear gates (including new cell state from now on) are usually calculated in one step since they share the same input vectors xt and ht−1. Therefore we can use a uniform representation for the four linear gates, that is
nett =Wxt + Uht−1 + b
The matrix W here stands for the combination of the matrices W i, W f , W o and W g and the matrix U stands for the combination of the matrices U i, Uf , Uo and Ug .
With these denotations, we can express the backward propagation as
dot = dht ◦ tanh(ct)
dct = dht ◦ (1− tanh2(ct)) ◦ ot + ft ◦ ct+1 dnet(g)t = dct ◦ it ◦ (1− g2t )
dnet(o)t = dht ◦ tanh(ct) ◦ (1− ot) ◦ ot dnet(f)t = dct ◦ ct−1 ◦ (1− ft) ◦ ft dnet(i)t = dct ◦ gt ◦ (1− it) ◦ it
dxt = dnettW T
dht−1 = dnettU T
dW+ = xtdnett
dU+ = ht−1dnett
In the equations of the backward propagation, we use dnet to denote the gradient of the linear gates.
From these equations we can see that for each linear gate gradient there is one term introduced by the sigmoid function or the tanh function, e.g. (1−g2t ) in dnet(g)t and (1−ot)◦ot in dnet(o)t. As we observed in the application characterization results, the activation values of these gates exhibit skewed distribution, which means a large fraction of ot, ft and it are close to 0 or 1 (gt close to -1 or 1). The skewed distribution makes a large fraction of the linear gate gradients close to zero because (1 − g2t ), (1 − ot) ◦ ot, (1 − ft) ◦ ft and (1 − it) ◦ it are mostly close to zero given the skewed distribution of the gate activations.
When implementing the LSTM-based RNNs, we usually use 32-bit floating point numbers to represent the gradients. Due to the precision limit, floating point numbers will round extremely small values to zero. Therefore, there is potential sparsity in dnet since a large fraction of the linear gate gradients are close to zero.
3.2 INDUCING MORE SPARSITY
In the previous section we showed how the skewed distribution in gate activations results in potential sparsity in linear gate gradients theoretically. However, from mathematical perspective, there will be no sparsity in linear gate gradients if the floating point numbers in computers have infinite precision since they are only close to zero rather than be zero. Even the precision of 32-bit floating point numbers is not infinite, the 8-bit exponential part can still accommodate an extremely large dynamic range, which makes the sparsity less interesting to hardware accelerator designers. Fortunately, recent attempts to train neural networks with 16-bit floating points (Gupta et al., 2015) and fixed points (Lin et al., 2015) have shown acceptable performance with smaller dynamic range. This inspires us to induce more sparsity by rounding very small linear gate gradients to zero, which is similar to replace 32-bit floating points with 16-bit floating points or fixed points.
The intuition behind this “rounding to zero” approach is that pruning CNNs will not affect the overall training performance. Similarly, thresholding very small gradient (dnet) values to zero is likely not to affect the overall training accuracy. Therefore, we propose a simple static thresholding approach which sets small dnet values below a threshold t to zero. By doing this, we can increase the sparsity in dnet even further than the original sparsity caused by limited dynamic range of floating
point numbers. With our static thresholding technique, the backward propagation of LSTM training becomes as below:
dot = dht ◦ tanh(ct) dct = dht ◦ (1− tanh2(ct)) ◦ ot + ft ◦ ct+1
dnet(g)t = dct ◦ it ◦ (1− g2t ) dnet(o)t = dht ◦ tanh(ct) ◦ (1− ot) ◦ ot dnet(f)t = dct ◦ ct−1 ◦ (1− ft) ◦ ft dnet(i)t = dct ◦ gt ◦ (1− it) ◦ it dnett = (dnett > t)?dnett : 0
dxt = dnettW T
dht−1 = dnettU T
dW+ = xtdnett
dU+ = ht−1dnett
In this “sparsified” SGD backward propagation, a new hyper-parameter t is introduced to control the sparsity we would like to induce in dnet. Clearly, the optimal threshold t is the highest one that has no impact on the training performance since it can induce the highest sparsity in dnet. Therefore, to select the threshold, we need to monitor the impact on the gradients. As the SGD only uses the gradients of the weights (dW ) to update the weights, dW is the only gradients we need to care about. From the equations of the backward propagation we can see that dW is computed based on dnet, which is sparsified by our approach. Although sparsifying dnet affects dW , we can control the change of dW by setting the threshold. To determine the largest acceptable threshold, we conducted an evaluation of the impact caused by different thresholds on one single step in LSTM training. The application here is the same as the one in the application characterization.
Figure 3 shows the evaluation result. We measure the change of dW by the normalized inner product of sparsified dW and the original dW without sparisifying (the baseline shown in Figure 3). If we denote the original weight gradient as dW0, the correlation between sparsified dW and dW0 can be measured by normalized inner product
correlation = dW · dW0
||dW || · ||dW0||
If the correlation is 1, it means dW is exactly the same to dW0. If the correlation is 0, it means dW is orthogonal to dW0. The higher the correlation is, the less impact the sparsification has on this single step backward propagation. From Figure 3 we can see that even without our thresholding technique, the dnet still exhibits approximately 10% sparsity. These zero values are resulted from the limited dynamic range of floating point numbers, in which extremely small values are rounded to zero. By applying the thresholds to dnet, we can induce more sparsity shown by the bars. Even with a low threshold (10−8), the sparsity in dnet is increased to about 45%. With a relatively high threshold (10−6), the sparsity can be increased to around 80%. Although the sparsity is high, the correlation between the sparsified dW and dW0 is close to 1 even with the high threshold. Therefore, we can hypothesize that we can safely induce a considerable amount of sparsity with an appropriate threshold. It is straightforward to understand that the threshold cannot be arbitrarily large since we need to contain the information of the gradients. For example, if we increase the threshold even further to 10−5, the correlation will drop to 0.26, which is far from the original dW0 and not acceptable.
We have demonstrated that we can induce more sparsity by rounding small dnet to zero while maintaining the information in dW . However, this is only an evaluation on one single iteration of training. To show the generality of our static thresholding approach, we applied the thresholds to the entire training process.
4 AN ENTIRE LSTM TRAINING WITH SPARSIFIED SGD
In this section, we first present the sparsity induced by applying our sprsified SGD to an entire training process, and then discuss the generality of our approach.
4.1 CHARACTER-BASED LANGUAGE MODEL
To validate our proposed static thresholding approach, we apply it to the entire LSTM-based RNN training process. We first conducted an experiment on training a character-based language model. The language model consists of one word2vec layer, three LSTM-based RNN layers, and one linear classifier layer. The number of LSTM cells per RNN layer is 256. We feed the network with sequences of 100 characters each. The training dataset is a truncated Wikipedia dataset. We apply a fixed threshold to all dnet gradients for every iteration during the whole training process.
Figure 4 shows the sparsity of the linear gate gradients (dnet) of each layer during the whole training process. In the baseline configuration, the training method is standard SGD without sparsifying (zero
threshold). The baseline configuration exhibits about 20% sparsity in dnet. By applying only a low threshold (10−7), the sparsity is increased to around 70%. And we can consistently increase the sparsity further by raising the threshold. However, we have to monitor the impact of the threshold on the overall training performance to check if the threshold is too large to use.
Figure 5 shows the validation loss of each iteration. We observe that up to the medium threshold (10−6), the validation loss of the model trained with sparsified SGD keeps close to the baseline. However, if we continues raising the threshold to 10−5, the validation loss becomes unacceptably higher than the baseline. Although the validation loss with the 10−5 threshold is consistently decreasing as the training goes on, we conservatively do not pick this configuration to train the LSTM network. So combining Figure 4 with Figure 5, we can choose the threshold 10−6 to train the character-based language model to achieve about 80% sparsity in dnet.
Since the linear gate gradients dnet are involved in all the four matrix multiplications in the backward propagation, there are 80% MAC operations in these matrix multiplications have zero operands. Furthermore, there are six matrix multiplications (all of them are of the same amount of computation) in one LSTM training iteration and four out of them (67%) are sparse. So there are more than 50% MAC operations will have zero operands introduced by our sparsified SGD in one LSTM training iteration. The MAC operations with zero operands produce zero output and thus make no contribution to the final results. These redundant MAC operations can be eliminated by hardware techniques similar to (Han et al., 2016a) (Albericio et al., 2016) to improve the energy efficiency of LSTM training.
4.2 SENSITIVITY TEST
Our static thresholding approach can induce more than 80% sparsity in linear gate gradients of the character-based language model training. To demonstrate the generality of our approach, we then changed the topology of the RNN layers in the character-based language model with several different LSTM-based RNNs for a sensitivity test. The network topologies used in the sensitivity test are shown below.
• Number of layers: 2, 3, 6, 9; • Number of LSTM cells per layer: 128, 256, 512; • Sequence length: 25, 50, 100.
We also trained the network with other datasets, such as the tiny-Shakespear dataset and the novel War and Peace. For all the data points we collected from the sensitivity test, we can always achieve
more than 80% sparsity in dnet with less than 1% loss of performance in terms of validation loss with respect to the baseline.
Moreover, we also validated our approach by training an image captioning application (Karpathy & Li, 2015) with MSCOCO dataset (Lin et al., 2014) and a machine translation application known as Seq2Seq (Sutskever et al., 2014) with WMT15 dataset. As both the two applications are implemented based on graph model (Torch and TensorFlow, respectively), we plugged a custom operation in the automatically generated backward propagation subgraph to implement our proposed sparsified SGD. The experiment results show that the conclusion for the character-based language model still holds for the two applications.
4.3 DISCUSSION
So far all our experiment results show promising results and we believe our sparsified SGD is a general approach to induce sparsity for LSTM-based RNN training. From the computer hardware perspective, the sparsified SGD is similar to reduced precision implementation while the impact of sparsified SGD is much less since we still use full 32-bit floating point numbers. From the theory perspective, SGD itself is a gradient descent with noise and thresholding very small gradients to zero is nothing more than an additional noise source. Since training with SGD is robust to noise, the thresholding approach will likely not affect the overall training performance. Additionally, the weight gradients dW are aggregated through many time steps, which makes the LSTM more robust to the noise introduce by sparsifying the linear gate gradients.
5 CONCLUSION AND FUTURE WORK
In this paper, we conducted an application characterization to an LSTM-based RNN application and observe skewed distribution in the sigmoid-based gates and the tanh-based new cell state, which indicates potential sparsity in the linear gate gradients during backward propagation with SGD. The linear gate gradients are involved with 67% MAC operations in an entire LSTM training process so that we can improve the energy efficiency of hardware implementations if the linear gate gradients are sparse. We propose a simple yet effective rounding to zero technique, which can make the sparsity of the linear gate gradients higher than 80% without loss of performance. Therefore, more than 50% MAC operations are redundant in an entire sparsified LSTM training.
Obviously, the static-threshold approach is not optimal. In future, we will design a dynamicthreshold approach based on the learning rate, L2-norm of the gradients and the network topology. Hardware techniques will also be introduced to exploit the sparsity to improve the energy efficiency and training speed of LSTM-based RNNs for GPU and other hardware accelerators.

This paper presents the observation that it is possible to utilize sparse operations in the training of LSTM networks without loss of accuracy. This observation is novel (although not too surprising) to my knowledge, but I must state that I am not very familiar with research on fast RNN implmentations.
Minor note:
The LSTM language model does not use a 'word2vec' layer. It is simply a linear embedding layer. Word2vec is the name of a specific model which is not directly to character level language models.
The paper presents the central observation clearly. However, it will be much more convincing if a well known dataset and experiment set up are used, such as Graves (2013) or Sutskever et al (2014), and actual training, validation and test performances are reported.
While the main observation is certainly interesting, I think it is not sufficient to be the subject of a full conference paper without implementation (or simulation) and benchmarking of the promised speedups on multiple tasks. For example, how would the gains be affected by various architecture choices?
At present this is an interesting technical report and I would like to see more detailed results in the future.


Deep learning has proven useful on many NLP tasks including reading comprehension. However, it requires large amounts of training data which are not available in some domains of application. Hence we examine the possibility of using data-rich domains to pre-train models and then apply them in domains where training data are harder to get. Specifically, we train a neural-network-based model on two context-question-answer datasets, the BookTest and CNN/Daily Mail, and we monitor transfer to subsets of bAbI, a set of artificial tasks designed to test specific reasoning abilities, and of SQuAD, a question-answering dataset which is much closer to real-world applications. Our experiments show very limited transfer if the model is not shown any training examples from the target domain however the results are encouraging if the model is shown at least a few target-domain examples. Furthermore we show that the effect of pre-training is not limited to word embeddings.
1 INTRODUCTION
Machine intelligence has had some notable successes, however often in narrow domains which are sometimes of little practical use to humans – for instance games like chess (Campbell et al., 2002) or Go (Silver et al., 2016). If we aimed to build a general AI that would be able to efficiently assist humans in a wide range of settings, we would want it to have a much larger set of skills – among them would be an ability to understand human language, to perform common-sense reasoning and to be able to generalize its abilities to new situations like humans do.
If we want to achieve this goal through Machine Learning, we need data to learn from. A lot of data if the task at hand is complex – which is the case for many useful tasks. One way to achieve wide applicability would be to provide training data for each specific task we would like the machine to perform. However it is unrealistic to obtain a sufficient amount of training data for some domains – it may for instance require expensive human annotation or all domains of application may be difficult to predict in advance – while the amount of training data in other domains is practically unlimited, (e.g. in language modelling or Cloze-style question answering).
The way to bridge this gap – and to achieve the aforementioned adaptability – is transfer learning (Pan & Yang, 2010) and closely related semi-supervised learning (Zhu & Goldberg, 2009) which allow the system to acquire a set of skills on domains where data are abundant and then use these skills to succeed on previously unseen domains. Despite how important generalization is for general AI, a lot of research keeps focusing on solving narrow tasks.
In this paper we would like to examine transfer of learnt skills and knowledge within the domain of text comprehension, a field that has lately attracted a lot of attention within the NLP community (Hermann et al., 2015; Hill et al., 2015; Kobayashi et al., 2016; Kadlec et al., 2016b; Chen et al., 2016; Sordoni et al., 2016; Dhingra et al., 2016; Trischler et al., 2016; Weissenborn, 2016; Cui et al., 2016b;a;
∗These authors contributed equally to this work.
Li et al., 2016; Shen et al., 2016). Specifically, we would like to address the following research questions:
1. Whether we could train models on natural-language tasks where data are abundant and transfer the learnt skills to tasks where in-domain training data may be difficult to obtain. We will first look into what reasoning abilities a model learns from two large-scale readingcomprehension datasets using artificial tasks, and then check whether it can transfer its skills to real world tasks. Spoiler: both these transfers are very poor if we allow no training at all on the target task.
2. Whether pre-training on large-scale datasets does help if we allow the model to train on a small sample of examples from the target tasks. Here the results are much more positive.
3. Finally we examine whether the benefits of pre-training are concentrated in any particular part of the model - namely the word-embedding part or the context encoder (the reasoning part). It turns out that pre-training is useful for both components.
Although our results do not improve current state of the art in any of the studied tasks, they show a clear positive effect of large-dataset pre-training on the performance of our baseline machine-learning model. Previous studies of transfer learning and semi-supervised learning in NLP focused on text classification (Dai & Le, 2015; Mou et al., 2016) and various parsing tasks (Collobert et al., 2011; Hashimoto et al., 2016). To our knowledge this work is the first study of transfer learning in reading comprehension, and we hope it will stimulate further work in this important area.
We will first briefly introduce the datasets we will be using on the pre-training and target sides, then our baseline model and afterwards in turn describe the method and results of each of the three experiments.
2 DATASETS
2.1 PRE-TRAINING DATASETS
We have mentioned that for the model pre-training we would want to use a task where training data are abundant. An example of such task is context-dependent cloze-style-question answering since the training data for this task can be generated automatically from a suitable corpus. We will use two such pre-training datasets in our experiments: the BookTest (Bajgar et al., 2016) and the CNN/Daily Mail (CNN/DM) news dataset (Hermann et al., 2015).
The task associated with both datasets is to answer a cloze-style question (i.e. fill in a blank in a sentence) the answer to which needs to be inferred from a context document provided with the question.
2.1.1 BOOKTEST
In the BookTest dataset, the context document is formed from 20 consecutive sentences from a book. The question is then formed by omitting a common noun or a named entity from the subsequent 21st sentence. Among datasets of this kind, the BookTest is among the largest with more than 14 million training examples coming from 3555 copyright-free books avalable thanks to Project Gutenberg.
2.1.2 CNN/DAILY MAIL
In the CNN/DM dataset the context document is formed from a news article while the cloze-style question is formed by removing a named entity from one of the short summary sentences which often appear at the top of the article.
To stop the model from using world knowledge from outside the context article (and hence truly test the comprehension of the article), all named entities were replaced by anonymous tags, which are further shuffled for each example. This may make the comprehension more difficult; however, since the answer is always one of the anonymized entities, it also reduces the number of possible answers making guessing easier.
2.2 TARGET DATASETS
2.2.1 BABI
The first target dataset are the bAbI tasks (Weston et al., 2016) – a set of artificial tasks each of which is designed to test a specific kind of reasoning. This toy dataset will allow us to observe what particular skills the model may be learning from each of the three training datasets.
For our experiments we will be using an architecture designed to select one word from the context document as the answer. Hence we have selected Tasks 1,2,3,4,5,11,12,13,14 and 16 which fulfill this requirement and added task 15 which required a slight modification. Furthermore because both pre-training datasets are cloze-style we converted also the bAbI task questions into cloze style (e.g. ”Where is John?” to ”John is in the XXXXX.”).
For the models pre-trained on CNN/DM we also anonymized the tasks in a way similar to the pre-training dataset - i.e. we replaced all names of characters and also all words that can appear as answers for the given task by anonymous tags in the style of CNN/DM. This gives even models that have not seen any training examples from the target domain a chance to answer the questions.
Full details about these alterations can be found in Appendix A.
2.2.2 SQUAD
Secondly, we will look on transfer to the SQuAD dataset (Rajpurkar et al., 2016); here the associated task may be already useful in the real world. Although cloze-style questions have the huge advantage in the possibility of being automatically generated from a suitable corpus – the path taken by CNN/DM and the BookTest – in practice humans would use a proper question, not its cloze-style substitute. This brings us to the need of transfer from the data-rich cloze-style training to the domain of proper questions where data are much scarcer due to the necessary human annotation.
The SQuAD dataset is a great target dataset to use for this. As opposed to the bAbI tasks, the goal of this dataset is actually a problem whose solving would be useful to humans - answering natural questions based on an natural language encyclopedic knowledge base.
For our experiments we selected only a subset of the SQuAD training and development examples where the answer is only a single word, since this is an inherent assumption of our machine learning model. This way we extracted 28,346 training examples out of the original 100,000 examples and 3,233 development examples out of 10,570.
3 MACHINE LEARNING MODEL: AS READER
We perform our experiments using the Attention Sum Reader (AS Reader) (Kadlec et al., 2016b) model. The AS Reader is simple to implement while it achieves strong performance on several text comprehension tasks (Kadlec et al., 2016b; Bajgar et al., 2016; Chu et al., 2016). Since the AS Reader is a building block of many recent text-comprehension models (Trischler et al., 2016; Sordoni et al., 2016; Dhingra et al., 2016; Cui et al., 2016b;a; Shen et al., 2016; Munkhdalai & Yu, 2016) it is a good representative of current research in this field.
A high level structure of the AS Reader is shown in Figure 1. The words from the document and the question are first converted into vector embeddings using a look-up matrix. The document is then read by a bidirectional Gated Recurrent Unit (GRU) network (Cho et al., 2014). A concatenation of the hidden states of the forward and backward GRUs at each word is then used as a contextual embedding of this word, intuitively representing the context in which the word is appearing. We can also understand it as representing the set of questions to which this word may be an answer.
Similarly the question is read by a bidirectional GRU but in this case only the final hidden states are concatenated to form the question embedding.
The attention over each word in the context is then calculated as the dot product of its contextual embedding with the question embedding. This attention is then normalized by the softmax function and summed across all occurrences of each answer candidate. The candidate with most accumulated attention is selected as the final answer.
For a more detailed description of the model including equations check Kadlec et al. (2016b).
4 EXPERIMENTS: TRANSFER LEARNING IN TEXT COMPREHENSION
Now let us turn in more detail to the three kinds of experiments that we performed.
4.1 PRE-TRAINED WITHOUT TARGET ADJUSTMENT
In the first experiment we tested how a model trained on one of the large-scale pre-training datasets performs on the bAbI tasks without any opportunity to train on bAbI. Since the BookTest and CNN/DM tasks involve only cloze-style questions, we can’t expect a model trained on them to answer natural ?-style questions. Hence we did not study the transfer to SQuAD in this case, only the transfer to the (cloze-converted) bAbI tasks.
4.1.1 METHOD
First we tested how the AS Reader architecture (Kadlec et al., 2016b) can handle the tasks if trained directly on the bAbI training data for each task. Then we tested the degree of transfer from the BookTest and CNN/DM data to the 11 selected bAbI tasks.
In the first part of the experiment we trained a separate instance of the AS Reader on the 10,000- example version of the bAbI training data for each of the 11 tasks (for more details see Appendix B.1). On 8 of them the architecture was able to learn the task with accuracy at least 95% 1 (results for each task can be found in Table 4 in Appendix C). Hence if given appropriate training the AS Reader is capable of the reasoning needed to solve most of the selected bAbI tasks. Now when we know that the AS Reader is powerful enough to learn the target tasks we can turn to transfer from the two large-scale datasets.
The main part of this first experiment was then straightforward: we pre-trained multiple models on the BookTest and CNN/DM datasets and then simply evaluated them on the test datasets of the 11 selected bAbI tasks.
4.1.2 RESULTS
Table 1 summarizes the results of this experiment. Both the models trained on the BookTest and those trained on the CNN/DM dataset perform quite poorly on bAbI and achieve much lower accuracy than
1It should be noted that there are several machine learning models that perform better than the AS Reader in the 10k weakly supervised setting, e.g. (Sukhbaatar et al., 2015; Xiong et al., 2016; Graves et al., 2016), however they often need significant fine-tuning. On the other hand we trained plain AS Reader model without any modifications. Hyperparameter and feature fine-tuning could probably further increase its performance on individual tasks however it goes directly against the idea of generality that is at the heart of this work. For comparison with state of the art we include results of DMN+ (Xiong et al., 2016) in Table 1 which had the best average performance over the original 20 tasks.
the models trained directly on each individual bAbI task. However there is some transfer between the tasks since the AS Reader trained on either the BookTest or CNN/DM outperforms a random baseline2 and even an improved baseline which selects the most frequent word from the context that also appears as an answer in the training data for this task.
The results also show that the models trained on CNN/DM perform somewhat better on most tasks than the BookTest models. This may be due to the fact that bAbI tasks generally require the model to summarize information from the context document, which is also what the CNN/DM dataset is testing. On the other hand, the BookTest requires prediction of a possible continuation of a story, where the required kind of reasoning is much less clear but certainly different from pure summarization. Another explanation for better performance of CNN/DM models might be that they solve slightly simpler task since the candidate answers were already pre-selected in the entity anonymization step.
Readers interested in how the training-dataset size affects this kind of transfer can check (Kadlec et al., 2016a) where we show that the target-task performance is a bit better if we use the large BookTest as opposed to its smaller subset, the Children’s Book Test (CBT) (Hill et al., 2015).
Conclusions from this experiment are that the skills learned from two large-scale datasets generalize surprisingly poorly to even simple toy tasks. This may make us ask whether most teams’ focus on solving narrow tasks is truly beneficial if the skills learnt on these tasks are hard to apply elsewhere. However it also brings us to our next experiment, where we try to provide some help to the struggling pre-trained models.
4.2 PRE-TRAINED WITH TARGET ADJUSTMENT
After showing that the skills learnt from the BookTest and CNN/DM datasets are by themselves insufficient for solving the toy tasks, the next natural question is whether they are useful if helped by training on a small sample of examples from the target task. We call this additional phase of training target adjustment. For this experiment we again use the bAbI tasks, however we also test transfer to a subset of the SQuAD dataset, which is much closer to real-world natural-language question answering.
The results presented in this and the following section are based on training 3701 model instances.
4.2.1 METHOD
Common to bAbI and SQuAD datasets. In this experiment we started with a pre-trained model which we used in the previous experiment. However, after it finished training on one of the large pre-training datasets, we allowed it to train on a subset of training examples from the target dataset. We tried subsets of various sizes ranging from a single example to thousands. We tried training four different pre-trained models and also, for comparison, four randomly-initialized models with the same hyperparameters (see Appendix B.2 for details). The experiment with each task-model couple was run on 4 different data samples of each size which were randomly drawn from the training dataset
2The random baseline selects randomly uniformly between all unique words contained in the context document.
of the task to account for variations between these random samples – which may be substantial given the small sample size.3
bAbI. For each of these models we observed the test accuracy at the best-validation epoch and compared this number between the randomly initialized and pre-trained models. Validation was done using 100 examples which were set aside from the task’s original 10k training data.4 We perform the experiment with models pre-trained on the BookTest and also on CNN/DM.
SQuAD subset. In the SQuAD experiment, we trained the model on a subset of the original training dataset where answers were only single words and its sub-subsets. We report the best-validation accuracy on a development set filtered in the same way. This experiment was performed only with the models pre-trained on BookTest.
4.2.2 RESULTS
The results of these experiments are summarized in Figures 2 and 3.
3We are planning to release the split training datasets soon. 4The other models trained on the full 10k dataset usually use 1000 validation examples (Sukhbaatar et al., 2015; Xiong et al., 2016), however we wanted to focus on low data regime thus we used 10 times less examples.
bAbI. Sub-figure 2a shows mean test accuracy of the models that achieved the best validation result for each single task. The results for both BookTest and CNN/DM experiments confirm positive effect of pre-training compared to randomly initialized baseline. Figure 3 shows performance on selected bAbI tasks where pre-training has clearly positive effect, such plot for each of the target tasks is provided in Appendix C.2 (Figure 4).
Note that the CNN/DM models cannot be directly compared to BookTest results due to entity anonymization that seems to simplify the task when the model is trained on smaller datasets.
Since our evaluation methodology with different training set sizes is novel, we can compare our result only to MemN2N (Sukhbaatar et al., 2015) trained on a 1k dataset. MemN2N is the only weakly supervised model that reports accuracy when trained on less than 10k examples. MemN2N achieves average accuracy 93.2%5 on the eleven selected tasks. This is substantially better than both our random baseline (78.0%) and the BookTest-pre-trained model (79.5%), however our model is not tuned in any way towards this particular task. One important conceptual difference is that the AS Reader processes the whole context as one sequence of words, whereas MemN2N receives the context split into single sentences, which simplifies the task for the network.
SQuAD subset. The results of SQuAD experiment also confirm positive effect of pre-training, see Sub-figure 2b, for now compare just lines showing performance of the fully pre-trained model and the randomly initialized model – the meaning of the remaining two lines shall become clear in the next section.
More detailed statistics about the results of this experiment can be found in Appendix D.
We should note that performance of our model is not competitive with the state of the art models on this dataset. For instance the DCR model (Yu et al., 2016) trained on our SQuAD subset achieves validation accuracy 74.9% in this task which is better than our randomly initialized (35.4%) and pre-trained (51.6%) models6. However, the DCR model is designed specifically for the SQuAD task, for instance it utilizes features that are not used by our model.
4.3 PARTIALLY PRE-TRAINED MODEL
Since our previous experiment confirmed positive effect of pre-training if followed by target-domain adjustment, we wondered which part of the model contains the knowledge transferable to new domains. To examine this we performed the following experiment.
4.3.1 METHOD
Our machine learning model, the AS Reader, consists of two main parts: the word-embedding look-up and the bidirectional GRUs used to encode the document and question (see Figure 1). Therefore a natural question was what the contribution of each of these parts is.
To test this we created two models out of each pre-trained model used in the previous experiment. The first model variant uses the pre-trained word embeddings from the original model while the GRU encoders are randomly initialized. We say that this model has pre-trained embeddings. The second model variant uses the opposite setting where the word embeddings are randomly initialized while the encoders are taken form a pre-trained model. We call this pre-trained encoders.
bAbI. For this experiment we selected only a subset of tasks with training set of 100 examples where there was significant difference in accuracy between randomly-initialized and pre-trained models. For evaluation we use the same methodology as in the previous experiment, that is, we report accuracy of the best-validation model averaged over 4 training splits.
SQuAD subset. We evaluated both model variants on all training sets from the previous SQuAD experiment using the same methodology.
5MemN2N trained on each single task with PE LS RN features, see (Sukhbaatar et al., 2015) for details. 6We would like to thank Yu et al. (2016) for training their system on our dataset.
4.3.2 RESULTS
bAbI. Table 2 shows improvement of pre-trained models over a randomly initialized baseline. In most cases (all except Task 5) the fully pre-trained model achieved the best accuracy.
SQuAD subset. The accuracies of the four model variants are plotted in Figure 2b together with results of the previous SQuAD experiment. The graph shows that both pre-trained embeddings and pre-trained encoders alone improve performance over the randomly initialized baseline, however the fully pre-trained model is always the best.
The overall result of this experiment is that both pre-training of the word embeddings and pre-training of the encoder parameters are important since the fully pre-trained model outperforms both partially pre-trained variants.
5 CONCLUSION
Our experiments show that transfer from two large cloze-style question-answering datasets to our two target tasks is suprisingly poor, if the models aren’t provided with any examples from the target domain. However we show that models that pre-trained models perform significantly better than a randomly initialized model if they are shown at least a few training examples from the target domain. The usefulness of pre-trained word embeddings is well known in the NLP community however we show that the power of our pre-trained model does not lie just in the embeddings. This suggests that once the text-comprehension community agrees on sufficiently versatile model, much larger parts of the model could start being reused than just the word-embeddings.
The generalization of skills from a training domain to new tasks is an important ingredient of any system we would want to call intelligent. This work is an early step to explore this direction.
A CLOZE STYLE BABI DATASET
Since our AS Reader architecture is designed to select a single word from the context document as an answer (the task of CBT and BookTest), we selected 10 bAbI tasks that fulfill this requirement out of the original 20. These tasks are: 1. single supporting fact, 2. two supporting facts, 3. three supporting facts, 4. two argument relations, 5. three argument relations, 11. basic coreference, 12. conjunction, 13. compound coreference, 14. time reasoning and 16. basic induction.
Task 15 needed a slight modification to satisfy this requirement: we converted the answers into plural (e.g. ”Q: What is Gertrude afraid of? A: wolf.” was converted into ”A: wolves” which also seems to be the more natural way to formulate the answer to such a question.).
Also since CBT and BookTest train the model for Cloze-style question answering, we modify the original bAbI dataset by reformulating the questions into Cloze-style. For example we translate a question ”Where is John ?” to ”John is in the XXXXX .”
For the models pre-trained on CNN/DM we also replace two kinds of words by anonymized tags (e.g. ”@entity56”) in a style similar to the pre-training dataset. Specifically we replace two (largely overlapping) categories of words:
1. Proper names of story characters (e.g. John, Sandra) 2. Any word that can appear as an answer for the particular task (e.g. kitchen, garden if the
task is asking about locations).
B METHOD DETAILS
B.1 DIRECT TRAINING ON BABI – METHOD
Here we give a more detailed description of the method we used to arrive to our results. We highlight only facts particular to this experiment. A more detailed general description of training the AS Reader is given in (Kadlec et al., 2016b).
The results given for AS Reader trained on bAbI are each for a single model with 64 hidden units in each direction of the GRU context encoder and embedding dimension 32 trained on the 10k training data provided with that particular task.
The results for AS Reader trained on the BookTest and the CNN/DM are for a greedy ensemble consisting of 4 models whose predictions were simply averaged. The models and ensemble were all validated on the validation set corresponding to the training dataset. The performance on the bAbI tasks oscillated notably during training however the ensemble averaging does somewhat mitigate this to get more representative numbers.
B.2 HYPERPARAMETERS FOR THE TARGET-ADJUSTMENT EXPERIMENTS
Table 3 lists hyperparameters of the pre-trained AS Reader instances used in our experiments with target adjustment.
C DETAILED RESULTS
C.1 EXPERIMENTS WITHOUT TARGET ADJUSTMENT
Table 4 shows detailed results for the experiments on models which were just pre-trained on one of the pre-training datasets without any target-adjustment. It also shows several baselines and results of a state-of-the-art model.
C.2 TARGET-ADJUSTMENT EXPERIMENTS
C.2.1 RESULTS FOR ALL BABI TASKS
Figure 4 shows the test accuracies of all models that we trained in the target-adjustment experiments as well as lines joining the accuracies of the best-validation models.
Ta bl
e 4:
Pe rf
or m
an ce
of th
e A
S R
ea de
r w
he n
tr ai
ne d
on th
e bA
bI 10
k, B
oo kT
es ta
nd C
N N
/D M
da ta
se ts
an d
th en
ev al
ua te
d on
bA bI
te st
da ta
. T
he D
yn am ic M em or y N et w or k (D M N +) is th e st at eof -t he -a rt m od el in a w ea kl y su pe rv is ed se tti ng on th e bA bI 10 k da ta se t. It s re su lts ar e ta ke n fr om (X io ng et al ., 20 16 ). M em N 2N (S uk hb aa ta re ta l., 20 15 )i s th e st at eof -t he -a rt m od el on th e 1k tr ai ni ng da ta se t; fo rc om pl et en es s w e al so in cl ud e its re su lts w ith th e 10 k tr ai ni ng .
M od
el :
R an
do m
R nd
ca nd
. M
em N 2N (s in gl e) (P E L S R
N )
M em
N 2N
(s in
gl e)
(P E
L S
LW R
N )
D M
N +
(s in
gl e)
A SR
ea de
r
a a
a a
a a a
a a a
Te st
da ta
se t Tr ai
n da
ta se
t no
t tr ai ne
d bA
bI 10
k bA
bI 1k
bA bI 10 k
bA bI 10 k
bA bI 10 k
B oo
kT es t 14
M D
M +C
N N
1. 2M
1 Si
ng le
su pp
or tin
g fa
ct 7.
80 31
.2 0
10 0.
00 10
0. 00
10 0.
00 10
0. 00
37 .3
0 51
.5 0
2 Tw
o su
pp or
tin g
fa ct
s 4.
40 26
.9 6
91 .7
0 99
.7 0
99 .7
0 91
.9 0
25 .8
0 28
.9 0
3 T
hr ee
su pp
or tin
g fa
ct s
3. 40
19 .1
4 59
.7 0
97 .9
0 98
.9 0
86 .0
0 22
.2 0
27 .4 0 4 Tw oar gu m en tr el at io ns 10 .5 0 33 .5 8 97 .2 0 10 0. 00 10 0. 00 10 0. 00 50 .3 0 54 .9 0 5 T hr ee -a rg um en tr el at io ns 4. 40 21 .4 2 86 .9 0 99 .2 0 99 .5 0 99 .8 0 67 .6 0 68 .1 0 11 B as ic co re fe re nc e 6. 20 30 .4 2 99 .1 0 99 .9 0 10 0. 00 10 0. 00 33 .0 0 20 .8 0 12 C on ju nc tio n 6. 70 27 .2 5 99 .8 0 10 0. 00 10 0. 00 10 0. 00 30 .4 0 37 .7 0 13 C om po un d co re fe re nc e 5. 60 27 .7 3 99 .6 0 10 0. 00 10 0. 00 10 0. 00 33 .8 0 14 .0 0 14 Ti m e re as on in g 5. 00 27 .8 2 98 .3 0 99 .9 0 99 .8 0 95 .0 0 27 .6 0 50 .5 0 15 B as ic de du ct io n 5. 20 37 .2 0 10 0. 00 10 0. 00 10 0. 00 96 .7 0 39 .9 0 17 .6 0 16 B as ic in du ct io n 7. 50 45 .6 5 98 .7 0 48 .2 0 54 .7 0 50 .3 0 15 .1 0 48 .0 0 bA bI m ea n (1 1 ta sk s) 6. 06 29 .8 5 93 .7 3 94 .9 8 95 .6 9 92 .7 0 34 .8 2 38 .1 3
C.2.2 AVERAGE OVER ALL MODELS TRAINED ON BABI TASKS
Figure 5 plots mean accuracy of all models trained in our experiments. This suggests that pre-training helped all models, not only the top performing ones selected by validation as already shown in Figure 2a.
D MEANS, STANDARD DEVIATIONS AND P-VALUES BY EXPERIMENT
Table 5 shows the mean accuracy across all models trained for each combination of task, pre-training dataset and target-adjustment dataset size. Table 6 shows the corresponding standard deviations.
Table 7 then shows the p-value that whether the expected accuracy of pre-trained models is greater than the expected accuracy of randomly initialized models. This shows that the pre-trained models are statistically significantly better for all target-adjustment set sizes on the SQuAD dataset. On bAbI the BookTest pre-trained models perform convincingly better especially for target-adjustment dataset sizes 100, 500 and 1000, with Task 16 being the main exception to this because the AS Reader struggles to learn it in any setting. For the CNN+DM pre-training the results are not conclusive.
Ta sk
Pr et
ra in
in g
Ta rg
et -a
dj us
tm en
ts et
si ze
0 1
10 10
0 50
0 10
00 50
00 10
00 0
28 17 4 SQ uA D B oo kT es t 1. 01 e45 4. 07 e05 7. 40 e05 7. 82 e08 N A 5. 17 e08 N A 3. 93 e08 8.
52 e03 Ta sk 1 B oo kT es t 3. 34 e83 1. 81 e03 1. 33 e01 2. 35 e19 9. 41 e04 1. 67 e02 1. 32 e01 N A N A Ta sk 2 B oo kT es t 1. 24 e34 3. 86 e07 7. 29 e03 2. 59 e01 1. 39 e08 2. 63 e06 7. 54 e09 2. 04 e01 N A Ta sk 3 B oo kT es t 9. 84 e55 1. 27 e05 7. 66 e03 1. 48 e03 3. 18 e04 2. 18 e03 2. 16 e04 1. 03 e01 N A Ta sk 4 B oo kT es t 7. 25 e78 9. 50 e01 9. 71 e01 1. 04 e05 6. 38 e03 1. 70 e02 1. 81 e02 N A N A Ta sk 5 B oo kT es t 6. 55 e11 5 9. 88 e22 8. 87 e19 5. 25 e05 3. 66 e03 8. 61 e02 5. 65 e03 N A N A Ta sk 11 B oo kT es t 6. 78 e15 2 1. 00 e+ 00 9. 94 e01 4. 07 e09 2. 50 e04 2. 28 e02 6. 37 e02 N A N A Ta sk 12 B oo kT es t 2. 27 e90 9. 10 e01 6. 46 e01 1. 89 e05 2. 78 e04 1. 43 e02 2. 36 e02 N A N A Ta sk 13 B oo kT es t 5. 30 e91 9. 75 e01 9. 99 e01 2. 88 e02 2. 74 e02 1. 03 e01 7. 06 e02 N A N A Ta sk 14 B oo kT es t 1. 97 e20 0 1. 01 e03 6. 79 e01 2. 22 e14 3. 40 e05 2. 93 e03 3. 66 e06 3. 97 e01 N A Ta sk 15 B oo kT es t 3. 64 e09 4. 75 e01 4. 12 e01 6. 70 e01 1. 68 e03 3. 70 e03 1. 03 e05 4. 54 e01 N A Ta sk 16 B oo kT es t 1. 81 e05 8. 28 e04 4. 38 e01 2. 72 e01 4. 89 e01 5. 71 e01 7. 40 e03 N A N A Ta sk 1 C N N +D M 9. 43 e09 2. 99 e01 1. 11 e01 1. 05 e01 9. 54 e02 1. 45 e01 3. 97 e03 N A N A Ta sk 2 C N N +D M 9. 38 e17 6. 93 e01 9. 02 e01 9. 15 e01 1. 05 e03 4. 20 e01 2. 64 e03 8. 49 e02 N A Ta sk 3 C N N +D M 2. 42 e16 4. 95 e02 6. 30 e01 1. 75 e01 2. 13 e03 6. 59 e04 4. 68 e02 1. 24 e01 N A Ta sk 4 C N N +D M 5. 84 e03 9. 70 e01 1. 37 e01 4. 83 e03 3. 33 e01 8. 84 e01 1. 08 e01 N A N A Ta sk 5 C N N +D M 1. 17 e10 7. 00 e03 7. 93 e04 5. 20 e01 9. 70 e01 5. 66 e01 1. 83 e01 N A N A Ta sk 11 C N N +D M 1. 00 e+ 00 9. 84 e01 9. 73 e01 2. 58 e01 7. 17 e01 1. 45 e01 6. 95 e01 N A N A Ta sk 12 C N N +D M 1. 93 e14 9. 32 e01 9. 92 e01 2. 57 e02 4. 06 e01 6. 65 e02 2. 09 e01 N A N A Ta sk 13 C N N +D M 8. 69 e02 9. 61 e01 9. 72 e01 9. 89 e01 6. 22 e01 9. 44 e01 2. 83 e01 N A N A Ta sk 14 C N N +D M 2. 17 e12 6. 64 e02 1. 11 e01 2. 05 e02 3. 66 e02 4. 52 e01 9. 10 e01 8. 24 e01 N A Ta sk 15 C N N +D M 1. 36 e52 5. 30 e03 3. 48 e02 7. 21 e02 8. 36 e01 3. 09 e01 8. 47 e01 9. 84 e01 N A Ta sk 16 C N N +D M 6. 39 e35 4. 56 e02 9. 66 e01 5. 95 e01 7. 19 e01 4. 09 e02 2. 51 e02 2. 22 e03 N A
Ta bl
e 7:
O ne
-s id
ed p-
va lu
e w
he th
er th
e m
ea n
ac cu
ra cy
of pr
etr
ai ne
d m
od el
s is
gr ea
te rt
ha n
th e
ac cu
ra cy
of th
e ra
nd om
ly in
iti al
iz ed
on es
fo re
ac h
co m
bi na
tio n of ta sk pr etr ai ni ng da ta se t. pva lu es be lo w 0. 05 ar e m ar ke d in gr ee n.

First I would like to apologize for the delay in reviewing.
summary : This work explores several experiments to transfer training a specific model of reading comprehension ( AS Reader), in an artificial and well populated dataset in order to perform in another target dataset.
Here is what I understand are their several experiments to transfer learning, but I am not 100% sure.
1. The model is trained on the big artificial dataset and tested on the small target datasets (section 4.1)
2. The model is pre-trained on the big artificial dataset like before, then fine-tuned on a few examples from the target dataset and tested on the remaining target examples. Several such models are trained using different sub-sets of fine-tuning examples. The results are tested against the performance of randomly intialized then fine-tuned models (section 4.2).
3. The model is pre-trained on the big artificial dataset like before. The model is made of an embedding component and an encoder  component. Alternatively, each component is reset to a random initialization, to test the importance of the pre-training in each component. Then the model is fine-tuned on a few examples from the target dataset and tested on the remaining target examples. (section 4.3)
I think what makes things difficult to follow is the fact that the test set is composed by several sub tasks, and sometimes what is reported is the mean performance across the tasks, sometimes the performance on a few tasks. Sometimes what we see is the mean performance of several models? You should report standard deviations also. Could you better explain what you mean by best validation ?
Interesting and unpretentious work. The clarity of the presentation could be improved maybe by simplifying the experimental setup? The interesting conclusion I think is reported at the end of the section 4.1, when the nuanced difference between the datasets are exposed.
Minor: unexplained acronyms: GRU, BT, CBT.
benfits p. 2
subsubset p. 6