Machine comprehension of text is an important problem in natural language processing. A recently released dataset, the Stanford Question Answering Dataset (SQuAD), offers a large number of real questions and their answers created by humans through crowdsourcing. SQuAD provides a challenging testbed for evaluating machine comprehension algorithms, partly because compared with previous datasets, in SQuAD the answers do not come from a small set of candidate answers and they have variable lengths. We propose an end-to-end neural architecture for the task. The architecture is based on match-LSTM, a model we proposed previously for textual entailment, and Pointer Net, a sequence-to-sequence model proposed by Vinyals et al. (2015) to constrain the output tokens to be from the input sequences. We propose two ways of using Pointer Net for our tasks. Our experiments show that both of our two models substantially outperform the best results obtained by Rajpurkar et al. (2016) using logistic regression and manually crafted features. Besides, our boundary model also achieves the best performance on the MSMARCO dataset (Nguyen et al., 2016).
1 INTRODUCTION
Machine comprehension of text is one of the ultimate goals of natural language processing. While the ability of a machine to understand text can be assessed in many different ways, in recent years, several benchmark datasets have been created to focus on answering questions as a way to evaluate machine comprehension (Richardson et al., 2013; Hermann et al., 2015; Hill et al., 2016; Weston et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016). In this setup, typically the machine is first presented with a piece of text such as a news article or a story. The machine is then expected to answer one or multiple questions related to the text.
In most of the benchmark datasets, a question can be treated as a multiple choice question, whose correct answer is to be chosen from a set of provided candidate answers (Richardson et al., 2013; Hill et al., 2016). Presumably, questions with more given candidate answers are more challenging. The Stanford Question Answering Dataset (SQuAD) introduced recently by Rajpurkar et al. (2016) contains such more challenging questions whose correct answers can be any sequence of tokens from the given text. Moreover, unlike some other datasets whose questions and answers were created automatically in Cloze style (Hermann et al., 2015; Hill et al., 2016), the questions and answers in SQuAD were created by humans through crowdsourcing, which makes the dataset more realistic. Another real dataset, the Human-Generated MAchine Reading COmprehension dataset (MSMARCO) (Nguyen et al., 2016), provided a query together with several related documents collected from Bing Index. The answer to the query is generated by human and the answer words can not only come from the given text.
Given these advantages of the SQuAD and MSMARCO datasets, in this paper, we focus on these new datasets to study machine comprehension of text. A sample piece of text and three of its associated questions from SQuAD are shown in Table 1. Traditional solutions to this kind of question answering tasks rely on NLP pipelines that involve multiple steps of linguistic analyses and feature engineering, including syntactic parsing, named entity recognition, question classification, semantic parsing, etc. Recently, with the advances of applying neural network models in NLP, there has been
much interest in building end-to-end neural architectures for various NLP tasks, including several pieces of work on machine comprehension (Hermann et al., 2015; Hill et al., 2016; Yin et al., 2016; Kadlec et al., 2016; Cui et al., 2016). However, given the properties of previous machine comprehension datasets, existing end-to-end neural architectures for the task either rely on the candidate answers (Hill et al., 2016; Yin et al., 2016) or assume that the answer is a single token (Hermann et al., 2015; Kadlec et al., 2016; Cui et al., 2016), which make these methods unsuitable for the SQuAD/MSMARCO dataset. In this paper, we propose a new end-to-end neural architecture to address the machine comprehension problem as defined in the SQuAD/MSMARCO dataset. And for the MSMARCO dataset, we will only make use of the words in the given text to generate the answer.
Specifically, observing that in the SQuAD/MSMARCO dataset many questions could be entailed from some sentences in the original text, we adopt a match-LSTM model that we developed earlier for textual entailment (Wang & Jiang, 2016) as one layer of our model. We build a bi-directional match-LSTM on the given passage with attentions on the question for each word so that each position in the paragraph will have a hidden representation reflecting its relation to the question. Then we further adopt the Pointer Net (Ptr-Net) model developed by Vinyals et al. (2015) to select the words in these positions based on the hidden representations built by match-LSTM as an answer. We propose two ways to apply the Ptr-Net model for our task: a sequence model which selects the answer word by word, and a boundary model which only selects the start and end points of the answer span. Experiments on the SQuAD dataset show that our two models both outperform the best performance reported by Rajpurkar et al. (2016). Moreover, using an ensemble of several of our models, we can achieve very competitive performance on SQuAD. For the MSMARCO dataset, a real query based problem, our boundary model outperforms our sequence model with a big margin. It also outperforms the golden passage baseline.
Our contributions can be summarized as follows: (1) We propose two new end-to-end neural network models for machine comprehension, which combine match-LSTM and Ptr-Net to handle the special properties of the SQuAD dataset. To the best of our knowledge, we are the first to propose the boundary model which is more suitable to the SQuAD/MSMARCO tasks. And we are the first to integrate the attention-based word pair matching into machine comprehension tasks. (2) We have achieved the performance of an exact match score of 71.3% and an F1 score of 80.8% on the unseen SQuAD test dataset, which is much better than the feature-engineered solution (Rajpurkar et al., 2016). Our performance is also close to the state of the art on SQuAD, which is 74.8% in terms of exact match and 82.2% in terms of F1 collected from the SQuAD Leaderboard 1. Besides, our boundary model achieves the state-of-art performance on the MSMARCO dataset with BLUE1/2/3/4 40.7/33.9/30.6/28.7 and Rouge-L 37.3 2. (3) Our further visualization of the models reveals some useful insights of the attention mechanism for reasoning the questions. And we also show that the boundary model can overcome the early stop prediction problem in the sequence model. Besides, we also made our code available online 3.
1https://rajpurkar.github.io/SQuAD-explorer/ 2http://www.msmarco.org/leaders.aspx 3 https://github.com/shuohangwang/SeqMatchSeq
2 METHOD
In this section, we first briefly review match-LSTM and Pointer Net. These two pieces of existing work lay the foundation of our method. We then present our end-to-end neural architecture for machine comprehension.
2.1 MATCH-LSTM
In a recent work on learning natural language inference, we proposed a match-LSTM model for predicting textual entailment (Wang & Jiang, 2016). In textual entailment, two sentences are given where one is a premise and the other is a hypothesis. To predict whether the premise entails the hypothesis, the match-LSTM model goes through the tokens of the hypothesis sequentially. At each position of the hypothesis, attention mechanism is used to obtain a weighted vector representation of the premise. This weighted premise is then to be combined with a vector representation of the current token of the hypothesis and fed into an LSTM, which we call the match-LSTM. The matchLSTM essentially sequentially aggregates the matching of the attention-weighted premise to each token of the hypothesis and uses the aggregated matching result to make a final prediction.
2.2 POINTER NET
Vinyals et al. (2015) proposed a Pointer Network (Ptr-Net) model to solve a special kind of problems where we want to generate an output sequence whose tokens must come from the input sequence. Instead of picking an output token from a fixed vocabulary, Ptr-Net uses attention mechanism as a pointer to select a position from the input sequence as an output symbol. The pointer mechanism has inspired some recent work on language processing (Gu et al., 2016; Kadlec et al., 2016). Here we adopt Ptr-Net in order to construct answers using tokens from the input text.
2.3 OUR METHOD
Formally, the problem we are trying to solve can be formulated as follows. We are given a piece of text, which we refer to as a passage, and a question related to the passage. The passage is
represented by matrix P ∈ Rd×P , where P is the length (number of tokens) of the passage and d is the dimensionality of word embeddings. Similarly, the question is represented by matrix Q ∈ Rd×Q where Q is the length of the question. Our goal is to identify a subsequence from the passage as the answer to the question.
As pointed out earlier, since the output tokens are from the input, we would like to adopt the Pointer Net for this problem. A straightforward way of applying Ptr-Net here is to treat an answer as a sequence of tokens from the input passage but ignore the fact that these tokens are consecutive in the original passage, because Ptr-Net does not make the consecutivity assumption. Specifically, we represent the answer as a sequence of integers a = (a1, a2, . . .), where each ai is an integer between 1 and P , indicating a certain position in the passage.
Alternatively, if we want to ensure consecutivity, that is, if we want to ensure that we indeed select a subsequence from the passage as an answer, we can use the Ptr-Net to predict only the start and the end of an answer. In this case, the Ptr-Net only needs to select two tokens from the input passage, and all the tokens between these two tokens in the passage are treated as the answer. Specifically, we can represent the answer to be predicted as two integers a = (as, ae), where as an ae are integers between 1 and P .
We refer to the first setting above as a sequence model and the second setting above as a boundary model. For either model, we assume that a set of training examples in the form of triplets {(Pn,Qn,an)}Nn=1 are given. An overview of the two neural network models are shown in Figure 1. Both models consist of three layers: (1) An LSTM preprocessing layer that preprocesses the passage and the question using LSTMs. (2) A match-LSTM layer that tries to match the passage against the question. (3) An Answer Pointer (Ans-Ptr) layer that uses Ptr-Net to select a set of tokens from the passage as the answer. The difference between the two models only lies in the third layer.
LSTM Preprocessing Layer
The purpose for the LSTM preprocessing layer is to incorporate contextual information into the representation of each token in the passage and the question. We use a standard one-directional LSTM (Hochreiter & Schmidhuber, 1997) to process the passage 4 and the question separately, as shown below:
Hp = −−−→ LSTM(P), Hq = −−−→ LSTM(Q). (1)
The resulting matrices Hp ∈ Rl×P and Hq ∈ Rl×Q are hidden representations of the passage and the question, where l is the dimensionality of the hidden vectors. In other words, the ith column vector hpi (or h q i ) in H
p (or Hq) represents the ith token in the passage (or the question) together with some contextual information from the left.
Match-LSTM Layer
We apply the match-LSTM model (Wang & Jiang, 2016) proposed for textual entailment to our machine comprehension problem by treating the question as a premise and the passage as a hypothesis. The match-LSTM sequentially goes through the passage. At position i of the passage, it first uses the standard word-by-word attention mechanism to obtain attention weight vector −→α i ∈ R1×Q as follows:
−→ Gi = tanh(WqHq + (Wph p i + W r−→h ri−1 + bp)⊗ eQ), −→α i = softmax(wᵀ −→ Gi + b⊗ eQ), (2)
where Wq,Wp,Wr ∈ Rl×l, bp,w ∈ Rl×1 and b ∈ R are parameters to be learned, −→ Gi ∈ Rl×Q is the intermediate result, −→ h ri−1 ∈ Rl×1 is the hidden vector of the one-directional match-LSTM (to be explained below) at position i−1, and the outer product (·⊗eQ) produces a matrix or row vector by repeating the vector or scalar on the left for Q times.
Essentially, the resulting attention weight −→α i,j above indicates the degree of matching between the ith token in the passage with the jth token in the question. Next, we use the attention weight vector
4For the MSMARCO dataset, P is actually consisted of several unrelated documents. The previous state of pre-processing LSTM and match-LSTM to compute the first state of each document is set to zero.
−→α i to obtain a weighted version of the question and combine it with the current token of the passage to form a vector −→z i:
−→z i = [
hpi Hq−→α ᵀi
] , (3)
where Hq ∈ Rl×Q, −→α i ∈ R1×Q and hpi ∈ Rl×1. This vector −→z i is fed into a standard onedirectional LSTM to form our so-called match-LSTM: −→ h ri = −−−→ LSTM(−→z i, −→ h ri−1), (4) where −→ h ri ∈ Rl×1.
We further build a similar match-LSTM in the reverse direction. The purpose is to obtain a representation that encodes the contexts from both directions for each token in the passage.
Let −→ Hr ∈ Rl×P represent the hidden states [ −→ h r1, −→ h r2, . . . , −→ h rP ] and ←− Hr ∈ Rl×P represent [ ←− h r1, ←− h r2, . . . , ←− h rP ], the hidden states of match-LSTM in the reverse direction. We define H
r ∈ R2l×P as the concatenation of the two:
Hr = [−→ Hr ←− Hr ] . (5)
Answer Pointer Layer
The top layer, the Answer Pointer (Ans-Ptr) layer, is motivated by the Pointer Net introduced by Vinyals et al. (2015). This layer uses the sequence Hr as input. Recall that we have two different models: The sequence model produces a sequence of answer tokens but these tokens may not be consecutive in the original passage. The boundary model produces only the start token and the end token of the answer, and then all the tokens between these two in the original passage are considered to be the answer. We now explain the two models separately.
The Sequence Model: Recall that in the sequence model, the answer is represented by a sequence of integers a = (a1, a2, . . .) indicating the positions of the selected tokens in the original passage. The Ans-Ptr layer models the generation of these integers in a sequential manner. Because the length of an answer is not fixed, in order to stop generating answer tokens at certain point, we allow each ak to take up an integer value between 1 and P + 1, where P + 1 is a special value indicating the end of the answer. Once ak is set to be P + 1, the generation of the answer stops.
In order to generate the kth answer token indicated by ak, first, the attention mechanism is used again to obtain an attention weight vector βk ∈ R1×(P+1), where βk,j (1 ≤ j ≤ P + 1) is the probability of selecting the jth token from the passage as the kth token in the answer, and βk,(P+1) is the probability of stopping the answer generation at position k. βk is modeled as follows:
Fk = tanh(VH̃r + (Wahak−1 + b a)⊗ e(P+1)), (6) βk = softmax(vᵀFk + c⊗ e(P+1)), (7)
where H̃r ∈ R2l×(P+1) is the concatenation of Hr with a zero vector, defined as H̃r = [Hr;0], V ∈ Rl×2l,Wa ∈ Rl×l, ba,v ∈ Rl×1 and c ∈ R are parameters to be learned, Fk ∈ Rl×(P+1) is the intermediate result, (· ⊗ e(P+1)) follows the same definition as before, and hak−1 ∈ Rl×1 is the hidden vector at position k − 1 of an answer LSTM as defined below:
hak = −−−→ LSTM(H̃rβᵀk ,h a k−1). (8)
We can then model the probability of generating the answer sequence as p(a|Hr) = ∏ k p(ak|a1, a2, . . . , ak−1,Hr), (9)
and
p(ak = j|a1, a2, . . . , ak−1,Hr) = βk,j . (10)
To train the model, we minimize the following loss function based on the training examples:
− N∑
n=1
log p(an|Pn,Qn). (11)
The Boundary Model: The boundary model works in a way very similar to the sequence model above, except that instead of predicting a sequence of indices a1, a2, . . ., we only need to predict two indices as and ae. So the main difference from the sequence model above is that in the boundary model we do not need to add the zero padding to Hr, and the probability of generating an answer is simply modeled as
p(a|Hr) = p(as|Hr)p(ae|as,Hr). (12)
As this boundary model could point to a span covering too many tokens without any restriction, we try to manually limit the length of the predicted span and then search the span with the highest probability computed by p(as)× p(ae|as) as the answer.
3 EXPERIMENTS
In this section, we present our experiment results and perform some analyses to better understand how our models works.
3.1 DATA
We use the Stanford Question Answering Dataset (SQuAD) v1.1 and the human-generated Microsoft MAchine Reading COmprehension (MSMARCO) dataset v1.1 to conduct our experiments.
Passages in SQuAD come from 536 articles in Wikipedia covering a wide range of topics. Each passage is a single paragraph from a Wikipedia article, and each passage has around 5 questions associated with it. In total, there are 23,215 passages and 107,785 questions. The data has been split into a training set (with 87,599 question-answer pairs), a development set (with 10,570 questionanswer pairs) and a hidden test set.
For the MSMARCO dataset, the questions are user queries issued to the Bing search engine, the context passages are real Web documents and the answers are human-generated. We select the span that has the highest F1 score with the gold standard answer for training and only predict the span in the passages during evaluation. The data has been split into a training set (82326 pairs), a development set (10047 pairs) and a test set (9650 pairs).
3.2 EXPERIMENT SETTINGS
We first tokenize all the passages, questions and answers. We use word embeddings from GloVe (Pennington et al., 2014) to initialize the model. Words not found in GloVe are initialized as zero vectors. The word embeddings are not updated during the training of the model.
The dimensionality l of the hidden layers is set to be 150. We use ADAMAX (Kingma & Ba, 2015) with the coefficients β1 = 0.9 and β2 = 0.999 to optimize the model. Each update is computed through a minibatch of 30 instances. We do not use L2-regularization.
For the SQuAD dataset, the performance is measured by two metrics: percentage of exact match with the ground truth answers and word-level F1 score when comparing the tokens in the predicted answers with the tokens in the ground truth answers. Note that in the development set and the test set each question has around three ground truth answers. F1 scores with the best matching answers are used to compute the average F1 score. For the MSMARCO dataset, the metrics in the official tool of MSMARCO evaluation are BLEU-1/2/3/4 and Rouge-L, which are widely used in many domains.
3.3 RESULTS
The SQuAD and MSMARCO results of our models as well as the results of the baselines (Rajpurkar et al., 2016; Yu et al., 2016) are shown in Table 2. For the “LSTM with Ans-Ptr” models, they are
the experiments with the ablation of attention mechanism in match-LSTM. Specifically, we use the final representation of the question to replace the weighted sum of the question representations. For the MSMARCO dataset, as the context for each question is consisted of around 10 documents, the “Golden Passage” is to directly use the human labeled document which could answer the question as the prediction.
From the results in Table 2, we can see that the boundary model could clearly outperform the sequence model in a big margin on both datasets. We hypothesis that the sequence model is more likely to stop word generation earlier, and the boundary model can somehow overcome this problem. We have a statistical analysis on the answers generated by our sequence and boundary models shown in Table 3. We can see that the length of the answers generated by the sequence model is much shorter than the ground truth. Especially for the MSMARCO task where the answers are usually much longer, the sequence model could only generate 7 words on average, while the ground truth answers are 16 on average and the boundary model could generate nearly the same number of words with the ground truth. Several answers generated by our models are shown in Appendix A. From Table 2, we can also see that the performance gets poorer by removing the attention mechanism in match-LSTM, while for the MSMARCO dataset, the attention mechanism effects less, with no more than 2 percent reduction in BLEU and Rouge-L scores by attention mechanism ablation.
Based on the effectiveness of boundary pointer and match-LSTM, we conduct further exploration of the boundary model by adding element-wise comparison (hpi−H qαᵀi ) and (h p i H
qαᵀi ) into Eqn 3 in match-LSTM layer, adding 2 more bi-directional LSTM layers between match-LSTM and Ans-Ptr layers, and adding bi-directional Ans-Ptr. We show the ablation study of this further tuned model in Table 4. We can see that adding element-wise matching could make the biggest improvement for our boundary model. We also try to remove the phrase-level representation by removing the pre-process LSTM and using the word-level representations as the inputs of match-LSTM. Interestingly, we find the phrase-level representation effects little on the MSMARCO task.
Overall, we can see that both of our match-LSTM models have clearly outperformed the logistic regression model by Rajpurkar et al. (2016), which relies on carefully designed features. The improvement of our models over the logistic regression model shows that our end-to-end neural network models without much feature engineering are very effective on these tasks and datasets. Our boundary model also outperformed the DCR model (Yu et al., 2016), which maximizes the probability of the gold standard span from all the candidate spans through a neural network structure.
3.4 FURTHER ANALYSES
To better understand the strengths and weaknesses of our models, we perform some further analyses of the results below.
First, we suspect that longer answers are harder to predict. To verify this hypothesis, we analysed the performance in terms of both exact match and F1 score with respect to the answer length on the development set, as shown in Figure 2. For example, for questions whose answers contain more than 9 tokens, the F1 score of the boundary model drops to around 55% and the exact match score drops to only around 30%, compared to the F1 score and exact match score of close to 72% and 67%, respectively, for questions with single-token answers. And that supports our hypothesis.
Next, we analyze the performance of our models on different groups of questions, as shown in Figure 2. We use a crude way to split the questions into different groups based on a set of question words we have defined, including “what,” “how,” “who,” “when,” “which,” “where,” and “why.” These different question words roughly refer to questions with different types of answers. For example, “when” questions look for temporal expressions as answers, whereas “where” questions look for locations as answers. According to the performance on the development dataset, our models work the best for “when” questions. This may be because in this dataset temporal expressions are relatively easier to recognize. Other groups of questions whose answers are noun phrases, such as “what” questions, “which” questions and “where” questions, also get relatively better results. On the other hand, “why” questions are the hardest to answer. This is not surprising because the answers to “why” questions can be very diverse, and they are not restricted to any certain type of phrases.
Finally, we would like to check whether the attention mechanism used in the match-LSTM layer is effective in helping the model locate the answer. We show the attention weights α in Figure 3. In the figure the darker the color is the higher the weight is. We can see that some words have been well aligned based on the attention weights. For example, the word “German” in the passage is aligned well to the word “language” in the first question, and the model successfully predicts “German” as the answer to the question. For the question word “who” in the second question, the
word “teacher” actually receives relatively higher attention weight, and the model has predicted the phrase “Martin Sekulic” after that as the answer, which is correct. For the third question that starts with “why”, the attention weights are more evenly distributed and it is not clear which words have been aligned to “why”. For the last question, we can see that the word knowledge needed for generating the answer can also be detected by match-LSTM. For example, the words “European”, “Parliament”, “Council”, “European” and “Union” have higher attention weights on “governing” in the question. Even though our models can solve this type of questions, they are still not able to solve the questions that need multi-sentences reasoning. More answers generated by our models for the questions related to different kinds of reasoning are shown in Appendix B.
4 RELATED WORK
Machine comprehension of text has gained much attention in recent years, and increasingly researchers are building data-drive, end-to-end neural network models for the task. We will first review the recently released datasets and then some end-to-end models on this task.
4.1 DATASETS
A number of datasets for studying machine comprehension were created in Cloze style by removing a single token from a sentence in the original corpus, and the task is to predict the missing word. For example, Hermann et al. (2015) created questions in Cloze style from CNN and Daily Mail highlights. Hill et al. (2016) created the Children’s Book Test dataset, which is based on children’s stories. Cui et al. (2016) released two similar datasets in Chinese, the People Daily dataset and the Children’s Fairy Tale dataset.
Instead of creating questions in Cloze style, a number of other datasets rely on human annotators to create real questions. Richardson et al. (2013) created the well-known MCTest dataset and Tapaswi et al. (2016) created the MovieQA dataset. In these datasets, candidate answers are provided for each question. Similar to these two datasets, the SQuAD dataset (Rajpurkar et al., 2016) was also created by human annotators. Different from the previous two, however, the SQuAD dataset does not provide candidate answers, and thus all possible subsequences from the given passage have to be considered as candidate answers.
Besides the datasets above, there are also a few other datasets created for machine comprehension, such as WikiReading dataset (Hewlett et al., 2016) and bAbI dataset (Weston et al., 2016), but they are quite different from the datasets above in nature.
4.2 END-TO-END NEURAL NETWORK MODELS FOR MACHINE COMPREHENSION
There have been a number of studies proposing end-to-end neural network models for machine comprehension. A common approach is to use recurrent neural networks (RNNs) to process the given text and the question in order to predict or generate the answers (Hermann et al., 2015). Attention mechanism is also widely used on top of RNNs in order to match the question with the given passage (Hermann et al., 2015; Chen et al., 2016). Given that answers often come from the given passage, Pointer Network has been adopted in a few studies in order to copy tokens from the given passage as answers (Kadlec et al., 2016; Trischler et al., 2016). Compared with existing work, we use match-LSTM to match a question and a given passage, and we use Pointer Network in a different way such that we can generate answers that contain multiple tokens from the given passage.
Memory Networks (Weston et al., 2015) have also been applied to machine comprehension (Sukhbaatar et al., 2015; Kumar et al., 2016; Hill et al., 2016), but its scalability when applied to a large dataset is still an issue. In this work, we did not consider memory networks for the SQuAD/MSMARCO datasets.
The setting of visual question answering (Antol et al., 2015) is quite similar to machine comprehension, while their answers are usually very short. So the sequence order of the word-level attention representation used to align the figure and the question(Xu & Saenko, 2016; Fukui et al., 2016; Lu et al., 2016), are not used in VQA. While our model focus on the word-by-word attention and use
LSTM to concatenate the aligned pairs and that would be helpful to generate a longer sequence as answer.
5 CONCLUSIONS
In this paper, We developed two models for the machine comprehension problem defined in the Stanford Question Answering (SQuAD) and A Human-Generated MAchine Reading COmprehension (MSMARCO) datasets, both making use of match-LSTM and Pointer Network. Experiments on the SQuAD and MSMARCO datasets showed that our second model, the boundary model, could achieve a performance close to the state-of-the-art performance on the SQuAD dataset and achieved the state-of-the-art on the MSMARCO dataset. We also show the boundary model could overcome the early stop prediction problem of the sequence model.
In the future, we plan to look further into the different types of questions and focus on those questions which currently have low performance, such as the “why’ questions and multi-sentences related questions. We also plan to test how our models could be applied to other machine comprehension datasets.
6 ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Prime Ministers Office, Singapore under its International Research Centres in Singapore Funding Initiative.
We thank Pranav Rajpurkar for testing our model on the hidden test dataset and Percy Liang for helping us with the Dockerfile for Codalab.
A APPENDIX
We show the predictions our boundary and sequence models on two cases from two datasets in Table 5. It can be seen that the sequence model is more likely to predict a shorter sequence which is the problem of early stop prediction.
B APPENDIX
We show how four different models work on different type of questions in SQuAD dataset through Table 6. After the analysis of a hundred cases, we see that our models are not able to solve the questions that need multi-sentences reasoning. And the model without attention mechanism has less power to identify the important key word like the third case shown in Table 6.

Summary:
The paper presents a deep neural network for the task of machine comprehension on the SQuAD dataset. The proposed model is based on two previous works -- match-LSTM and Pointer Net. Match-LSTM produces attention over each word in the given question for each word in the given passage, and sequentially aggregates this matching of each word in the passage with the words in the question. The pointer net is used to generate the answer by either generating each word in the answer or by predicting the starting and ending tokens in the answer from the provided passage. The experimental results show that both the variants of the proposed model outperform the baseline presented in the SQuAD paper. The paper also shows some analysis of the results obtained such as variation of performance across answer lengths and question types.
Strengths:
1. A novel end-to-end model for the task of machine comprehension rather than using hand-crafted features.
2. Significant performance boost over the baseline presented in the SQuAD paper.
3. Some insightful analyses of the results such as performance is better when answers are short, "why" questions are difficult to answer.
Weaknesses/Questions/Suggestions:
1. The paper does not show quantitatively how much modelling attention in match-LSTM and answer pointer layer helps. So, it would be insightful if authors could compare the model performance with and without attention in match-LSTM, and with and without attention in answer pointer layer.
2. It would be good if the paper could provide some insights into why there is a huge performance gap between boundary model and sequence model in the answer pointer layer.
3. I would like to see the variation in the performance of the proposed model for questions that require different types of reasoning (table 3 in SQuAD paper). This would provide insights into what are the strengths and weaknesses of the proposed model w.r.t the type reasoning required.
4. Could authors please explain why the activations resulting from {h^p}_i and {h^r}_{i-1} in G_i in equation 2 are being repeated across dimension of Q. Why not learn different activations for each dimension?
5. I wonder why Bi-Ans-Ptr is not used in the ensemble model (last row in table 2) when it is shown that Bi-Ans-Ptr improves performance by 1.2% in F1.
6. Could authors please discuss and compare the DCR model (in table 2) in the paper in more detail?
Review Summary: The paper presents a reasonable end-to-end model for the task of machine comprehension on the SQuAD dataset, which outperforms the baseline model significantly. However, it would be good if more analyses / ablation studies / insights are included regarding -- how much attention helps, why is boundary model better than sequence model, how does the performance change when the reasoning required becomes difficult.


The optimization of deep neural networks can be more challenging than the traditional convex optimization problems due to highly non-convex nature of the loss function, e.g. it can involve pathological landscapes such as saddle-surfaces that can be difficult to escape from for algorithms based on simple gradient descent. In this paper, we attack the problem of optimization of highly non-convex neural networks objectives by starting with a smoothed – or mollified – objective function which becomes more complex as the training proceeds. Our proposition is inspired by the recent studies in continuation methods: similarly to curriculum methods, we begin by learning an easier (possibly convex) objective function and let it evolve during training until it eventually becomes the original, difficult to optimize objective function. The complexity of the mollified networks is controlled by a single hyperparameter that is annealed during training. We show improvements on various difficult optimization tasks and establish a relationship between recent works on continuation methods for neural networks and mollifiers.
1 INTRODUCTION
In the last few years, deep neural networks – i.e. convolutional networks (LeCun et al., 1989), LSTMs (Hochreiter & Schmidhuber, 1997a) or GRUs (Cho et al., 2014) – set the state of the art on a range of challenging tasks (Szegedy et al., 2014; Visin et al., 2015; Hinton et al., 2012; Sutskever et al., 2014; Bahdanau et al., 2014; Mnih et al., 2013; Silver et al., 2016). However when trained with variants of SGD (Bottou, 1998) deep networks can be difficult to optimize due to their highly non-linear and non-convex nature (Choromanska et al., 2014; Dauphin et al., 2014).
A number of approaches were proposed to alleviate the difficulty of optimization: addressing the problem of the internal covariate shift with Batch Normalization (Ioffe & Szegedy, 2015), learning with a curriculum (Bengio et al., 2009), recently an approach to train RNNs with diffusion process (Mobahi, 2016), and graduated optimization (Hazan et al., 2015). The impact of noise injection on the behavior of modern deep learning methods has been explored by Neelakantan et al. (2015a). Hazan et al. (2015) have shown that injecting a particular noise and scheduling it carefully, can guarantee the convergence in O(1/σ2 2) steps for -optimal and σ-nice functions. Similar to our work graduated optimization optimizes a smoothed objective function without performing expensive convolutions. Injecting noise to the activation functions and scheduling it have been recently shown to improve the performance on a wide variety of tasks (Gulcehre et al., 2016).
We connect the ideas of curriculum learning and continuation methods with those arising from models with skip connections and using layers that compute near-identity transformations. Skip connections allow to train very deep residual and highway architectures (He et al., 2015; Srivastava et al., 2015) by skipping layers or block of layers. Similarly, it has been shown that stochastically changing the depth of a network during training (Huang et al., 2016b) does not prevent convergence and enables better generalization performance.
We discuss the idea of mollification for neural networks – a form of differentiable smoothing of the loss function connected to noisy activations – which in our case can be interpreted as a form of adaptive noise injection which is controlled by a single hyperparameter. Inspired by Huang et al. (2016b), we use a hyperparameter to stochastically control the depth of our network. This allows us to start the optimization from a convex objective function (as long as the optimized criterion is convex, e.g. linear or logistic regression) and to slowly introduce more complexity into the model by annealing the hyperparameter, thus making the network deeper and increasingly non-linear.
∗ This work was done while these students were interning at the MILA lab in University of Montreal.
An important difference of our work compared to injecting noise to the gradients as it is explored in (Hazan et al., 2015; Neelakantan et al., 2015b) is that we inject the noise in the forward computation of the graph and we shape the cost function directly. As a result the cost function for the mollified network both at the test time and during the training are consistent and this makes the early-stopping much easier.
2 MOLLIFYING OBJECTIVE FUNCTIONS
2.1 CONTINUATION AND ANNEALING METHODS
Continuation methods and simulated annealing provide a general strategy to reduce the impact of local minima and deal with non-convex, continuous, but not necessarily everywhere differentiable objective functions by smoothing the original objective function and gradually reducing the amount of smoothing during training (Allgower & Georg, 1980) (see Fig. 1).
In machine learning, approaches based on curriculum learning (Bengio et al., 2009) are inspired by this principle and define a sequence of gradually more difficult training tasks (or training distributions) that eventually converge to the task of interest.
In the context of stochastic gradient descent, we use a stochastic estimation of the gradient for the smoothed objective function. This is convenient because it may not be analytically feasible to compute the smoothed function, but a Monte-Carlo estimate can often be obtained easily.
In this paper we construct a sequence of smoothed objective functions obtained with a form of mollification and we progressively optimize them. The training procedure iterates over the sequence of objective functions starting from the simpler ones – i.e. with a smoother loss surface – and moving towards more complex ones until the last, original, objective function is reached.1
2.2 MOLLIFIERS AND WEAK GRADIENTS
We smooth the loss function L, which is parametrized by θ ∈ Rn, by convolving it with another function K(·) with stride τ ∈ Rn:
LK(θ) = (L ∗K)(θ) = ∫ +∞ −∞ L(θ − τ )K(τ )dτ (1)
Although there are many choices for the function K(·), we focus on those that satisfy the definition of a mollifier.
A mollifier is an infinitely differentiable function that behaves like an approximate identity in the group of convolutions of integrable functions. If K(·) is an infinitely differentiable function, that converges to the Dirac delta function when appropriately rescaled and for any integrable function L, then it is a mollifier:
L(θ) = lim →0
∫ −nK(τ/ )L(θ − τ )dτ . (2)
1We plan to release the source code of the models and experiments under, http://github.com/ caglar/molly_nets/.
If we choose K(·) to be a mollifier and obtain the smoothed loss function LK as in Eqn. 1, we can take its gradient with respect to θ using directly the result from Evans (1998):
∇θLK(θ) = ∇θ(L ∗K)(θ) = (L ∗ ∇K)(θ). (3)
To relate the resulting gradient ∇θLK to the gradient of the original function L, we introduce the notion of weak gradient, i.e. an extension to the idea of weak/distributional derivatives to functions with multidimensional arguments, such as loss functions of neural networks.
For an integrable function L in space L ∈ L([a, b]), g ∈ L([a, b]n) is a n-dimensional weak gradient of L if it satisfies: ∫
C g(τ )K(τ )dτ = − ∫ C L(τ )∇K(τ )dτ , (4)
where K(τ ) is an infinitely differentiable function vanishing at infinity, C ∈ [a, b]n and τ ∈ Rn.
As long as the chosen K(·) fulfills the definition of a mollifier we can use Eqn. 3 and Eqn. 4 2 to rewrite the gradient as:
∇θLK(θ) = (L ∗ ∇K)(θ) by Eqn. 3 (5)
= ∫ C L(θ − τ )∇K(τ )dτ (6)
= − ∫ C g(θ − τ )K(τ )dτ by Eqn. 4 (7)
For a differentiable almost everywhere function L, the weak gradient g(θ) is equal to ∇θL almost everywhere. With a slight abuse of notation we can therefore write:
∇θLK(θ) = − ∫ C ∇θL(θ − τ )K(τ )dτ (8)
2.3 GAUSSIAN MOLLIFIERS
It is possible to use the standard Gaussian distribution N (0, I) as a mollifier K(·), as it satisfies the desired properties: it is infinitely differentiable, a sequence of properly rescaled Gaussian distributions converges to the Dirac delta function and it vanishes in infinity. With such a K(·) the gradient becomes:
∇θLK=N (θ) = − ∫ C ∇θL(θ − τ )p(τ )dτ (9)
= Eτ [∇θL(θ − τ ) ], with τ ∼ N (0, I) (10)
Exploiting the fact that a Gaussian distribution is a mollifier, we can focus on a sequence of mollifications indexed by scaling parameter introduced in Eqn. 2. A single element of this sequence takes the following form:
∇θLN , (θ) = − ∫ C ∇θL(θ − τ ) −1p(τ/ )dτ (11)
= Eτ [∇θL(θ − τ ) ], with τ ∼ N (0, 2I) (12)
Replacing with σ yields a sequence of mollifications indexed by σ:
∇θLN ,σ(θ) = Eτ [∇θL(θ − τ ) ], with τ ∼ N (0, σ2I) (13) with the following property (by Eqn. 2):
lim σ→0 ∇θLN ,σ(θ) = ∇θL(θ) (14)
An intuitive interpretation of the result is that σ determines the standard deviation of a mollifying Gaussian and is annealed in order to construct a sequence of gradually less "blurred" and closer
2We omit for brevity the algebraic details involved with a translation of the argument.
approximations to L. This is consistent with the property that when σ is annealed to zero we are optimizing the original function L. So far we obtained the mollified version LK(θ) of the cost function L(θ) by convolving it with a mollifier K(θ). The kernel K(θ) corresponds to the average effect of injecting noise ξ sampled from standard Normal distribution. The amount of noise controls the amount of smoothing. Gradually reducing the noise during training is related to a form of simulated annealing (Kirkpatrick et al., 1983). Similarly to the analysis in Mobahi (2016), we can write a Monte-Carlo estimate of LK(θ) = (L ∗K)(θ) ≈ 1N ∑N i=1 L(θ − ξ(i)). We provide the derivation and the gradient of this equation in Appendix A.
The Monte-Carlo estimators of the mollifiers can be easily implemented with neural networks, where the layers typically have the form:
hl = f(Wlhl−1) (15)
with hl−1 a vector of activations from the previous layer in the hierarchy, Wl a matrix representing a linear transformation and f an element-wise non-linearity of choice.
A mollification of such a layer can be formulated as:
hl = f((Wl − ξl)hl−1), where ξl ∼ N (µ, σ2) (16)
From Eqn. 16, it is easy to see that both weight noise methods proposed by Hinton & van Camp (1993) and Graves (2011) can be seen as a variation of Monte-Carlo estimate of mollifiers.
2.4 GENERALIZED AND NOISY MOLLIFIERS
We introduce a generalization of the concept of mollifiers that encompasses the approach we explored here and that is targeted during optimization via a continuation method using stochastic gradient descent. Definition 2.1. (Generalized Mollifier). A generalized mollifier is an operator, where Tσ(f) defines a mapping between two functions, such that Tσ : f → f∗.
lim σ→0 Tσf = f, (17)
f0 = lim σ→∞ Tσf is an identity function (18)
∂(Tσf)(x)
∂x exists ∀x, σ > 0 (19)
In addition, we consider noisy mollifiers which can be defined as an expected value of a stochastic function φ(x, ξ) under some noise source ξ with variance σ:
(Tσf)(x) = Eξ[φ(x, ξσ)] (20)
Definition 2.2. (Noisy Mollifier). We call a stochastic function φ(x, ξσ) with input x and noise ξ a noisy mollifier if its expected value corresponds to the application of a generalized mollifier Tσ, as per Eqn. 20.
The composition of two noisy mollifiers sharing the same σ is also a noisy mollifier, since the three properties in the definition (Eqns. 17,18,19) are still satisfied. When σ = 0 no noise is injected and therefore the original function will be optimized. If σ → ∞ instead, the function will become an identity function. Thus, for instance, if we mollify each layer of a feed-forward network except the output layer, when σ →∞ all the mollified layers will become identity function and the objective function of the network with respect to its inputs will be convex.
Consequently, corrupting separately the activation function of each level of a deep neural network (but with a shared noise level σ) and annealing σ yields a noisy mollifier for the objective function. This is related to the work of Mobahi (2016), who recently introduced a way of analytically smoothing of the non-linearities to help the training of recurrent networks. The differences of that approach from our algorithm is two-fold: we use a noisy mollifier (rather than an analytic smoothing of the network’s non-linearities) and we introduce (in the next section) a particular form of the noisy mollifier that empirically proved to work well.
Shaping the cost function to define a sequence of costs that are progressing from easier to more difficult ones can be related to the reward shaping (Ng et al., 1999; Ng, 2003) algorithms. In our algorithm, we shape the cost and the model architecture itself, rather than rewards or the targets in order to make the optimization easier. In that sense, reward shaping can be considered to be more closer to curriculum learning.
3 METHOD
We propose an algorithm to mollify the cost of a neural network which also addresses an important drawback of the previously proposed noisy training procedures: as the noise gets larger, it can dominate the learning process and lead the algorithm to perform a random walk on the energy landscape of the objective function. Conversely in our algorithm, as the noise gets larger gradient descent minimizes a simpler (e.g. convex) but still meaningful objective function.
We define the desired behavior of the network in the limit cases where the noise is very large or very small, and modify the model architecture accordingly. Specifically, during training we minimize a sequence of increasingly complex noisy objectives L = (L1(θ; ξσ1),L2(θ; ξσ2), · · · ,Lk(θ; ξσk)) that we obtain by annealing the scale (variance) of the noise σi. Let us note that our algorithm satisfies the fundamental properties of the generalized and noisy mollifiers that we introduced earlier.
We use a noisy mollifier based on our definition in Section 2.4. Instead of convolving the objective function with a kernel:
1. We start the training by optimizing a convex objective function that is obtained by configuring all the layers between the input and the last cost layer to compute an identity function, i.e., by skipping both the affine transformations and the blocks followed by nonlinearities. 2. During training, the magnitude of noise which is proportional to p is annealed, allowing to gradually evolve from identity transformations to linear transformations between the layers. 3. Simultaneously, as we decrease the p, the noisy mollification procedure allows the element-wise activation functions to gradually change from linear to be nonlinear.
4 SIMPLIFYING THE OBJECTIVE FUNCTION FOR FEEDFORWARD NETWORKS
For every unit of each layer, we either copy the activation (output) of the corresponding unit of the previous layer (the identity path in Figure 2) or output a noisy activation h̃l of a non-linear transformation of it ψ(hl−1, ξ;Wl), where ξ is noise, Wl is a weight matrix applied on hl−1 and π is a vector of binary decisions for each unit (the convolutional path in Figure 2):
h̃l = ψ(hl−1, ξ;Wl) (21)
φ(hl−1, ξ,πl;Wl) = πl hl−1 + (1− πl) h̃l (22) hl = φ(hl−1, ξ,πl;Wl). (23)
To decide which path to take, for each unit in the network, a binary stochastic decision is taken by drawing from a Bernoulli distribution with probability dependent on the decaying value of pl:
πl ∼ Bernoulli(pl) (24)
If the number of hidden units of layer l−1 and layer l+1 is not the same, we can either zero-pad layer l−1 before feeding it into the next layer or apply a linear projection to obtain the right dimensionality.
For pl = 1, the layer computes the identity function leading to a convex objective. If pl = 0 the layer computes the original non-linear transformation unfolding the full capacity of the model. We call the connections that are introduced this way as unitwise stochastic skip connections(USSC).
In DropIn layers (Smith et al., 2016) a binary random vector is sampled from a Bernoulli distribution to decide whether to introduce skip a connection from the layer below l − 1 for each layer l and they use this as a regularization. As opposed to the USSC, DropIn layers do not necessarily do not necessarily achieve a convex objective function as the DropIn ratio (pl) increases.
The pseudo-code for the mollified activations is reported in Algorithm 1.
Algorithm 1 Activation of a unit i at layer l. 1: xi ← w>i hl−1 + bi . an affine transformation of hl−1 2: ∆i ← u(xi)− f(xi) . ∆i is a measure of a saturation of a unit 3: σ(xi)← (sigmoid(ai∆i)− 0.5)2 . std of the injected noise depends on ∆i 4: ξi ∼ N (0, 1) . sampling the noise from a basic Normal distribution 5: si ← pl c σ(xi)|ξi| . Half-Normal noise controlled by σ(xi), const. c and prob-ty pl 6: ψ(xi, ξi)← sgn(u∗(xi))min(|u∗(xi)|, |f∗(xi) + sgn(u∗(xi))|si||) + u(0) . noisy activation 7: πli ∼ Bernoulli(pl) . pl controls the variance of the noise AND the prob of skipping a unit 8: h̃li = ψ(xi, ξi) . h̃ l i is a noisy activation candidate
9: φ(hl−1, ξi, πli;wi) = π l ih l−1 i + (1− πli)h̃li . make a HARD decision between h l−1 i and h̃ l i
5 LINEARIZING THE NETWORK
In Section 2, we show that convolving the objective function with a particular kernel can be approximated by adding noise to the activation function. This method may suffer from excessive random exploration when the noise is very large.
We address this issue by bounding the element-wise activation function f(·) with its linear approximation when the variance of the noise is very large, after centering it at the origin. The resulting function f∗(·) is bounded and centered around the origin. Note that centering the sigmoid or hard-sigmoid will make them symmetric with respect to the origin. With a proper choice of the standard deviation σ(h), the noisy activation function becomes a linear function of the input when p is large, as illustrated by Figure 10.
Let u∗(x) = u(x)−u(0), where u(0) is the offset of the function from the origin, and xi the i-th dimension of an affine transformation of the output of the previous layer hl−1: xi = w>i h l−1 + bi. Then:
ψ(xi, ξi;wi) = sgn(u∗(xi))min(|u∗(xi)|, |f∗(xi) + sgn(u∗(xi))|si||) + u(0) (25) The noise is sampled from a Normal distribution with mean 0 and whose standard deviation depends on c:
si ∼ N (0, p c σ(xi))
5.1 LINEARIZING RELU ACTIVATION FUNCTION
We have a simpler form of the equations to linearize ReLU (Nair & Hinton, 2010) activation function when pl →∞. Instead of the complicated Eqn. 23. We can use a simpler equation as in Eqn. 26 to
achieve the linearization of the activation function when we have a very large noise in the activation function:
si = minimum(|xi|, pσ(xi)|ξ|) (26) ψ(xi, ξi,wi) = f(xi)− si (27)
6 MOLLIFYING LSTMS AND GRUS
In a similar vein it is possible to smooth the objective functions of LSTM and GRU networks by starting the optimization procedure with a simpler objective function such as optimizing a word2vec, BoW-LM or CRF objective function at the beginning of training and gradually increasing the difficulty of the optimization by increasing the capacity of the network.
For GRUs we set the update gate to 1t – where t is the time-step index – and reset the gate to 1 if the noise is very large, using Algorithm 1. Similarly for LSTMs, we can set the output gate to 1 and input gate to 1t and forget gate to 1 − 1 t when the noise is very large. The output gate is 1 or close to 1 when the noise is very large. This way the LSTM will behave like a BOW model. In order to achieve this behavior, the activations ψ(xt, ξi) of the gates can be formulated as:
ψ(xlt, ξ) = f(x l t + p lσ(x)|ξ|) By using a particular formulation of σ(x) that constraints it to be in expectation over ξ when pl = 1, we can obtain a function for γ ∈ R within the range of f(·) that is discrete in expectation, but still per sample differentiable:
σ(xlt) = f−1(γ)− xlt
Eξ[|ξ|] (28)
We provide the derivation of Eqn. 28 in Appendix B. The gradient of the Eqn. 28 will be a Monte-Carlo approximation to the gradient of f(xlt).
7 ANNEALING SCHEDULES FOR p
We used a different schedule for each layer of the network, such that the noise in the lower layers will anneal faster. This is similar to the linearly decaying probability of layers in Huang et al. (2016b).
Exponential Decay In our experiments, we focused on using an annealing schedule similar to the inverse sigmoid rule in Bengio et al. (2015) with plt,
plt = 1− e− kvtl tL (29)
with hyper-parameter k ≥ 0 at tth update for the lth layer, where L is the number of layers of the model. We stop annealing when the expected depth pt = ∑L i=1 p l t reaches some threshold δ. In our experiment we set vt to be a moving average of the loss3 of the network, but for some of our experimnets that resulted unstable behaviors in the training and thus we have to fix vt to 1. An advantage of using running average of loss for the vt is because the behavior of the loss/optimization can directly influence the annealing behavior of the network. Because we will have:
lim vt→∞ plt = 1 and, lim vt→0 plt = 0. (30)
This has a desirable property: when the training-loss is high, the noise injected into the system will be large as well. As a result, the model is encouraged to do more exploration, while if the model converges the noise injected into the system by the mollification procedure will be zero.
Square-root Decay The square root decaying of the learning rate is very popular. In our experiments we have tried to decay the plt according to min(pmin, 1 −
√ t
Nepochs ) where t is the
index of the iteration/epoch over the data and Nepochs corresponds to the total number of iterations that the model will do on the dataset. pmin is hyperparameter to define the lower bound on the plt.
Linear Decay We have also experimented with linearly decaying the plt via min(pmin, 1− tNepochs ).
We have compared the plot of different annealing methods described in this paper as in Figure 4.
Furthermore, in our experiments we observe that training with noisy mollifiers can potentially be helpful for the generalization. This can be due to the noise induced to the backpropagation through the noisy mollification, that makes SGD more likely to converge to a flatter-minima (Hochreiter & Schmidhuber, 1997b) because the noise will help it escape from sharper local minima.
8 EXPERIMENTS
In this section we mainly focus on training of difficult to optimize models, in particular deep MLPs with sigmoid or tanh activation functions. The details of the experimental procedure is provided in Appendix C.
8.1 EXPLORATORY STUDY ON MNIST
We train a thin deep neural network on MNIST (LeCun & Cortes, 1998) dataset with 72 hidden layers and 100 hidden units. We train our model with Adam(Kingma & Ba, 2014) optimizer and fix the
3Depending on whether the model overfits or not, this can be a moving average of training or validation loss.
learning rate of all the models to 3e−4. We have used the same learning-rate for all the models in order to factor out the possibility that a model converges faster, due to the fact of using a larger learning rate.
Firstly, in Figure 5, we investigate the effect of using different annealing schedules. Exponential decay converges faster compared to linear decay and square root decay of p. We find it to be very unstable to train our model with linear and square root decay in particular for large c values. Thus we have to use smaller c value (20 instead of 100) to be able to train the model with causing it to diverge.
In Figure 6, we show the effect of using the noisy training procedure that we have introduced by sampling a mask from Bernoulli and Gaussian distributions versu using the deterministic approximation of this noisy procedure which we also use for the test time but during the training as well.
In Figure 7, we compare the results obtained for the model using mollification obtained with or without batch normalization and feed-forward residual networks. The mollified model performs very closely to the MLP trained with residual connections and the batchnorm. However, using residual connections and batch-norm does not seem to improve the results.
We have tried to run experiment with the Monte-Carlo approximation of the mollification which is derived in Appendix A, however when we start with large noise and anneal the noise during the training, the model was very unstable and training was diverging. If we start with small noise and anneal the magnitude of the noise during the training, we could not observe any effect of it on the training.
8.2 DEEP MLP EXPERIMENTS
Deep Parity Experiments Training neural networks on a high-dimensional parity problem can be challenging (Graves, 2016; Kalchbrenner et al., 2015). We experiment on forty dimensional (bits) parity problem with 6-layer MLP using sigmoid activation function. All the models are initialized with Glorot initialization Glorot et al. (2011) and trained with SGD with momentum. We compare an MLP with residual connections using batch normalization and a mollified network with sigmoid activation function. As can be seen in Figure 8, the mollified network converges faster.
Deep Pentomino Pentomino is a toy-image dataset where each image has 3 Pentomino blocks. The task is to predict whether if there is a different shape in the image or not (Gülçehre & Bengio, 2013). The best reported result on this task with MLPs is 68.15% accuracy (Gulcehre et al., 2014). The same model as ours trained without noisy activation function and vanilla residual connections scored 69.5% accuracy, while our mollified version scored 75.15% accuracy after 100 epochs of training on the 80k dataset.
CIFAR10 We experimented with deep convolutional neural networks of 110-layers with residual blocks and residual connections comparing our model against ResNet and Stochastic depth. We adapted the hyperparameters of the Stochastic depth network from Huang et al. (2016a) and we used the same hyperparameters for our algorithm. We report the training and validation curves of the three models in Figure 10 and the best test accuracy obtained early stopping on validation accuracy over 500 epochs in Table 1. Our model achieves better generalization than ResNet. Stochastic depth achieves better generalization, but it might be possible to combine both and obtain better results.
Test PPL
LSTM 119.4 Mollified LSTM 115.7
Table 2: 3-layered LSTM network on word-level language modeling for PTB.
9 LSTM EXPERIMENTS
Predicting the Character Embeddings from Characters Learning the mapping from sequences of characters to the word-embeddings is a difficult problem. Thus one needs to use a highly non-linear function. We trained a word2vec model on Wikipedia with embeddings of size 500 (Mikolov et al., 2014) with a vocabulary of size 374557.
LSTM Language Modeling We evaluate our model on LSTM language modeling. Our baseline model is a 3-layer stacked LSTM without any regularization. We observed that mollified model converges faster and achieves better results. We provide the results for PTB language modeling in Table 2.
10 CONCLUSION
We propose a novel method for training neural networks inspired by an idea of continuation, smoothing techniques and recent advances in non-convex optimization algorithms. The method makes learning easier by starting from a simpler model, solving a well-behaved problem, and gradually transitioning to a more complicated setting. We show improvements on very deep models, difficult to optimize tasks and compare with powerful techniques such as batch-normalization and residual connections. We also show that the mollification procedure improves the generalization performance of the model on two tasks.
Our future work includes testing this method on large-scale language tasks that require long training time, e.g., machine translation and language modeling. Moreover, (Kaiser & Sutskever, 2015) observed that the training of Neural-GPU model can be improved significantly by using gradient noise which can be related to the smoothing of the loss surface, it would be interesting to try mollification on this model to see if the training of Neural GPU can be made easier by using mollification procedure.
ACKNOWLEDGEMENTS
We thank Nicholas Ballas and Misha Denil for the valuable discussions and their feedback. We would like to also thank the developers of Theano 4, for developing such a powerful tool for scientific computing Theano Development Team (2016). We acknowledge the support of the following organizations for research funding and computing support: NSERC, Samsung, Calcul Québec, Compute Canada, the Canada Research Chairs and CIFAR.
A MONTE-CARLO ESTIMATE OF MOLLIFICATION
LK(θ) = (L ∗K)(θ) = ∫ C L(θ − ξ)K(ξ)dξ which can be estimated by a Monte Carlo:
≈ 1 N N∑ i=1 L(θ − ξ(i)), where ξ(i) is a realization of the noise random variable ξ
yielding ∂LK(θ) ∂θ
≈ 1 N N∑ i=1 ∂L(θ − ξ(i)) ∂θ .
(31)
Therefore introducing additive noise to the input of L(θ) is equivalent to mollification.
B DERIVATION OF THE NOISY ACTIVATIONS FOR THE GATING
Assume that zlt = x l t + p l tσ(x)|ξlt| and Eξ[ψ(xlt, ξ)] = t. Thus for all zlt,
Eξ[ψ(x l t, ξ l t)] = Eξ[f(z l t)], (32)
t = Eξ[f(zlt)], assuming f(·) behaves similar to a linear function: (33) Eξ[f(zlt)] ≈ f(Eξ[zlt]) since we use hard-sigmoid for f(·) this will hold. (34)
f−1(t) ≈ Eξ[zlt] (35) (36)
As in Eqn. 32, we can write the expectation of this equation as:
f−1(t) ≈ xlt + pltσ(x)Eξ[ξlt] Corollary, the value that σ(xlt) should take in expectation for p l t = 1 would be:
σ(xlt) ≈ f−1(t)− xlt
Eξ[ξlt]
In our experiments for f(·) we used the hard-sigmoid activation function. We used the following piecewise activation function in order to use it as f−1(x) = 4(x − 0.5). During inference we use the expected value of random variables π and ξ.
C EXPERIMENTAL DETAILS
C.1 MNIST
The weights of the models are initialized with Glorot & Bengio initialization Glorot et al. (2011). We use the learning rate of 4e − 4 along with RMSProp. We initialize ai parameters of mollified activation function by sampling it from a uniform distribution, U[−2, 2]. We used 100 hidden units at each layer with a minibatches of size 500.
C.2 PENTOMINO
We train a 6−layer MLP with sigmoid activation function using SGD and momentum. We used 200 units per layer with sigmoid activation functions. We use a learning rate of 1e− 3.
C.3 CIFAR10
We use the same model with the same hyperparameters for both ResNet, mollified network and the stochastic depth. We borrowed the hyperparameters of the model from Huang et al. (2016a). Our mollified convnet model has residual connections coming from its layer below.
C.4 PARITY
The n-dimensional parity task is the task to figure out whether the sum of n-bits in a binary vector is even or odd. We use SGD with Nesterov momentum and initialize the weight matrices by using Glorot&Bengio initializationGlorot et al. (2011). For all models, we use the learning rate of 1e− 3 and momentum of 0.92. ai is the parameters of mollified activation function are initialized by sampling from uniform distribution, U [−2, 2].
C.5 LSTM LANGUAGE MODELING
We trained 2-layered LSTM language models on PTB word-level. We used the models with the same hyperparameters as in Zaremba & Sutskever (2014). We used the same hyperparameters for both the mollified LSTM language model and the LSTM. We use hard-sigmoid activation function for both the LSTM and mollified LSTM language model. We use hard-sigmoid activation function for the gates of the LSTM.
C.6 PREDICTING THE CHARACTER EMBEDDINGS FROM CHARACTERS
We use 10k of these words as a validation and another 10k word embeddings as test set. We train a bidirectional-LSTM on top of each sequence of characters for each word and on top of the representation of bidirectional LSTM, we use a 5-layered tanh-MLP to predict the word-embedding.
We train our models using RMSProp and momentum with learning rate of 6e− 4 and momentum 0.92. The size of the minibatches, we used is 64. As seen in Figure 9, mollified LSTM network converges faster.

This paper first discusses a general framework for improving optimization of a complicated function using a series of approximations. If the series of approximations are well-behaved compared to the original function, the optimization can in principle be sped up. This is then connected to a particular formulation in which a neural network can behave as a simpler network at high noise levels but regain full capacity as training proceeds and noise lowers.
The idea and motivation of this paper are interesting and sound. As mentioned in my pre-review question, I was wondering about the relationship with shaping methods in RL. I agree with the authors that this paper differs from how shaping typically works (by modifying the problem itself) because in their implementation the architecture is what is "shaped". Nevertheless, the central idea in both cases is to solve a series of optimization problems of increasing difficulty. Therefore, I strongly suggest including a discussion of the differences between shaping, curriculum learning (I'm also not sure how this is different from shaping), and the present approach.
The presentation of the method for neural networks lacks clarity in presentation. Improving this presentation will make this paper much easier to digest. In particular:
- Alg. 1 can not be understood at the point that it is referenced.
- Please explain the steps to Eq. 25 more clearly and connect to steps 1-6 in Alg. 1.
- Define u(x) clearly before defining u*(x)
There are several concerns with the experimental evaluations. There should be a discussion about why doesn't the method work for solving much more challenging network training problems, such as thin and deep networks. Some specific concerns:
- The MLPs trained (Parity and Pentomino) are not very deep at all. An experiment of training thin networks with systematically increasing depth would be a better fit to test this method. Network depth is well known to pose optimization challenges. Instead, it is stated without reference that "Learning the mapping from sequences of characters to the word-embeddings is a difficult problem."
- For cases where the gain is primarily due to the regularization effect, this method should be compared to other weight noise regularization methods.
- I also suggest comparing to highway networks, since there are thematic similarities in Eq. 22, and it is possible that they can automatically anneal their behavior from simple to complex nets during training, considering that they are typically initialized with a bias towards copying behavior.
- For CIFAR-10 experiment, does the mollified model also use Residual connections? If so, why? In either case, why does the mollified net actually train slower than the residual and stochastic depth networks? This is inconsistent with the MLP results.
Overall, the ideas and developments in this paper are promising, but it needs more work to be a clear accept for me.