Although deep learning models have proven effective at solving problems in natural language processing, the mechanism by which they come to their conclusions is often unclear. As a result, these models are generally treated as black boxes, yielding no insight of the underlying learned patterns. In this paper we consider Long Short Term Memory networks (LSTMs) and demonstrate a new approach for tracking the importance of a given input to the LSTM for a given output. By identifying consistently important patterns of words, we are able to distill state of the art LSTMs on sentiment analysis and question answering into a set of representative phrases. This representation is then quantitatively validated by using the extracted phrases to construct a simple, rule-based classifier which approximates the output of the LSTM.
1 INTRODUCTION
Neural network language models, especially recurrent neural networks (RNN), are now standard tools for natural language processing. Amongst other things, they are used for translation Sutskever et al. (2014), language modelling Jozefowicz et al. (2016), and question answering Hewlett et al. (2016). In particular, the Long Short Term Memory (LSTM) Hochreiter & Schmidhuber (1997) architecture has become a basic building block of neural NLP. Although LSTM’s are regularly used in state of the art systems, their operation is not well understood. Besides the basic desire from a scientific viewpoint to clarify their workings, it is often the case that it is important to understand why a machine learning algorithm made a particular choice. Moreover, LSTM’s are computationally intensive compared to discrete models with lookup tables and pattern matching.
In this work, we describe a novel method for visualizing the importance of specific inputs for determining the output of an LSTM. We then demonstrate that, by searching for phrases which are consistently important, the importance scores can be used to extract simple phrase patterns consisting of one to five words from a trained LSTM. The phrase extraction is first done in a general document classification framework on two different sentiment analysis datasets. We then demonstrate that it can also be specialized to more complex models by applying it to WikiMovies, a recently introduced question answer dataset. To concretely validate the extracted patterns, we use them as input to a rules-based classifier which approximates the performance of the original LSTM.
2 RELATED WORK
There are two lines of related work on visualizing LSTMs. First, Hendrik et al. (2016) and Karpathy et al. (2016) analyse the movement of the raw gate activations over a sequence. Karpathy et al. (2016) is able to identify co-ordinates of ct that correspond to semantically meaningful attributes such as whether the text is in quotes and how far along the sentence a word is. However, most of the cell co-ordinates are harder to interpret, and in particular, it is often not obvious from their activations which inputs are important for specific outputs.
∗Work started during an internship at Facebook AI Research
Another approach that has emerged in the literature Alikaniotis et al. (2016) Denil et al. (2015) Bansal et al. (2016) is for each word in the document, looking at the norm of the derivative of the loss function with respect to the embedding parameters for that word. This bridges the gap between high-dimensional cell state and low-dimensional outputs. These techniques are general- they are applicable to visualizing the importance of sets of input coordinates to output coordinates of any differentiable function. In this work, we describe techniques that are designed around the structure of LSTM’s, and show that they can give better results in that setting.
A recent line of work Li et al. (2016) Hewlett et al. (2016) Rajpurkar et al. (2016) Miller et al. (2016) has focused on neural network techniques for extracting answers directly from documents. Previous work had focused on Knowledge Bases (KBs), and techniques to map questions to logical forms suitable for querying them. Although they are effective within their domain, KBs are inevitably incomplete, and are thus an unsatisfactory solution to the general problem of question-answering. Wikipedia, in contrast, has enough information to answer a far broader array of questions, but is not as easy to query. Originally introduced in Miller et al. (2016), the WikiMovies dataset consists of questions about movies paired with Wikipedia articles.
3 WORD IMPORTANCE SCORES IN LSTMS
We present a novel decomposition of the output of an LSTM into a product of factors, where each term in the product can be interpreted as the contribution of a particular word. Thus, we can assign importance scores to words according to their contribution to the LSTM’s prediction
3.1 LONG SHORT TERM MEMORY NETWORKS
Over the past few years, LSTMs have become an important part of neural NLP systems. Given a sequence of word embeddings x1, ..., xT ∈ Rd, an LSTM processes one word at a time, keeping track of cell and state vectors (c1, h1), ..., (cT , hT ) which contain information in the sentence up to word i. ht and ct are computed as a function of xt, ct−1 using the below updates
ft = σ(Wfxt + Vfht−1 + bf ) (1) it = σ(Wixt + Viht−1 + bi) (2) ot = σ(Woxt + Voht−1 + bo) (3) c̃t = tanh(Wcxt + Vcht−1 + bc) (4) ct = ftct−1 + itc̃t (5) ht = ot tanh(ct) (6)
As initial values, we define c0 = h0 = 0. After processing the full sequence, a probability distribution over C classes is specified by p, with
pi = SoftMax(WhT ) = eWihT∑C j=1 e Wjht (7)
where Wi is the i’th row of the matrix W
3.2 DECOMPOSING THE OUTPUT OF A LSTM
We now show that we can decompose the numerator of pi in Equation 7 into a product of factors, and interpret those factors as the contribution of individual words to the predicted probability of class i. Define βi,j = exp (Wi(oT (tanh(cj)− tanh(cj−1))) , (8) so that
exp(WihT ) = exp  T∑ j=1 Wi(oT (tanh(cj)− tanh(cj−1))  = T∏ j=1 βi,j .
As tanh(cj)− tanh(cj−1) can be viewed as the update resulting from word j, so βi,j can be interpreted as the multiplicative contribution to pi by word j.
3.3 AN ADDITIVE DECOMPOSITION OF THE LSTM CELL
We will show below that the βi,j capture some notion of the importance of a word to the LSTM’s output. However, these terms fail to account for how the information contributed by word j is affected by the LSTM’s forget gates between words j and T . Consequently, we empirically found that the importance scores from this approach often yield a considerable amount of false positives. A more nuanced approach is obtained by considering the additive decomposition of cT in equation (9), where each term ej can be interpreted as the contribution to the cell state cT by word j. By iterating the equation ct = ftct−1 + itc̃t, we get that
cT = T∑ i=1 ( T∏ j=i+1 fj)iic̃i = T∑ i=1 ei,T (9)
This suggests a natural definition of an alternative score to the βi,j , corresponding to augmenting the cj terms with products of forget gates to reflect the upstream changes made to cj after initially processing word j.
exp(WihT ) = T∏ j=1 exp
( Wi(oT (tanh(
j∑ k=1 ek,T )− tanh( j−1∑ k=1 ek,T )))
) (10)
= T∏ j=1 exp Wi(oT (tanh(( t∏ k=j+1 fk)cj)− tanh(( t∏ k=j fk)cj−1)))  (11) =
T∏ j=1 γi,j (12)
4 PHRASE EXTRACTION FOR DOCUMENT CLASSIFICATION
We now introduce a technique for using our variable importance scores to extract phrases from a trained LSTM. To do so, we search for phrases which consistently provide a large contribution to the prediction of a particular class relative to other classes. The utility of these patterns is validated by using them as input for a rules based classifier. For simplicity, we focus on the binary classification case.
4.1 PHRASE EXTRACTION
A phrase can be reasonably described as predictive if, whenever it occurs, it causes a document to both be labelled as a particular class, and not be labelled as any other. As our importance scores introduced above correspond to the contribution of particular words to class predictions, they can be used to score potential patterns by looking at a pattern’s average contribution to the prediction of a given class relative to other classes. More precisely, given a collection of D documents {{xi,j}Ndi=1}Dj=1, for a given phrase w1, ..., wk we can compute scores S1, S2 for classes 1 and 2, as well as a combined score S and class C as
S1(w1, ..., wk) = Averagej,b
{∏k l=1 β1,b+l,j |xb+i,j = wi, i = 1, ..., k } Averagej,b {∏k l=1 β2,b+l,j |xb+i,j = wi, i = 1, ..., k
} (13) S2(w1, .., wk) = 1
S1(w1, ..., wk) (14)
S(w1, ..., wk) = max i (Si(w1, ..., wk)) (15)
C(w1, ..., wk) = argmaxi(Si(w1, ..., wk)) (16)
where βi,j,k denotes βi,j applied to document k.
The numerator of S1 denotes the average contribution of the phrase to the prediction of class 1 across all occurrences of the phrase. The denominator denotes the same statistic, but for class 2. Thus, if
S1 is high, then w1, ..., wk is a strong signal for class 1, and likewise for S2. We propose to use S as a score function in order to search for high scoring, representative, phrases which provide insight into the trained LSTM, and C to denote the class corresponding to a phrase.
In practice, the number of phrases is too large to feasibly compute the score of them all. Thus, we approximate a brute force search through a two step procedure. First, we construct a list of candidate phrases by searching for strings of consecutive words j with importance scores βi,j > c for any i and some threshold c; in the experiments below we use c = 1.1. Then, we score and rank the set of candidate phrases, which is much smaller than the set of all phrases.
4.2 RULES BASED CLASSIFIER
The extracted patterns from Section 4.1 can be used to construct a simple, rules-based classifier which approximates the output of the original LSTM. Given a document and a list of patterns sorted by descending score given by S, the classifier sequentially searches for each pattern within the document using simple string matching. Once it finds a pattern, the classifier returns the associated class given by C, ignoring the lower ranked patterns. The resulting classifier is interpretable, and despite its simplicity, retains much of the accuracy of the LSTM used to build it.
5 EXPERIMENTS
We now present the results of our experiments.
5.1 TRAINING DETAILS
We implemented all models in Torch using default hyperparameters for weight initializations. For WikiMovies, all documents and questions were pre-processed so that multiple word entities were concatenated into a single word. For a given question, relevant articles were found by first extracting from the question the rarest entity, then returning a list of Wikipedia articles containing any of those words. We use the pre-defined splits into train, validation and test sets, containing 96k, 10k and 10k questions, respectively. The word and hidden representations of the LSTM were both set to dimension 200 for WikiMovies, 300 and 512 for Yelp, and 300 and 150 for Stanford Sentiment Treebank. All models were optimized using Adam Kingma & Ba (2015) with the default learning rate of 0.001 using early stopping on the validation set. For rule extraction using gradient scores, the product in the reward function is replaced by a sum. In both datasets, we found that normalizing the gradient scores by the largest gradient improved results.
5.2 SENTIMENT ANALYSIS
We first applied the document classification framework to two different sentiment analysis datasets. Originally introduced in Zhang et al. (2015), the Yelp review polarity dataset was obtained from the Yelp Dataset Challenge and has train and test sets of size 560,000 and 38,000. The task is binary prediction for whether the review is positive (four or five stars) or negative (one or two stars). The reviews are relatively long, with an average length of 160.1 words. We also used the binary classification task from the Stanford Sentiment Treebank (SST) Socher et al. (2013), which has less data with train/dev/test sizes of 6920/872/1821, and is done at a sentence level, so has much shorter document lengths.
We report results in Table 1 for seven different models. We report state of the art results from prior work using convolutional neural networks; Kim (2014) for SST and Zhang et al. (2015) for Yelp. We also report our LSTM baselines, which are competitive with state of the art, along with the three different pattern matching models described above. For SST, we also report prior results using bag of words features with Naive Bayes.
The additive cell decomposition pattern equals or outperforms the cell-difference patterns, which handily beat the gradient results. This coincides with our empirical observations regarding the information contained within the importance measures, and validates our introduced measure. The differences between measures become more pronounced in Yelp, as the longer document sizes provide more opportunities for false positives.
Although our pattern matching algorithms underperform other methods, we emphasize that pure performance is not our goal, nor would we expect more from such a simple model. Rather, the fact that our method provides reasonable accuracy is one piece of evidence, in addition to the qualitative evidence given later, that our word importance scores and extracted patterns contain useful information for understanding the actions of a LSTM.
5.3 WIKIMOVIES
Although document classification comprises a sizeable portion of current research in natural language processing, much recent work focuses on more complex problems and models. In this section, we examine WikiMovies, a recently introduced question answer dataset, and show that with some simple modifications our approach can be adapted to this problem.
5.3.1 DATASET
WikiMovies is a dataset consisting of more than 100,000 questions about movies, paired with relevant Wikipedia articles. It was constructed using the pre-existing dataset MovieLens, paired with templates extracted from the SimpleQuestions dataset Bordes et al. (2015), a open-domain question answering dataset based on Freebase. They then selected a set of Wikipedia articles about movies by identifying a set of movies from OMDb that had an associated article by title match, and kept the title and first section for each article.
For a given question, the task is to read through the relevant articles and extract the answer, which is contained somewhere within the text. The dataset also provides a list of 43k entities containing all possible answers.
5.3.2 LSTMS FOR WIKIMOVIES
We propose a simplified version of recent work Li et al. (2016). Given a pair of question xq1, ..., x q N and document xd1, ..., x d T , we first compute an embedding for the question using a LSTM. Then, for each word t in the document, we augment the word embedding xt with the computed question embedding. This is equivalent to adding an additional term which is linear in the question embedding into the gate equations 3-6, allowing the patterns an LSTM absorbs to be directly conditioned upon the question at hand.
hqt = LSTM(x q t ) (17)
ht = LSTM(xdt ‖h q N ) (18)
Having run the above model over the document while conditioning on a question, we are given contextual representations h1, ..., hT of the words in the document. For each entity t in the document
we use pt to conduct a binary prediction for whether or not the entity is the answer. At test time, we return the entity with the highest probability as the answer.
pt = SoftMax(Wht) (19)
5.3.3 PHRASE EXTRACTION
We now introduce some simple modifications that were useful in adapting our pattern extraction framework to this specific task. First, in order to define the set of classifications problems to search over, we treat each entity t within each document as a separate binary classification task with corresponding predictor pt. Given this set of classification problems, rather than search over the space of all possible phrases, we restrict ourselves to those ending at the entity in question. We also distinguish patterns starting at the beginning of the document with those that do not and introduce an entity character into our pattern vocabulary, which can be matched by any entity. Template examples can be seen below, in Table 4. Once we have extracted a list of patterns, in the rules-based classifier we only search for positive examples, and return as the answer the entity matched to the highest ranked positive pattern.
5.3.4 RESULTS
We report results on six different models in Tables 2 and 3. We show the results from Miller et al. (2016), which fit a key-value memory network (KV-MemNN) on representations from information extraction (IE) and raw text (Doc). Next, we report the results of the LSTM described in Section 5.3.2. Finally, we show the results of using three variants of the pattern matching algorithm described in Section 5.3.3: using patterns extracted using the additive decomposition (cell decomposition), difference in cells approaches (cell-difference) and gradient importance scores (gradient), as discussed in Section 2. Performance is reported using the accuracy of the top hit over all possible answers (all entities), i.e. the hits@1 metric.
As shown in Table 2, our LSTM model surpasses the prior state of the art by nearly 4%. Moreover, our automatic pattern matching model approximates the LSTM with less than 6% error, which is surprisingly small for such a simple model, and falls within 2% of the prior state of the art. Similarly to sentiment analysis, we observe a clear ordering of the results across question categories, with our cell decomposition scores providing the best performance, followed by the cell difference and gradient scores.
6 DISCUSSION
6.1 LEARNED PATTERNS
We present extracted patterns for both sentiment tasks, and some WikiMovies question categories in Table 4. These patterns are qualitatively sensible, providing further validation of our approach. The increased size of the Yelp dataset allowed for longer phrases to be extracted relative to SST.
Category Top Patterns Yelp Polarity Positive definitely come back again., love love love this
place, great food and great service., highly recommended!, will definitely be coming back, overall great experience, love everything about, hidden gem.
Yelp Polarity Negative worst customer service ever, horrible horrible horrible, won’t be back, disappointed in this place, never go back there, not worth the money, not recommend this place SST Positive riveting documentary, is a real charmer, funny and touching, well worth your time, journey of the heart, emotional wallop, pleasure to watch, the whole family, cast is uniformly superb, comes from the heart, best films of the year, surprisingly funny, deeply satisfying SST Negative pretentious mess ..., plain bad, worst film of the year, disappointingly generic, fart jokes, banal dialogue, poorly executed, waste of time, a weak script, dullard, how bad it is, platitudes, never catches fire, tries too hard to be, bad acting, untalented artistes, derivative horror film, lackluster WikiMovies movie to writer film adaptation of Charles Dickens’, film adapted from ENT, by journalist ENT, written by ENT WikiMovies movie to actor western film starring ENT, starring Ben Affleck, . The movie stars ENT, that stars ENT WikiMovies movie to language is a 2014 french, icelandic, finnish, russian, danish, bengali, dutch, original german, zulu,czech, estonian, mandarin, filipino, hungarian
Table 4: Selected top patterns using cell decomposition scores, ENT denotes an entity placeholder
6.2 APPROXIMATION ERROR BETWEEN LSTM AND PATTERN MATCHING
Although our approach is able to extract sensible patterns and achieve reasonable performance, there is still an approximation gap between our algorithm and the LSTM. In Table 5 we present some examples of instances where the LSTM was able to correctly classify a sentence, and our algorithm was not, along with the pattern used by our algorithm. At first glance, the extracted patterns are sensible, as ”gets the job done” or ”witty dialogue” are phrases you’d expect to see in a positive review of a movie. However, when placed in the broader context of these particular reviews, they cease to be predictive. This demonstrates that, although our work is useful as a firstorder approximation, there are still additional relationships that an LSTM is able to learn from data.
6.3 COMPARISON BETWEEN WORD IMPORTANCE MEASURES
While the prediction accuracy of our rules-based classifier provides quantitative validation of the relative merits of our visualizations, the qualitative differences are also insightful. In Table 6, we provide a side-by-side comparison between the different measures. As discussed before, the difference in cells technique fails to account for how the updates resulting from word j are affected by the LSTM’s forget gates between when the word is initially processed and the answer. Consequently, we empirically found that without the interluding forget gates to dampen cell movements, the variable importance scores were far noisier than in additive cell decomposition approach. Under the additive cell decomposition, it identifies the phrase ’it stars’, as well as the actor’s name Aqib Khan as being important, a sensible conclusion. Moreover, the vast majority of words are labelled with an importance score of 1, corresponding to irrelevant. On the other hand, the difference in cells approach yields widely changing importance scores, which are challenging to interpret. In terms of noise, the gradient measures seem to lie somewhere in the middle. These patterns are broadly consistent with what we have observed, and provide qualitative validation of our metrics.
7 CONCLUSION
In this paper, we introduced a novel method for visualizing the importance of specific inputs in determining the output of an LSTM. By searching for phrases which consistently provide large contributions, we are able to distill trained, state of the art, LSTMs into an ordered set of representative phrases. We quantitatively validate the extracted phrases through their performance in a simple, rules-based classifier. Results are shown in a general document classification framework, then specialized to a more complex, recently introduced, question answer dataset. Our introduced measures provide superior predictive ability and cleaner visualizations relative to prior work. We believe that this represents an exciting new paradigm for analysing the behaviour of LSTM’s.
Additive cell decomposition Difference in cell values Gradient
west is west is a 2010 british comedy - drama film , which is a sequel to the 1999 comedy ”
east is east ” . it stars aqib khan
west is west is a 2010 british comedy -
drama film , which is a sequel to the 1999 comedy ” east is east ” . it starsaqib khan
west is west is a 2010 british comedy - drama film , which is a sequel to the 1999 comedy ” east is east ”. itstars aqib khan
Table 6: Comparison of importance scores acquired by three different approaches, conditioning on the question ”the film west is west starred which actors?”. Bigger and darker means more important.
ACKNOWLEDGEMENTS
This research was partially funded by Air Force grant FA9550-14-1-0016. It was also supported by the Center for Science of Information (CSoI), an US NSF Science and Technology Center, under grant agreement CCF-0939370.
8 APPENDIX - HEAT MAPS
We provide an example heat map using the cell decomposition metric for each class in both sentiment analysis datasets, and selected WikiMovie question categories
Dataset Category Heat Map
Yelp Polarity Positive we went here twice for breakfast . had the bananas foster waffles with fresh whipped cream , they were amazing ! ! perfect seat out side on the terrace
Yelp Polarity Negative call me spoiled ...this sushi is gross and the orange chicken , well it was so thin i don ’t think it had chicken in it. gosomewhereelse
Stanford Sentiment Positive Whether or not you ’re enlightened by any of Derrida ’s lectures on “ the other ” and “ the self ,
” Derrida is an undeniablyfascinatingandplayfulfellow
Stanford Sentiment Negative ... begins with promise , but runs aground after being snared in its own tangled plot
Pattern Question Heat Map
Movie to Year What was the release year of another 48 hours?
another 48 hrs is a 1990
Movie to Writer Which person wrote the movie last of the dogmen? last of the dogmen is a 1995 western adventure film written and directed by tab murphy
Movie to Actor Who acted in the movie thunderbolt? thunderbolt ( ) ( ” piklik foh ” ) is a 1995 hong kong action filmstarring jackie chan
Movie to Director Who directed bloody bloody bible camp? bloody bloody bible cam p is a 2012 american horror - comedy /s platter film . the film
was directed by vito trabucco
Movie to Genre What genre is trespass in? trespassisa 1992 action Movie to Votes How would people rate the pool? though filmed in hindi , a language smith didn ’t know , the film earned
good∗ Movie to Rating How popular was les miserables? les mis rables is a 1935 american drama film starring fredric march and charles laughton
based upon thefamous Movie to Tags Describe rough magic?
rough magic is a 1995 comedy film directed by clare peploe and starring bridget fonda , russell crowe
Movie to Language What is the main language in fate?
fate ( ) is a 2001 turkish

EDIT: the revisions made to this paper are very thorough and address many of my concerns, and the paper is also easier to understand. i recommend the latest version of this paper for acceptance and have increased my score.
This paper presents a way of interpreting LSTM models, which are notable for their opaqueness. In particular, the authors propose decomposing the LSTM's predictions for a QA task into importance scores for words, which are then used to generate patterns that are used to find answers with a simple matching algorithm. On the WikiMovies dataset, the extracted pattern matching method achieves accuracies competitive with a normal LSTM, which shows the power of the proposed approach.
I really like the motivation of the paper, as interpreting LSTMs is definitely still a work-in-progress, and the high performance of the pattern matching was surprising. However, several details of the pattern extraction process are not very clear, and  the evaluation is conducted on a very specific task, where predictions are made at every word. As such, I recommend the paper in its current form as a weak accept but hope that the authors clarify their approach, as I believe the proposed method is potentially useful for NLP researchers.
Comments:
- Please introduce in more detail the specific QA tasks you are applying your models on before section 3.3, as it's not clear at that point that the answer is an entity within the document.
- 3.3: is the softmax predicting a 0/1 value (e.g., is this word the answer or not?)
- 3.3: what are the P and Q vectors? do you just mean that you are transforming the hidden state into a 2-dimensional vector for binary prediction?
- how does performance of the pattern matching change with different cutoff constant values?
- 5.2: are there questions whose answers are not entities?
- how could the proposed approach be used when predictions aren't made at every word? is there any extension for, say, sentence-level sentiment classification?


DIALOGUE LEARNING WITH HUMAN-IN-THE-LOOP
An important aspect of developing conversational agents is to give a bot the ability to improve through communicating with humans and to learn from the mistakes that it makes. Most research has focused on learning from fixed training sets of labeled data rather than interacting with a dialogue partner in an online fashion. In this paper we explore this direction in a reinforcement learning setting where the bot improves its question-answering ability from feedback a teacher gives following its generated responses. We build a simulator that tests various aspects of such learning in a synthetic environment, and introduce models that work in this regime. Finally, real experiments with Mechanical Turk validate the approach.
An important aspect of developing conversational agents is to give a bot the ability to improve through communicating with humans and to learn from the mistakes that it makes. Most research has focused on learning from fixed training sets of labeled data rather than interacting with a dialogue partner in an online fashion. In this paper we explore this direction in a reinforcement learning setting where the bot improves its question-answering ability from feedback a teacher gives following its generated responses. We build a simulator that tests various aspects of such learning in a synthetic environment, and introduce models that work in this regime. Finally, real experiments with Mechanical Turk validate the approach.
1 INTRODUCTION
A good conversational agent (which we sometimes refer to as a learner or bot1) should have the ability to learn from the online feedback from a teacher: adapting its model when making mistakes and reinforcing the model when the teacher’s feedback is positive. This is particularly important in the situation where the bot is initially trained in a supervised way on a fixed synthetic, domainspecific or pre-built dataset before release, but will be exposed to a different environment after release (e.g., more diverse natural language utterance usage when talking with real humans, different distributions, special cases, etc.). Most recent research has focused on training a bot from fixed training sets of labeled data but seldom on how the bot can improve through online interaction with humans. Human (rather than machine) language learning happens during communication (Bassiri, 2011; Werts et al., 1995), and not from labeled datasets, hence making this an important subject to study.
In this work, we explore this direction by training a bot through interaction with teachers in an online fashion. The task is formalized under the general framework of reinforcement learning via the teacher’s (dialogue partner’s) feedback to the dialogue actions from the bot. The dialogue takes place in the context of question-answering tasks and the bot has to, given either a short story or a set of facts, answer a set of questions from the teacher. We consider two types of feedback: explicit numerical rewards as in conventional reinforcement learning, and textual feedback which is more natural in human dialogue, following (Weston, 2016). We consider two online training scenarios: (i) where the task is built with a dialogue simulator allowing for easy analysis and repeatability of experiments; and (ii) where the teachers are real humans using Amazon Mechanical Turk.
We explore important issues involved in online learning such as how a bot can be most efficiently trained using a minimal amount of teacher’s feedback, how a bot can harness different types of feedback signal, how to avoid pitfalls such as instability during online learing with different types of feedback via data balancing and exploration, and how to make learning with real humans feasible via data batching. Our findings indicate that it is feasible to build a pipeline that starts from a model trained with fixed data and then learns from interactions with humans to improve itself.
1In this paper, we refer to a learner (either a human or a bot/dialogue agent which is a machine learning algorithm) as the student, and their more knowledgeable dialogue partner as the teacher.
2 RELATED WORK
Reinforcement learning has been widely applied to dialogue, especially in slot filling to solve domain-specific tasks (Walker, 2000; Schatzmann et al., 2006; Singh et al., 2000; 2002). Efforts include Markov Decision Processes (MDPs) (Levin et al., 1997; 2000; Walker et al., 2003; Pieraccini et al., 2009), POMDP models (Young et al., 2010; 2013; Gašic et al., 2013; 2014) and policy learning (Su et al., 2016). Such a line of research focuses mainly on frames with slots to fill, where the bot will use reinforcement learning to model a state transition pattern, generating dialogue utterances to prompt the appropriate user responses to put in the desired slots. This goal is different from ours, where we study end-to-end learning systems and also consider non-reward based setups via textual feedback.
Our work is related to the line of research that focuses on supervised learning for question answering (QA) from dialogues (Dodge et al., 2015; Weston, 2016), either given a database of knowledge (Bordes et al., 2015; Miller et al., 2016) or short texts (Weston et al., 2015; Hermann et al., 2015; Rajpurkar et al., 2016). In our work, the discourse includes the statements made in the past, the question and answer, and crucially the response from the teacher. The latter is what makes the setting different from the standard QA setting, i.e. we use methods that leverage this response also, not just answering questions. Further, QA works only consider fixed datasets with gold annotations, i.e. they do not consider a reinforcement learning setting.
Our work is closely related to a recent work from Weston (2016) that learns through conducting conversations where supervision is given naturally in the response during the conversation. That work introduced the use of forward prediction that learns by predicting the teacher’s feedback, in addition to using reward-based learning of correct answers. However, two important issues were not addressed: (i) it did not use a reinforcement learning setting, but instead used pre-built datasets with fixed policies given in advance; and (ii) experiments used only simulated and no real language data. Hence, models that can learn policies from real online communication were not investigated. To make the differences with our work clear, we will now detail these points further.
The experiments in (Weston, 2016) involve constructing pre-built fixed datasets, rather than training the learner within a simulator, as in our work. Pre-built datasets can only be made by fixing a prior in advance. They achieve this by choosing an omniscient (but deliberately imperfect) labeler that gets πacc examples always correct (the paper looked at values 50%, 10% and 1%). Again, this was not learned, and was fixed to generate the datasets. Note that the paper refers to these answers as coming from “the learner” (which should be the model), but since the policy is fixed it actually does not depend on the model. In a realistic setting one does not have access to an omniscient labeler, one has to learn a policy completely from scratch, online, starting with a random policy, so their setting was not practically viable. In our work, when policy training is viewed as batch learning over iterations of the dataset, updating the policy on each iteration, (Weston, 2016) can be viewed as training only one iteration, whereas we perform multiple iterations. This is explained further in Sections 4.2 and 5.1. We show in our experiments that performance improves over the iterations, i.e. it is better than the first iteration. We show that such online learning works for both rewardbased numerical feedback and for forward prediction methods using textual feedback (under certain conditions which are detailed). This is a key contribution of our work.
Finally, (Weston, 2016) only conducted experiments on synthetic or templated language, and not real language, especially the feedback from the teacher was scripted. While we believe that synthetic datasets are very important for developing understanding (hence we develop a simulator and conduct experiments also with synthetic data), for a new method to gain traction it must be shown to work on real data. We hence employ Mechanical Turk to collect real language data for the questions and importantly for the teacher feedback and construct experiments in this real setting.
3 DATASET AND TASKS
We begin by describing the data setup we use. In our first set of experiments we build a simulator as a testbed for learning algorithms. In our second set of experiments we use Mechanical Turk to provide real human teachers giving feedback.
3.1 SIMULATOR
The simulator adapts two existing fixed datasets to our online setting. Following Weston (2016), we use (i) the single supporting fact problem from the bAbI datasets (Weston et al., 2015) which consists of 1000 short stories from a simulated world interspersed with questions; and (ii) the WikiMovies dataset (Weston et al., 2015) which consists of roughly 100k (templated) questions over 75k entities based on questions with answers in the open movie database (OMDb). Each dialogue takes place between a teacher, scripted by the simulation, and a bot. The communication protocol is as follows: (1) the teacher first asks a question from the fixed set of questions existing in the dataset, (2) the bot answers the question, and finally (3) the teacher gives feedback on the bot’s answer.
We follow the paradigm defined in (Weston, 2016) where the teacher’s feedback takes the form of either textual feedback, a numerical reward, or both, depending on the task. For each dataset, there are ten tasks, which are further described in Sec. A and illustrated in Figure 5 of the appendix. We also refer the readers to (Weston, 2016) for more detailed descriptions and the motivation behind these tasks. In the main text of this paper we only consider Task 6 (“partial feedback”): the teacher replies with positive textual feedback (6 possible templates) when the bot answers correctly, and positive reward is given only 50% of the time. When the bot is wrong, the teacher gives textual feedback containing the answer. Descriptions and experiments on the other tasks are detailed in the appendix. Example dialogues are given in Figure 1.
The difference between our simulation and the original fixed tasks of Weston (2016) is that models are trained on-the-fly. After receiving feedback and/or rewards, we update the model (policy) and then deploy it to collect teacher’s feedback in the next episode or batch. This means the model’s policy affects the data which is used to train it, which was not the case in the previous work.
3.2 MECHANICAL TURK EXPERIMENTS
Finally, we extended WikiMovies using Mechanical Turk so that real human teachers are giving feedback rather than using a simulation. As both the questions and feedback are templated in the simulation, they are now both replaced with natural human utterances. Rather than having a set of simulated tasks, we have only one task, and we gave instructions to the teachers that they could give feedback as they see fit. The exact instructions given to the Turkers is given in Appendix B. In general, each independent response contains feedback like (i) positive or negative sentences; or (ii) a phrase containing the answer or (iii) a hint, which are similar to setups defined in the simulator. However, some human responses cannot be so easily categorized, and the lexical variability is much larger in human responses. Some examples of the collected data are given in Figure 2.
4 METHODS
4.1 MODEL ARCHITECTURE
In our experiments, we used variants of the End-to-End Memory Network (MemN2N) model (Sukhbaatar et al., 2015) as our underlying architecture for learning from dialogue.
The input to MemN2N is the last utterance of the dialogue history x as well as a set of memories (context) C=c1, c2, ..., cN . The memory C encodes both short-term memory, e.g., dialogue histories between the bot and the teacher, and long-term memories, e.g., the knowledge base facts that the bot has access to. Given the input x and C, the goal is to produce an output/label a.
In the first step, the query x is transformed to a vector representation u0 by summing up its constituent word embeddings: u0 = Ax. The input x is a bag-of-words vector and A is the d× V word embedding matrix where d denotes the emebbding dimension and V denotes the vocabulary size. Each memory ci is similarly transformed to a vector mi. The model will read information from the memory by comparing input representation u0 with memory vectors mi using softmax weights:
o1 = ∑ i p1imi p 1 i = softmax(u T 0mi) (1)
This process selects memories relevant to the last utterance x, i.e., the memories with large values of p1i . The returned memory vector o1 is the weighted sum of memory vectors. This process can be repeated to query the memory N times (so called “hops”) by adding on to the original input, u1 = o1 + u0, or to the previous state, un = on + un−1, and then using un to query the memories again.
In the end, uN is input to a softmax function for the final prediction:
a = softmax(uTNy1, u T Ny2, ..., u T NyL) (2)
where y1, . . . , yL denote the set of candidate answers. If the answer is a word, yi is the corresponding word embedding. If the answer is a sentence, yi is the embedding for the sentence achieved in the same way that we obtain embeddings for query x and memory C.
The standard way MemN2N is trained is via a cross entropy criterion on known input-output pairs, which we refer to as supervised or imitation learning. As our work is in a reinforcement learning setup where our model must make predictions to learn, this procedure will not work, so we instead consider reinforcement learning algorithms which we describe next.
4.2 REINFORCEMENT LEARNING
In this section, we present the algorithms we used to train MemN2N in an online fashion. Our learning setup can be cast as a particular form of Reinforcement Learning. The policy is implemented by the MemN2N model. The state is the dialogue history. The action space corresponds to the set of answers the MemN2N has to choose from to answer the teacher’s question. In our setting, the policy chooses only one action for each episode. The reward is either 1 (a reward from the teacher when the bot answers correctly) or 0 otherwise. Note that in our experiments, a reward equal to 0 might mean that the answer is incorrect or that the positive reward is simply missing. The overall setup is closest to standard contextual bandits, except that the reward is binary.
When working with real human dialogues, e.g. collecting data via Mechanical Turk, it is easier to set up a task whereby a bot is deployed to respond to a large batch of utterances, as opposed to a single one. The latter would be more difficult to manage and scale up since it would require some form of synchronization between the model replicas interacting with each human.
This is comparable to the real world situation where a teacher can either ask a student a single question and give feedback right away, or set up a test that contains many questions and grade all of them at once. Only after the learner completes all questions, it can hear feedback from the teacher.
We use batch size to refer to how many dialogue episodes the current model is used to collect feedback before updating its parameters. In the Reinforcement Learning literature, batch size is related to off-policy learning since the MemN2N policy is trained using episodes collected with a stale version of the model. Our experiments show that our model and base algorithms are very robust to the choice of batch size, alleviating the need for correction terms in the learning algorithm (Bottou et al., 2013).
We consider two strategies: (i) online batch size, whereby the target policy is updated after doing a single pass over each batch (a batch size of 1 reverts to the usual on-policy online learning); and (ii) dataset-sized batch, whereby training is continued to convergence on the batch which is the size of the dataset, and then the target policy is updated with the new model, and a new batch is drawn and the procedure iterates. These strategies can be applied to all the methods we use, described below.
Next, we discuss the learning algorithms we considered in this work.
4.2.1 REWARD-BASED IMITATION (RBI)
The simplest algorithm we first consider is the one employed in Weston (2016). RBI relies on positive rewards provided by the teacher. It is trained to imitate the correct behavior of the learner, i.e., learning to predict the correct answers (with reward 1) at training time and disregarding the other ones. This is implemented by using a MemN2N that maps a dialogue input to a prediction, i.e. using the cross entropy criterion on the positively rewarded subset of the data.
In order to make this work in the online setting which requires exploration to find the correct answer, we employ an -greedy strategy: the learner makes a prediction using its own model (the answer assigned the highest probability) with probability 1 − , otherwise it picks a random answer with probability . The teacher will then give a reward of +1 if the answer is correct, otherwise 0. The bot will then learn to imitate the correct answers: predicting the correct answers while ignoring the incorrect ones.
4.2.2 REINFORCE
The second algorithm we use is the REINFORCE algorithm (Williams, 1992), which maximizes the expected cumulative reward of the episode, in our case the expected reward provided by the teacher. The expectation is approximated by sampling an answer from the model distribution. Let a denote the answer that the learner gives, p(a) denote the probability that current model assigns to a, r denote the teacher’s reward, and J(θ) denote the expectation of the reward. We have:
∇J(θ) ≈ ∇ log p(a)[r − b] (3)
where b is the baseline value, which is estimated using a linear regression model that takes as input the output of the memory network after the last hop, and outputs a scalar b denoting the estimation of the future reward. The baseline model is trained by minimizing the mean squared loss between the estimated reward b and actual reward r, ||r − b||2. We refer the readers to (Ranzato et al., 2015; Zaremba & Sutskever, 2015) for more details. The baseline estimator model is independent from the policy model, and its error is not backpropagated through the policy model.
The major difference between RBI and REINFORCE is that (i) the learner only tries to imitate correct behavior in RBI while in REINFORCE it also leverages the incorrect behavior, and (ii) the learner explores using an -greedy strategy in RBI while in REINFORCE it uses the distribution over actions produced by the model itself.
4.2.3 FORWARD PREDICTION (FP)
FP (Weston, 2016) handles the situation where a numerical reward for a bot’s answer is not available, meaning that there are no +1 or 0 labels available after a student’s utterance. Instead, the model assumes the teacher gives textual feedback t to the bot’s answer, taking the form of a dialogue utterance, and the model tries to predict this instead. Suppose that x denotes the teacher’s question andC=c1, c2, ..., cN denotes the dialogue history as before. In FP, the model first maps the teacher’s initial question x and dialogue history C to a vector representation u using a memory network with multiple hops. Then the model will perform another hop of attention over all possible student’s answers in A, with an additional part that incorporates the information of which candidate (i.e., a) was actually selected in the dialogue:
pâ = softmax(u T yâ) o = ∑ â∈A pâ(yâ + β · 1[â = a]) (4)
where yâ denotes the vector representation for the student’s answer candidate â. β is a (learned) d-dimensional vector to signify the actual action a that the student chooses. o is then combined with u to predict the teacher’s feedback t using a softmax:
u1 = o+ u t = softmax(u T 1 xr1 , u T 1 xr2 , ..., u T 1 xrN ) (5)
where xri denotes the embedding for the i th response. In the online setting, the teacher will give textual feedback, and the learner needs to update its model using the feedback. It was shown in Weston (2016) that in an off-line setting this procedure can work either on its own, or in conjunction with a method that uses numerical rewards as well for improved performance. In the online setting, we consider two simple extensions:
• -greedy exploration: with probability the student will give a random answer, and with probability 1− it will give the answer that its model assigns the largest probability. This method enables the model to explore the space of actions and to potentially discover correct answers.
• data balancing: cluster the set of teacher responses t and then balance training across the clusters equally.2 This is a type of experience replay (Mnih et al., 2013) but sampling with an evened distribution. Balancing stops part of the distribution dominating the learning. For example, if the model is not exposed to sufficient positive and negative feedback, and one class overly dominates, the learning process degenerates to a model that always predicts the same output regardless of its input.
5 EXPERIMENTS
Experiments are first conducted using our simulator, and then using Amazon Mechanical Turk with real human subjects taking the role of the teacher3.
5.1 SIMULATOR
Online Experiments In our first experiments, we considered both the bAbI and WikiMovies tasks and varied batch size, random exploration rate , and type of model. Figure 3 and Figure 4 shows (Task 6) results on bAbI and WikiMovies. Other tasks yield similar conclusions and are reported in the appendix.
Overall, we obtain the following conclusions:
• In general RBI and FP do work in a reinforcement learning setting, but can perform better with random exploration.
• In particular RBI can fail without exploration. RBI needs random noise for exploring labels otherwise it can get stuck predicting a subset of labels and fail.
2In the simulated data, because the responses are templates, this can be implemented by first randomly sampling the response, and then randomly sampling a story with that response; we keep the history of all stories seen from which we sample. For real data slightly more sophisticated clustering should be used.
3 Code and data are available at https://github.com/facebook/MemNN/tree/master/HITL.
• REINFORCE obtains similar performance to RBI with optimal . • FP with balancing or with exploration via both outperform FP alone. • For both RBI and FP, performance is largely independent of online batch size.
Dataset Batch Size Experiments Given that larger online batch sizes appear to work well, and that this could be important in a real-world data collection setup where the same model is deployed to gather a large amount of feedback from humans, we conducted further experiments where the batch size is exactly equal to the dataset size and for each batch training is completed to convergence.
After the model has been trained on the dataset, it is deployed to collect a new dataset of questions and answers, and the process is repeated. Table 1 reports test error at each iteration of training, using the bAbI Task 6 as the case study (see the appendix for results on other tasks). The following conclusions can be made for this setting:
• RBI improves in performance as we iterate. Unlike in the online case, RBI does not need random exploration. We believe this is because the first batch, which is collected with a randomly initialized model, contains enough variety of examples with positive rewards that the model does not get stuck predicting a subset of labels.
• FP is not stable in this setting. This is because once the model gets very good at making predictions (at the third iteration), it is not exposed to a sufficient number of negative responses anymore. From that point on, learning degenerates and performance drops as the model always predicts the same responses. At the next iteration, it will recover again since it has a more balanced training set, but then it will collapse again in an oscillating behavior.
• FP does work if extended with balancing or random exploration with sufficiently large .
• RBI+FP also works well and helps with the instability of FP, alleviating the need for random exploration and data balancing.
Overall, our simulation results indicate that while a bot can be effectively trained fully online from bot-teacher interactions, collecting real dialogue data in batches (which is easier to collect and iterate experiments over) is also a viable approach. We hence pursue the latter approach in our next set of experiments.
Relation to experiments in Weston (2016) As described in detail in Section 2 the datasets we use in our experiments were introduced in (Weston et al., 2015). However, that work involved constructing pre-built fixed policies (and hence, datasets), rather than training the learner in a reinforcement/interactive learning using a simulator, as in our work. They achieved this by choosing an omniscient (but deliberately imperfect) labeler that gets πacc examples always correct (the paper looked at values 1%, 10% and 50%). In a realistic setting one does not have access to an omniscient labeler, one has to learn a policy completely from scratch, online, starting with a random policy, as we do here. Nevertheless, it is possible to compare our learnt policies to those results because we use the same train/valid/test splits.
The clearest comparison comparison is via Table 1, where the policy is learnt using batch iterations of the dataset, updating the policy on each iteration. Weston et al. (2015) can be viewed as training only one iteration, with a pre-built policy, as explained above, where 59%, 81% and 99% accuracy was obtained for RBI for πacc with 1%, 10% and 50% respectively4. While πacc of 50% is good enough to solve the task, lower values are not. In this work a random policy begins with 74% accuracy on the first iteration, but importantly on each iteration the policy is updated and improves, with values of 87%, 90% on iterations 2 and 3 respectively, and 98% on iteration 6. This is a key differentiator to the work of (Weston et al., 2015) where such improvement was not shown. We show that such online learning works for both reward-based numerical feedback and for forward prediction methods using textual feedback (as long as balancing or random exploration is performed sufficiently). The final performance outperforms most values of πacc from Weston et al. (2015) unless π is so large that the task is already solved. This is a key contribution of our work.
Similar conclusions can be made for Figures 3 and 4. Despite our initial random policy starting at close to 0% accuracy, if random exploration ≥ 0.2 is employed then after a number of epochs the performance is better than most values of πacc from Weston et al. (2015), e.g. compare the accuracies given in the previous paragraph (59%, 81% and 99%) to Figure 3, top left.
5.2 HUMAN FEEDBACK
We employed Turkers to both ask questions and then give textual feedback on the bot’s answers, as described in Section 3.2. Our experimental protocol was as follows. We first trained a MemN2N using supervised (i.e., imitation) learning on a training set of 1000 questions produced by Turkers and using the known correct answers provided by the original dataset (and no textual feedback). Next, using the trained policy, we collected textual feedback for the responses of the bot for an additional 10,000 questions. Examples from the collected dataset are given in Figure 2. Given this dataset, we compare various models: RBI, FP and FP+RBI. As we know the correct answers to the additional questions, we can assign a positive reward to questions the bot got correct. We hence measure the impact of the sparseness of this reward signal, where a fraction r of additional examples have rewards. The models are tested on a test set of ∼8,000 questions (produced by Turkers), and hyperparameters are tuned on a similarly sized validation set. Note this is a harder task than the WikiMovies task in the simulator due to the use natural language from Turkers, hence lower test performance is expected.
4Note, this is not the same as a randomly initialized neural network policy, because due to the synthetic construction with an omniscient labeler the labels will be balanced. In our work, we learn the policy from randomly initialized weights which are updated as we learn the policy.
Results are given in Table 2. They indicate that both RBI and FP are useful. When rewards are sparse, FP still works via the textual feedback while RBI can only use the initial 1000 examples when r = 0. As FP does not use numericalrewards at all, it is invariant to the parameter r. The combination of FP and RBI outperforms either alone.
We also conducted additional experiments comparing with (i) synthetic feedback and (ii) the fully supervised case which are given in Appendix C.1. They show that the results with human feedback are competitive with these approaches.
6 CONCLUSION
We studied dialogue learning of end-to-end models using textual feedback and numerical rewards. Both fully online and iterative batch settings are viable approaches to policy learning, as long as possible instabilities in the learning algorithms are taken into account. Secondly, we showed for the first time that the recently introduced FP method can work in both an online setting and on real human feedback. Overall, our results indicate that it is feasible to build a practical pipeline that starts with a model trained on an initial fixed dataset, which then learns from interactions with humans in a (semi-)online fashion to improve itself. Future research should work towards doing this in a never-ending learning setup.
A FURTHER SIMULATOR TASK DETAILS
The tasks in Weston (2016) were specifically: - Task 1: The teacher tells the student exactly what they should have said (supervised baseline). - Task 2: The teacher replies with positive textual feedback and reward, or negative textual feedback. - Task 3: The teacher gives textual feedback containing the answer when the bot is wrong. - Task 4: The teacher provides a hint by providing the class of the correct answer, e.g., “No it’s a movie” for the question “which movie did Forest Gump star in?”. - Task 5: The teacher provides a reason why the student’s answer is wrong by pointing out the relevant supporting fact from the knowledge base. - Task 6: The teacher gives positive reward only 50% of the time. - Task 7: Rewards are missing and the teacher only gives natural language feedback. - Task 8: Combines Tasks 1 and 2 to see whether a learner can learn successfully from both forms of supervision at once. - Task 9: The bot asks questions of the teacher about what it has done wrong. - Task 10: The bot will receive a hint rather than the correct answer after asking for help.
We refer the readers to (Weston, 2016) for more detailed descriptions and the motivation behind these tasks. The difference in our system is that the model can be trained on-the-fly via the simulator: after receiving feedback and/or rewards, the model can update itself and apply its learning to the next episode. We present results on Tasks 2, 3 and 4 in this appendix
B INSTRUCTIONS GIVEN TO TURKERS
These are the instructions given for the textual feedback mechanical turk task (we also constructed a separate task to collect the initial questions, not described here):
Title: Write brief responses to given dialogue exchanges (about 15 min)
Description: Write a brief response to a student’s answer to a teacher’s question, providing feedback to the student on their answer.
Instructions:
Each task consists of the following triplets:
1. a question by the teacher 2. the correct answer(s) to the question (separated by “OR”) 3. a proposed answer in reply to the question from the student
Consider the scenario where you are the teacher and have already asked the question, and received the reply from the student. Please compose a brief response giving feedback to the student about their answer. The correct answers are provided so that you know whether the student was correct or not.
For example, given 1) question: “what is a color in the united states flag?”; 2) correct answer: “white, blue, red”; 3) student reply: “red”, your response could be something like “that’s right!”; for 3) reply: “green”, you might say “no that’s not right” or “nope, a correct answer is actually white”.
Please vary responses and try to minimize spelling mistakes. If the same responses are copied/pasted or overused, we’ll reject the HIT.
Avoid naming the student or addressing “the class” directly.
We will consider bonuses for higher quality responses during review.
C ADDITIONAL EXPERIMENTS
C.1 ADDITIONAL EXPERIMENTS FOR MECHANICAL TURK SETUP
In the experiment in Section 5.2 we conducted experiments with real human feedback. Here, we compare this to a form of synthetic feedback, mostly as a sanity check, but also to see how much improvement we can get if the signal is simpler and cleaner (as it is synthetic). We hence constructed synthetic feedback for the 10,000 responses, using either Task 2 (positive or negative feedback), Task 3 (answers provided by teacher) or a mix (Task 2+3) where we use one or the other for each example (50% chance of each). The latter makes the synthetic data have a mixed setup of responses, which more closely mimics the real data case. The results are given in Table 4. The RBI+FP combination is better using the synthetic data than the real data with Task 2+3 or Task 3, which is to be expected, but the real data is competitive, despite the difficulty of dealing with its lexical and semantic variability. The real data is better than using Task 2 synthetic data.
For comparison purposes, we also ran a supervised (imitation learning) MemN2N on different sized training sets of turker authored questions with gold annotated labels (so, there are no numerical rewards or textual feedback, this is a pure supervised setting). The results are given in Table 5. They indicate that RBI+FP and even FP alone get close to the performance of fully supervised learning.
C.2 SECOND ITERATION OF FEEDBACK
We conducted experiments with an additional iteration of data collection for the case of binary rewards and textual feedback using the synthetic Task 2+3 mix. We selected the best model from the previous training, using RBI+FP with r = 1 which previously gave a test accuracy of 0.478 (see Table 4). Using that model as a predictor, we collected an additional 10,000 training examples.
We then continue to train our model using the original 1k+10k training set, plus the additional 10k. As before, we report the test accuracy varying r on the additional collected set. We also report the performance from varying , the proportion of random exploration of predictions on the new set. The results are reported in Table 6. Overall, performance is improved in the second iteration, with slightly better performance for large r and = 0.5. However, the improvement is mostly invariant to those parameters, likely because FP takes advantage of feedback from incorrect predictions in any case.

SUMMARY: This paper describes a set of experiments evaluating techniques for
training a dialogue agent via reinforcement learning. A
standard memory network architecture is trained on both bAbI and a version of
the WikiMovies dataset (as in Weston 2016, which this work extends). Numerous
experiments are performed comparing the behavior of different training
algorithms under various experimental conditions.
STRENGTHS: The experimentation is comprehensive. I agree with the authors that
these results provide additional useful insight into the performance of the
model in the 2016 paper (henceforth W16).
WEAKNESSES: This is essentially an appendix to the earlier paper. There is no
new machine learning content. Secondarily, the paper seems to confuse the
distinction between "training with an adaptive sampling procedure" and "training
in interactive environments" more generally. In particular, no comparisons are
presented to the to the experiments with a static exploration policy presented
in W16, when the two training can & should be evaluated side-by-side.
The only meaningful changes between this work and W16 involve simple
(and already well-studied) changes to the form of this exploration policy.
My primary concern remains about novelty: the extra data introduced here is
welcome enough, but probably belongs in a *ACL short paper or a technical
report. This work does not stand on its own, and an ICLR submission is not an
appropriate vehicle for presenting it.
"REINFORCEMENT LEARNING"
[Update: concerns in this section have been addressed by the authors.]
This paper attempts to make a hard distinction between the reinforcement
learning condition considered here and the ("non-RL") condition considered in
W16. I don't think this distinction is nearly as sharp as it's
made out to be.
As already noted in Weston 2016, the RBI objective is a special case of vanilla
policy gradient with a zero baseline and off-policy samples. In this sense the
version of RBI considered in this paper is the same as in W16, but with a
different exploration policy; REINFORCE is the same objective with a nontrivial
baseline. Similarly, the change in FP is only a change to the sampling policy.
The fixed dataset / online learning distinction is not especially meaningful
when the fixed dataset consists of endless synthetic data.
It should be noted that some variants of the exploration policy in W16 provide a
stronger training signal than is available in the RL "from scratch" setting
here: in particular, when $\pi_acc = 0.5$ the training samples will feature much
denser reward. However, if I correctly understand Figures 3 and 4 in this paper,
the completely random initial policy achieves an average reward of ~0.3 on bAbI
and ~0.1 on movies---as good or better than the other exploration policies in
W16!
I think this paper would be a lot clearer if the delta from W16 were expressed
directly in terms of their different exploration policies, rather than trying to
cast all of the previous work as "not RL" when it can be straightforwardly
accommodated in the RL framework.
I was quite confused by the fact that no direct comparisons are made to the
training conditions in the earlier work. I think this is a symptom of the
problem discussed above: once this paper adopts the position that this work is
about RL and the previous work is not, it becomes possible to declare that the
two training scenarios are incomparable. I really think this is a mistake---to
the extent that the off-policy sample generators used in the previous paper are
worse than chance, it is always possible to compare to them fairly here.
Evaluating everything in the "online" setting and presenting side-by-side
experiments would provide a much more informative picture of the comparative
behavior of the various training objectives.
ON-POLICY VS OFF-POLICY
Vanilla policy gradient methods like the ones here typically can't use
off-policy samples without a little extra hand-holding (importance sampling,
trust region methods, etc.). They seem to work out of the box for a few of the
experiments in this paper, which is an interesting result on its own. It would
be nice to have some discussion of why that might be the case.
OTHER NOTES
- The claim that "batch size is related to off-policy learning" is a little
odd. There are lots of on-policy algorithms that require the agent to collect a
large batch of transitions from the current policy before performing an
(on-policy) update.
- I think the experiments on fine-tuning to human workers are the most exciting
part of this work, and I would have preferred to see these discussed (and
explored with) in much more detail rather than being relegated to the
penultimate paragraphs.


We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how overparametrized the system is, and for the edges that depend on the input data.
We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how overparametrized the system is, and for the edges that depend on the input data.
1 INTRODUCTION
Given a (piece-wise) differentiable loss function, and a gradient based algorithm to minimize it, the knowledge of the second order information about it can tell us quite a bit about how the landscape looks like, and how we could modify our algorithm to make it go faster and find better solutions. But, one of the biggest challenges in second order optimization methods is in accessing that second order information itself. In particular, in deep learning there have been many proposals to accelerate training using second order information. Ngiam et al. (2011) has an in depth review of some of the proposals for approximating the Hessian of the loss function. Nevertheless, given the computational complexity of the problems at hand, it is hard to acquire information on what the actual Hessian looks like. This work is part of a series of papers that explore the data-model-algorithm connection along with Sagun et al. (2014; 2015) and it builds on top of the intuition developed in LeCun et al. (2012). We also note that the singularity of the Fisher information matrix have been explored (see for instance Watanabe (2007)). In another recent work, Dauphin et al. (2014) investigate saddle points of the landscape, in particular, they locate saddle points that are near the training path. In this work, however, we strictly focus on the Hessian of the loss function at the exact point of the training. We train the main examples using gradient descent. We perform our calculations of the Hessian using the implementation for the exact Hessian vector product that has been introduced in Pearlmutter (1994). And we find two new observations: one where the eigenvalues are zero, and one where we have a large, positive, and discrete set of eigenvalues.
In this short note, we show how the data and the architecture depends on the eigenvalues of the Hessian of the loss function. In particular, we observe that the top discrete eigenvalues depend on the data, and the bulk of the eigenvalues depend on the architecture. Furthermore, as we keep growing the size of the network, we observe that the discrete part that depends on data remains the same, but the concentration around zero sharpens.
There are various conclusions and implications of this singularity. Recent research suggest new insights into convergence properties of gradient based algorithms in non-convex systems (Lee et al., 2016; Hardt et al., 2015). The results come together with their implications on neural networks. However, the proofs require the system at hand to be non-degenerate. An immediate conclusion of our observation is that the Hessian of the loss function is very singular. Therefore, a lot of the theory and methodology that assumes non-singular Hessian cannot be applied without an appropriate modification.
2 THE CASE OF THE FULLY-CONNECTED NETWORK
2.1 MNIST WITH INCREASING SIZES OF HIDDEN LAYERS
We calculate the exact Hessian of the loss function of a network with two hidden layers. The inputs are 1000 randomly selected examples of 28 ∗ 28 MNIST data, the network has one hidden layer with ReLU nonlinearity, the top layer has a softmax and a negative log likelihood loss function at the end. We train the system with gradient descent (i.e. with minibatch size equal to the number of examples). We plot the histogram of the eigenvalues of the Hessian for a varying number of hidden units after convergence. The Hessian at the end of the training turns out to be extremely singular, and increasing the number of units in hidden layers only add to the singularity of the Hessian (see figure 1). The effect of the training on the eigenvalue spectrum of the model with 10 hidden-units is visible when comparing figure 1 and figure 2 (left).
2.2 VARYING THE DATA
To demonstrate how the eigenvalue distribution may depend on data itself, we keep the same architecture and change the inputs to random patterns. Initially, a random point in the weight space is selected, and we calculated the Hessian without any training (first two histograms of figure 2). After training the system until the norm of the gradient is close to zero. We again calculate the exact Hessian and plot the histogram of their eigenvalues which can be seen in the last one in figure 2.
3 A SIMPLER CASE
In this section, we will repeat the same experiment in two-dimensional data, in an attempt to understand better the connection between the data and the spectrum of the Hessian. A simple figure can be seen in figure 3. We create two Gaussian blobs, centered at (1, 1) and (−1,−1), and first we keep the standard deviation the same, and increase the network size.
The network architecture is similar, this time with two hidden layers and a fully connected network with ReLU nonlinearities including a softmax at the top layer combined with a negative loglikelihood loss function. We train the system with gradient descent with constant step size. At the end of the training the norm of the gradient is at the order of 10−4.
There are two eigenvalues that are isolated, and away from the bulk of the spectrum. Increasing the network only adds to the concentration of eigenvalues at and around zero (see figure 4). To give an insight into how the Hessian’s themselves look like, in figure 6 we plot the full Hessian matrices for three of the systems above after training.
Moreover, this property of the singular and discrete parts is not specific to the log loss. In figure 5, we plot the histogram for a system that is trained on the same data as in figure 3, and the same
architecture. But the training is carried out with the mean square loss rather than the negative loglikelihood. Consistent with our previous observations, we still see the same discrete, data-dependent part, and the part that is at zero.
The training procedure itself acts like a process by which the eigenvalues concentrate at zero. To demonstrate this in further detail we calculated the full Hessian peridoically throughout the training. In figure 7 we plot the eigenvalue profile as the training progresses.
All of the training has been done with random initializations on the weight space with the same standard deviation. In other words, initial points are randomly chosen on the surface of a sphere with a fixed radius given the total number of parameters. This begs the question of the effect of the choice of the initial point. Therefore, now we fix the network size, and repeat the experiment with different random initializations over 5K times. In figure 8 we plot the fluctuations of the top eigenvalue.
The next question is how the spectrum responds to the increased complexity of data. The notion of complexity for a given dataset can be tricky to describe, here we use a loose notion of complexity to point out the fact that the more complex data is the less separable one. To this end, we keep the architecture the same, and increase the standard deviation of the two Gaussian blobs. They are still centered at the same two points, but it becomes harder to separate them as they merge together. Gradient descent still converges to a low-cost value, but the error is higher, and it can’t learn how to separate them perfectly as the blobs merge together. In figure 9, we observe that the top two
eigenvalues grow significantly, and beyond its natural fluctuations due to the initialization. We also note that the norm of the weights are similar for all the cases, therefore the growth in the sizes of eigenvalues can not solely be accounted for the growth in weights.
4 CONCLUSION
We show that the Hessian of the loss functions in deep learning is degenerate. This has implications on the theoretical work which requires improvements in its premises. One such step has been taken in Panageas & Piliouras (2016) in relaxing the isolated singularity condition that was assumed in Lee et al. (2016). From a practical point of view this has multiple implications:
• The landscape may be flat beyond the notion of wide basins. • Training stops at a point that has a small gradient. The norm of the gradient is not zero,
therefore it does not, technically speaking, converge to a critical point. • There are still negative eigenvalues even when they are small in magnitude.
This suggests that we may be able to look beyond the classical notions of basins when exploring the energy landscapes of loss functions. Next obvious question is to find low energy paths between
solutions to show the kind of flatness in such landscapes. This will be explored in a subsequent work in the same series.
We also demonstrate the two phases of the spectrum, one that is concentrated around zero that depends on the size of the model, and the second part that is away from the bulk of the spectrum, that is isolated and depends on the data.
This kind of two-phased non-degeneracy can, in fact, be a desirable property. A degenerate Hessian implies locally flat regions. A degenerate Hessian at the scale that we observe in deep learning may imply flat regions across space, at the global scale.
• We can devise separate methods for the directions that correspond to the top eigenvalues. • We can take advantage of the directions that correspond to the zero or small eigenvalues by
attempting to find paths of low energies in the weight space.
As a first step to the last item, initial experiments are promising: Let’s take a random point on the weight space and train two systems from that point: (1) with gradient descent, and (2) with stochastic gradient descent. At each step, take a straight line between the two points and interpolate the cost value. The resulting profile is completely flat even when the points keep diverging from one another. Next, take two random initial points on the weight space, so now they are orthogonal to each other. And train two systems with different shuffling of data for SGD. This time one would expect the line interpolation to give arbitrary values since the initial points are completely orthogonal, surprisingly, the line interpolation also decreases albeit not as flat as the previous one. Further considerations on
connecting paths between solutions in the weight space of loss functions can be found in Freeman & Bruna (2016).
ACKNOWLEDGMENTS
We would like to thank Afonso Bandeira, Yann Dauphin, Ruoyu Sun, Arthur Szlam and Soumith Chintala for valuable discussions. We also thank the reviewers for valuable feedback. Part of the research has been conducted when the first author was an intern at FAIR.

Studying the Hessian in deep learning, the experiments in this paper suggest that the eigenvalue distribution is concentrated around zero and the non zero eigenvalues are related to the complexity of the input data. I find most of the discussions and experiments to be interesting and insightful. However, the current paper could be significantly improved.
Quality:
It seems that the arguments in the paper could be enhanced by more effort and more comprehensive experiments. Performing some of the experiments discussed in the conclusion could certainly help a lot. Some other suggestions:
1- It would be very helpful to add other plots showing the distribution of eigenvalues for some other machine learning method for the purpose of comparison to deep learning.
2- There are some issues about the scaling of the weights and it make sense to normalize the weights each time before calculating the Hessian otherwise the result might be misleading.
3- It might worth trying to find a quantity that measures the singularity of Hessian because it is difficult to visually conclude something from the plots.
4- Adding some plots for the Hessian during the optimization is definitely needed because we mostly care about the Hessian during the optimization not after the convergence.
Clarity:
1- There is no reference to figures in the main text which makes it confusing for the reading to know the context for each figure. For example, when looking at Figure 1, it is not clear that the Hessian is calculated at the beginning of optimization or after convergence.
2- The texts in the figures are very small and hard to read.