REFERENCE-AWARE LANGUAGE MODELS
We propose a general class of language models that treat reference as an explicit stochastic latent variable. This architecture allows models to create mentions of entities and their attributes by accessing external databases (required by, e.g., dialogue generation and recipe generation) and internal state (required by, e.g. language models which are aware of coreference). This facilitates the incorporation of information that can be accessed in predictable locations in databases or discourse context, even when the targets of the reference may be rare words. Experiments on three tasks show our model variants outperform models based on deterministic attention.
1 INTRODUCTION
Referring expressions (REs) in natural language are noun phrases (proper nouns, common nouns, and pronouns) that identify objects, entities, and events in an environment. REs occur frequently and they play a key role in communicating information efficiently. While REs are common, previous works neglect to model REs explicitly, either treating REs as ordinary words in the model or replacing them with special tokens. Here we propose a language modeling framework that explicitly incorporates reference decisions.
In Figure 1 we list examples of REs in the context of the three tasks that we consider in this work. Firstly, reference to a database is crucial in many applications. One example is in task oriented dialogue where access to a database is necessary to answer a user’s query (Young et al., 2013; Li et al., 2016; Vinyals & Le, 2015; Wen et al., 2015; Sordoni et al., 2015; Serban et al., 2016; Bordes & Weston, 2016; Williams & Zweig, 2016; Shang et al., 2015; Wen et al., 2016). Here we consider the domain of restaurant recommendation where a system refers to restaurants (name) and their attributes (address, phone number etc) in its responses. When the system says “the nirala is a nice restaurant”, it refers to the restaurant name the nirala from the database. Secondly, many models need to refer to a list of items (Kiddon et al., 2016; Wen et al., 2015). In the task of recipe generation from a list of ingredients (Kiddon et al., 2016), the generation of the recipe will frequently reference these items. As shown in Figure 1, in the recipe “Blend soy milk and . . . ”, soy milk refers to the ingredient summaries. Finally, we address references within a document (Mikolov et al., 2010; Ji et al., 2015; Wang & Cho, 2015), as the generation of words will ofter refer to previously generated words. For instance the same entity will often be referred to throughout a document. In Figure 1, the entity you refers to I in a previous utterance.
In this work we develop a language model that has a specific module for generating REs. A series of latent decisions (should I generate a RE? If yes, which entity in the context should I refer to? How should the RE be rendered?) augment a traditional recurrent neural network language model and the two components are combined as a mixture model. Selecting an entity in context is similar to familiar models of attention (Bahdanau et al., 2014), but rather than being a deterministic function that reweights representations of elements in the context, it is treated as a distribution over contextual elements which are stochastically selected and then copied or, if the task warrants it, transformed (e.g., a pronoun rather than a proper name is produced as output). Two variants are possible for updating the RNN state: one that only looks at the generated output form; and a second that looks at values of the latent variables. The former admits trivial unsupervised learning, latent decisions are conditionally independent of each other given observed context, whereas the latter enables more
∗Work completed at DeepMind.
expressive models that can extract information from the entity that is being referred to. In each of the three tasks, we demonstrate our reference aware model’s efficacy in evaluations against models that do not explicitly include a reference operation.
Our contributions are as follows:
• We propose a general framework to model reference in language and instantiate it in the context of dialogue modeling, recipe generation and coreference based language models.
• We build three data sets to test our models. There lack existing data sets that satisfy our need, so we build these data sets ourselves. These data sets are either built on top existing data set (we constructed the table for DSTC2 data set for dialogue evaluation), crawled from websites (we crawled all recipes in www.allrecipes.com) or annotated with NLP tools (we annotate the coreference with Gigaword corpus for our evaluation).
• We perform comprehensive evaluation of our models on the three data sets and verify our models perform better than strong baselines.
2 REFERENCE-AWARE LANGUAGE MODELS
Here we propose a general framework for reference-aware language models.
We denote each document as a series of tokens x1, . . . , xL, where L is the number of tokens in the document. Our goal is to maximize the probabilities p(xi | ci), for each word in the document based on its previous context ci = x1, . . . , xi−1. In contrast to traditional neural language models, we introduce a variable at each position zi, which controls the decision on which source xi is generated from. The token conditional probably is then obtained by:
p(xi | ci) = p(xi | zi, ci)p(zi | ci). (1)
In dialogue modeling and recipe generation, zi will simply taken on values in {0, 1}. Where zi = 1 denotes that xi is generated as a reference, either to a database entry or an item in a list. However, zi can also be defined as a distribution over previous entities, allowing the model to predict xi conditioned on its a previous mention word. This will be the focus of the coreference language model. When zi is not observed (which it generally will not be), we will train our model to maximize the marginal probability in Eq. 1 directly.
2.1 DIALOGUE MODEL WITH DATABASE SUPPORT
We first apply our model on task-oriented dialogue systems in the domain of restaurant recommendations, and work on the data set from the second Dialogue State Tracking Challenge (DSTC2) (Henderson et al., 2014). Table. 1 is one example dialogue from this dataset.
We can observe from this example, users get recommendations of restaurants based on queries that specify the area, price and food type of the restaurant. We can support the system’s decisions by incorporating a mechanism that allows the model to query the database allowing the model to find restaurants that satisfy the users queries. Thus, we crawled TripAdvisor for restaurants in the
Cambridge area, where the dialog dataset was collected. Then, we remove restaurants that do not appear in the data set and create a database with 109 entries with restaurants and their attributes (e.g. food type). A sample of our database is shown in Table. 2. We can observe that each restaurant contains 6 attributes that are generally referred in the dialogue dataset. As such, if the user requests a restaurant that serves “indian” food, we wish to train a model that can search for entries whose “food” column contains “indian”. Now, we describe how we deploy a model that fulfills these requirements.
2.1.1 DIALOGUE MODEL
We build a model based on the hierarchical RNN model described in (Serban et al., 2016), as in dialogues, the generation of the response is not only dependent on the previous sentence, but on all sentences leading to the response. We assume that a dialogue is alternated between a machine and a user. An illustration of the model is shown in Figure 2.
Consider a dialogue with T turns, and the utterance from a user is denoted as X = {xi}Ti=1, where i is the i-th utterance, whereas the utterance from a machine is denoted as Y = {yi}Ti=1, where i is the i-th utterance. We define xi = {xij}|xi|j=1, yi = {yiv} |yi| v=1, where xij denotes the j-th token in the i-th utterance from the user, whereas yiv denotes the v-th token in the i-th utterance from the machine. Finally, |xi| and |yi| denote the number of tokens in the user and machine utterances, respectively. The dialogue sequence starts with machine utterance {y1, x1, y2, x2, . . . , yT , xT }. We would like to model the utterances from the machine
p(y1, y2, . . . , yT |x1, x2, . . . , xT ) = ∏ i p(yi|y<i, x<i) = ∏ i,v p(yi,v|yi,<v, y<i, x<i),
where y<i denotes all the utterances before i and yi,<v denotes the first v − 1 tokens in the i-th utterance of the machine. A neural model is employed to predict p(yi,v|yi,<v, y<i, x<i), which operates as follows:
Sentence Encoder: We first encode previous utterances y<i and x<i into continuous space by generating employing a LSTM encoder. Thus, for a given utterance xi, and start with the initial LSTM state hxi,0 and apply the recursion h x i,j = LSTME(WExi,j , h x i,j−1), where WExi,j denotes a word
embedding lookup for the token xi,j , and LSTME denotes the LSTM transition function described in Hochreiter & Schmidhuber (1997). The representation of the user utterance is represented by the final LSTM state hxi = h x i,|xi|. The same process is applied to obtain the machine utterance representation hyi = h y i,|yi|.
Turn Encoder: Then, combine all the representations of all the utterances with a second LSTM, which encodes the sequence {hy1, hx1 , ..., h y i , h x i } into a continuous vector. Once again, we start with an initial state u0 and feed each of the utterance representation to obtain the following LSTM state, until the final state is obtained. For simplicity, we shall refer to this as ui, which can be seen as the hierarchical encoding of the previous i utterances.
Seq2Seq Decoder: As for decoding, in order to generate each utterance yi, we can feed ui−1 into the decoder LSTM as the initial state si,0 = ui−1 and decode each token in yi. Thus, we can express the decoder as:
syi,v = LSTMD(WEyi,v−1, si,v−1), pyi,v = softmax(Ws y i,v),
where the desired probability p(yi,v|yi,<v, y<i, x<i) is expressed by pyi,v .
Attention based decoder: We can also incorporate the attention mechanism in our hierarchical model. An attention model builds a representation d by averaging over a set of vectors p. We define the attention function as a = ATTN(p, q), where a is a probability distribution over the set of vectors p, conditioned on any input representation q. A full description of this operation is described in (Bahdanau et al., 2014). Thus, for each generated token yi,v, we compute the attentions ai,v , conditioned on the current decoder state syi,v, obtaining the attentions over input tokens from previous turn (i−1). We denote the vector of all tokens in previous turn as hx,yi−1 = [{hxi−1,j} |xi−1| j=1 , {h y i−1,v} |yi−1| v=1 ]. Let K = |hx,yi−1| be the number of tokens in previous turn. Thus, we obtain the attention probabilities over all previous tokens ai,v as ATTN(s y i,v, h x,y i−1). Then, the weighted sum is computed over these
probabilities di,v = ∑ k∈K ai,v,kh x,y i−1,k, where ai,v,k is the probability of aligning to the k-th token from previous turn. The resulting vector di,v is used to obtain the probability of the following word pyi,v . Thus, we express the decoder as:
syi,v = LSTMD([WEyi,v−1, di,v−1], si,v−1), ai,v = ATTN(h x,y i−1, s y i,v),
di,v = ∑ k∈K ai,v,kh x,y i−1,k, pyi,v = softmax(W [s y i,v, di,v]).
2.1.2 INCORPORATING TABLE ATTENTION
We now extend the attention model in order to allow the attention to be computed over a table, allowing the model to condition the generation on a database.
We denote a table with R rows and C columns as {fr,c}, r ∈ [1, R], c ∈ [1, C], where fr,c is the cell in row r and column c. The attribute of each column is denoted as sc, where c is the c-th attribute. fr,c and sc are one-hot vector.
Table Encoding: To encode the table, we build an attribute vector gc for each column. For each cell fr,c of the table, we concatenate it with the corresponding attribute gc and then feed it through a one-layer MLP as follows: gc = WEsc and then er,c = tanh(W [WEfr,c, gc]).
Table Attention: The diagram for table attention is shown in Figure 3a. The attention over cells in the table is conditioned on a given vector q, similarly to the attention model for sequences ATTN(p, q). However, rather than a sequence p, we now operate over a table f . Our attention model computes a attribute attention followed by row attention of the table. We first use the attention mechanism on the attributes to find out which attribute the user asks about. Suppose a user says cheap, then we should focus on the price attribute. After we get the attention probability pa = ATTN({gc}, q), over the attribute, we calculate the weighted representation for each row er = ∑ c p a cerc conditioned on p
a. Then er has the price information of each row. We further use attention mechanism on er and get the probability pr = ATTN({er}, q) over the rows. Then restaurants with cheap price will be picked. Then, using the probabilities pr, we compute the weighted average over the all rows ec = ∑ r p
r rer,c, which is used in the decoder. The detailed process is: pa = ATTN({gc}, q), (2)
er = ∑ c pacerc ∀r, (3)
pr = ATTN({er}, q), (4) ec = ∑ r prrer,c ∀c. (5)
This is embedded in the decoder by replacing the conditioned state q as the current decoder state syi,0 and then at each step, conditioning the prediction of yi,v on {ec} by using attention mechanism at each step. The detailed diagram of table attention is shown in Figure 3a.
2.1.3 INCORPORATING TABLE POINTER NETWORKS
We now describe the mechanism used to refer to specific database entries during decoding. At each timestep, the model needs to decide whether to generate the next token from an entry of the database or from the word softmax. This is performed as follows.
Pointer Switch: We use zi,v ∈ [0, 1] to denote the decision of whether to copy one cell from the table. We compute this probability as follows:
p(zi,v|si,v) = sigmoid(W [si,v, di,v]). Thus, if zi,v = 1, the next token yi,v will be generated from the database, whereas if zi,v = 0, then the following token is generated from a softmax. We shall now describe how we generate tokens from the database.
Table Pointer: If zi,v = 1, the token is generated from the table. The detailed process of calculating the probability distribution over the table is shown in Figure 3b. This is similar to the attention mechanism, except that we perform a column attention to compute the probabilities of copying from each column after Equation. 5. More formally:
pc = ATTN({ec}, q), (6) pcopy = pr ⊗ pc, (7)
where pc is a probability distribution over columns, whereas pr is a probability distribution over rows. In order to compute a matrix with the probability of copying each cell, we simply compute the outer product pcopy = pr ⊗ pc. Objective: As we treat zi as a latent variable, we wish to maximize the marginal probability of the sequence yi over all possible values of zi. Thus, our objective function is defined as:
p(yi,v|si,v) = pvocabp(0|si,v) + pcopyp(1|si,v) = pvocab(1− p(1|si,v)) + pcopyp(1|si,v). (8)
The model can also be trained in a fully supervised fashion, if zi,v is observed. In such cases, we simply maximize the likelihood of p(zi,v|si,v), based on the observations, rather than using the marginal probability over zi,v .
2.2 RECIPE GENERATION
Next, we consider the task of recipe generation conditioning on the ingredient lists. In this task, we must generate the recipe from a list of ingredients. Table. 3 illustrates the ingredient list and recipe for Spinach and Banana Power Smoothie. We can see that the ingredients soy milk, spinach leaves, and banana occur in the recipe.
Let the ingredients of a recipe be X = {xi}Ti=1 and each ingredient contains L tokens xi = {xij}Lj=1. The corresponding recipe is y = {yv}Kv=1. We first use a LSTM to encode each ingredient:
hi,j = LSTME(WExij , hi,j−1) ∀i. Then, we sum the resulting state of each ingredient to obtain the starting LSTM state of the decoder. Once again we use an attention based decoder:
sv = LSTMD(sv−1, dv−1,WEyv−1),
pcopyv = ATTN({{hi,j}Ti=1}Lj=1, sv), dv = ∑ ij pv,i,jhi,j ,
p(zv|sv) = sigmoid(W [sv, dv]), pvocabv = softmax(W [sv, dv]).
Similar to the previous task, the decision to copy from the ingredient list or generate a new word from the softmax is performed using a switch, denoted as p(zv|sv). We can obtain a probability distribution of copying each of the words in the ingredients by computing pcopyv = ATTN({{hi,j}Ti=1}Lj=1, sv) in the attention mechanism. For training, we optimize the marginal likelihood function employed in the previous task.
2.3 COREFERENCE BASED LANGUAGE MODEL
Finally, we build a language model that uses coreference links to point to previous words. Before generating a word, we first make the decision on whether it is an entity mention. If so, we decide
which entity this mention belongs to, then we generate the word based on that entity. Denote the document as X = {xi}Li=1, and the entities are E = {ei}Ni=1, each entity has Mi mentions, ei = {mij}Mij=1, such that {xmij} Mi j=1 refer to the same entity. We use a LSTM to model the document, the hidden state of each token is hi = LSTM(WExi, hi−1). We use a set he = {he0, he1, ..., heM} to keep track of the entity states, where hej is the state of entity j.
um and [I]1 think that is whats - Go ahead [Linda]2. Well and thanks goes to [you]1 and to [the media]3 to help [us]4...So [our]4 hat is off to all of [you]5...
Word generation: At each time step before generating the next word, we predict whether the word is an entity mention:
pcoref(vi|hi−1, he) = ATTN(he, hi−1), di = ∑ vi p(vi)h e vi
p(zi|hi−1) = sigmoid(W [di, hi−1]),
where zi denotes whether the next word is an entity and if yes vi denotes which entity the next word corefers to. If the next word is an entity mention, then p(xi|vi, hi−1, he) = softmax(W1 tanh(W2[hevi , hi−1])) else p(xi|hi−1) = softmax(W1hi−1),
p(xi|x<i) = { p(xi|hi−1)p(zi|hi−1, he) if zi = 0. p(xi|vi, hi−1, he)pcoref(vi|hi−1, he)p(zi|hi−1, he) if zi = 1.
(9)
Entity state update: We update the entity state he at each time step. In the beginning, he = {he0}, he0 denotes the state of an virtual empty entity and is a learnable variable. If zi = 1 and vi = 0, then it indicates the next word is a new entity mention, then in the next step, we append hi to he, i.e., he = {he, hi}, if ei > 0, then we update the corresponding entity state with the new hidden state, he[vi] = hi. Another way to update the entity state is to use one LSTM to encode the mention states and get the new entity state. Here we use the latest entity mention state as the new entity state for simplicity. The detailed update process is shown in Figure 5.
3 EXPERIMENTS
4 DATA SETS AND PREPROCESSING
Dialogue: We use the DSTC2 data set. We only extracted the dialogue transcript from data set. There are about 3,200 dialogues in total. Since this is a small data set, we use 5-fold cross validation and report the average result over the 5 partitions. There may be multiple tokens in each table cell, for example in Table.2, the name, address, post code and phone number have multiple tokens, we replace them with one special token. For the name, address, post code and phone number of the j-th row, we replace the tokens in each cell with NAME j, ADDR j, POSTCODE j, PHONE j. If a table cell is empty, we replace it with an empty token EMPTY. We do a string match in the transcript and replace the corresponding tokens in transcripts from the table with the special tokens.
Each dialogue on average has 8 turns (16 sentences). We use a vocabulary size of 900, including about 400 table tokens and 500 words.
Recipes: We crawl all recipes from www.allrecipes.com. There are about 31, 000 recipes in total, and every recipe has a ingredient list and a corresponding recipe. We exclude the recipes that have less than 10 tokens or more than 500 tokens, those recipes take about 0.1% of all data set. On average each recipe has 118 tokens and 9 ingredients. We random shuffle the whole data set and take 80% as training and 10% for validation and test. We use a vocabulary size of 10,000 in the model.
Coref LM: We use the Xinhua News data set from Gigaword Fifth Edition and sample 100,000 documents from it that has length in range from 100 to 500. Each document has on average 234 tokens, so there are 23 million tokens in total. We use a tool to annotate all the entity mentions and use the annotation in the training. We take 80% as training and 10% as validation and test respectively. We ignore the entities that have only one mention and for the mentions that have multiple tokens, we take the token that is most frequent in the all the mentions for this entity. After the preprocessing, tokens that are entity mentions take about 10% of all tokens. We use a vocabulary size of 50,000 in the model.
4.1 MODEL TRAINING AND EVALUATION
We train all models with simple stochastic gradient descent with clipping. We use a one-layer LSTM for all RNN components. Hyper-parameters are selected using grid search based on the validation set. We use dropout after the input embedding and LSTM output. The learning rate is selected from [0.1, 0.2, 0.5, 1], maximum gradient norm is selected from [1, 2, 5, 10] and drop ratio is selected from [0.2, 0.3, 0.5]. The batch size and LSTM dimension size is slightly different for different tasks so as to make the model fit into memory. The number of epochs to train are different for each task and we drop the learning rate after reaching a given number of epochs. We report the per-word perplexity for all tasks, specifically, we report the perplexity of all words, words that can be generated from reference and non-reference words. For recipe generation, we also generate the recipe using beam size of 10 and evaluate the generated recipe with BLEU.
4.2 RESULTS AND ANALYSIS
The results for dialogue, recipe generation and coref language model are shown in Table 4, 5 and 6 respectively. We can see from Table 4 that models that condition on table performs better in predicting table tokens in general. Table pointer has the lowest perplexity for token in the table. Since the table token appears rarely in the dialogue, the overall perplexity does not differ much and the non-table tokens perplexity are similar. With attention mechanism over the table, the perplexity of table token improves over basic seq2seq model, but not as good as directly pointing to cells in the table. As expected, using sentence attention improves significantly over models without sentence attention. Surprisingly, table latent performs much worse than table pointer. We also measure the perplexity of table tokens that appear only in test set. For models other than table pointer, because the tokens never appear in training set, the perplexity is quite high, while table pointer can predict these tokens much more accurately. The recipe results in Table 5 in general follows that findings from the dialogue. But the latent model performs better than pointer model since that tokens in ingredients that match with recipe does not necessarily come from the ingredients. Imposing a supervised signal will give wrong information to the model and hence make the result worse. Hence with latent decision, the model learns to when to copy and when to generate it from the vocabulary. The coref LM results are shown in Table 6. We find that coref based LM performs much better on the entities perplexities, but however is a little bit worse than for non-entity words. We found it is an optimization problem and perhaps the model is stuck in local optimum. So we initialize the pointer model with the weights learned from LM, the pointer model performs better than LM both for entity perplexity and non-entity words perplexity.
5 RELATED WORK
Recently, there has been great progresses in modeling languages based on neural network, including language modeling (Mikolov et al., 2010; Jozefowicz et al., 2016), machine translation (Sutskever et al., 2014; Bahdanau et al., 2014), question answering (Hermann et al., 2015) etc. Based on the success of seq2seq models, neural networks are applied in modeling chit-chat dialogue (Li et al., 2016; Vinyals & Le, 2015; Sordoni et al., 2015; Serban et al., 2016; Shang et al., 2015) and task oriented dialogue (Wen et al., 2015; Bordes & Weston, 2016; Williams & Zweig, 2016; Wen et al., 2016). Most of the chit-chat neural dialogue models are simply applying the seq2seq models. For the task oriented dialogues, most of them embed the seq2seq model in traditional dialogue systems, in which the table query part is not differentiable. while our model queries the database directly. Recipe generation was proposed in (Kiddon et al., 2016). Their model extents previous work on attention models (Allamanis et al., 2016) to checklists, whereas our work models explicit references to those checklists. Context dependent language models (Mikolov et al., 2010; Ji et al., 2015; Wang & Cho, 2015) are proposed to capture long term dependency of text. There are also lots of works on coreference resolution (Haghighi & Klein, 2010; Wiseman et al., 2016). We are the first to combine coreference with language modeling, to the best of our knowledge. Much effort has been invested in embedding a copying mechanism for neural models (Gülçehre et al., 2016; Gu et al., 2016; Ling et al., 2016). In general, a gating mechanism is employed to combine the softmax over observed words and a pointer network (Vinyals et al., 2015). These gates can be trained either by marginalizing over both outcomes, or using heuristics (e.g. copy low frequency words). Our models are similar to models proposed in (Ahn et al., 2016; Merity et al., 2016), where the generation of each word can be conditioned on a particular entry in knowledge lists and previous words. In our work, we describe a model with broader applications, allowing us to condition, on databases, lists and dynamic lists.
6 CONCLUSION
We introduce reference-aware language models which explicitly model the decision of from where to generate the token at each step. Our model can also learns the decision by treating it as a latent variable. We demonstrate on three tasks, table based dialogue modeling, recipe generation and coref based LM, that our model performs better than attention based model, which does not incorporate this decision explicitly. There are several directions to explore further based on our framework. The current evaluation method is based on perplexity and BLEU. In task oriented dialogues, we can also try human evaluation to see if the model can reply users’ query accurately. It is also interesting to use reinforcement learning to learn the actions in each step.

This paper introduces pointer-network neural networks, which are applied to referring expressions in three small-scale language modeling tasks: dialogue modeling, recipe modeling and news article modeling. When conditioned on the co-reference chain, the proposed models outperform standard sequence-to-sequence models with attention.
The proposed models are essentially variants of pointer networks with copy mechanisms (Gulcehre et al., 2016; Gu et al., 2016; Ling et al., 2016), which have been modified to take into account reference chains. As such, the main architectural novelty lies in 1) restricting the pointer mechanism to focus on co-referenced entities, 2) applying pointer mechanism to 2D arrays (tables), and 3) training with supervised alignments. Although useful in practice, these are minor contributions from an architectural perspective.
The empirical contributions are centred around measuring perplexity on the three language modeling tasks. Measuring perplexity is typical for standard language modeling tasks, but is really an unreliable proxy for dialogue modeling and recipe generation performance. In addition to this, both the dialogue and recipe tasks are tiny compared to standard language modeling tasks. This makes it difficult to evaluate the impact of the dialogue and recipe modeling results. For example, if one was to bootstrap from a larger corpus, it seems likely that a standard sequence-to-sequence model with attention would yield performance comparable to the proposed models (with enough data, the attention mechanism could learn to align referring entities by itself). The language modeling task on news article (Gigaword) seems to yield the most conclusive results. However, the dataset for this task is non-standard and results are provided for only a single baseline. Overall, this limits the conclusions we can draw from the empirical experiments.
Finally, the paper itself contains many errors, including mathematical errors, grammatical errors and typos:
- Eq. (1) is missing a sum over $z_i$.
- "into the a decoder LSTM" -> "into the decoder LSTM"
- "denoted as his" -> "denoted as"
- "Surprising," -> "Surprisingly,"
- "torkens" -> "tokens"
- "if follows that the next token" -> "the next token"
- In the "COREFERENCE BASED LANGUAGE MODEL" sub-section, what does $M$ denote?
- In the sentence: "The attribute of each column is denoted as $s_c, where $c$ is the c-th attribute". For these definitions to be make sense, $s_c$ has to be a one-hot vector. If yes, please clarify this in the text.
- "the weighted sum is performed" -> "the weighted sum is computed"
- "a attribute" -> "an attribute"
- In the paragraph on Pointer Switch, change $p(z_{i,v} |s_{i,v}) = 1$ -> $p(z_{i,v} |s_{i,v}) = 0$.
- In the "Table Pointer" paragraph, I assume you mean outer-product instead of cross-product? Otherwise, I don't see how the equations add up.
Other comments:
- For the "Attention based decoder", is the attention computed using the word embeddings themselves or the hidden states of the sentence encoder? Also, it applied only to the previous turn of the dialogue or to the entire dialogue history? Please clarify this.
- What's the advantage of using an "Entity state update" rule, compared to a pointer network or copy network, which you used in the dialogue and recipe tasks? Please elaborate on this.
- In the Related Work section, the following sentence is not quite accurate: "For the task oriented dialogues, most of them embed the seq2seq model in traditional dialogue systems while our model queries the database directly.". There are task-oriented dialogue models which do query databases during natural language generation. See, for example, "A Network-based End-to-End Trainable Task-oriented Dialogue System" by Wen et al.


MULTI-MODAL VARIATIONAL ENCODER-DECODERS
Recent advances in neural variational inference have facilitated efficient training of powerful directed graphical models with continuous latent variables, such as variational autoencoders. However, these models usually assume simple, unimodal priors — such as the multivariate Gaussian distribution — yet many realworld data distributions are highly complex and multi-modal. Examples of complex and multi-modal distributions range from topics in newswire text to conversational dialogue responses. When such latent variable models are applied to these domains, the restriction of the simple, uni-modal prior hinders the overall expressivity of the learned model as it cannot possibly capture more complex aspects of the data distribution. To overcome this critical restriction, we propose a flexible, simple prior distribution which can be learned efficiently and potentially capture an exponential number of modes of a target distribution. We develop the multi-modal variational encoder-decoder framework and investigate the effectiveness of the proposed prior in several natural language processing modeling tasks, including document modeling and dialogue modeling.
Recent advances in neural variational inference have facilitated efficient training of powerful directed graphical models with continuous latent variables, such as variational autoencoders. However, these models usually assume simple, unimodal priors — such as the multivariate Gaussian distribution — yet many realworld data distributions are highly complex and multi-modal. Examples of complex and multi-modal distributions range from topics in newswire text to conversational dialogue responses. When such latent variable models are applied to these domains, the restriction of the simple, uni-modal prior hinders the overall expressivity of the learned model as it cannot possibly capture more complex aspects of the data distribution. To overcome this critical restriction, we propose a flexible, simple prior distribution which can be learned efficiently and potentially capture an exponential number of modes of a target distribution. We develop the multi-modal variational encoder-decoder framework and investigate the effectiveness of the proposed prior in several natural language processing modeling tasks, including document modeling and dialogue modeling.
1 INTRODUCTION
With the development of the variational autoencoding framework (Kingma & Welling, 2013; Rezende et al., 2014), a tremendous amount of progress has been made in learning large-scale, directed latent variable models. This approach has lead to improved performance in applications ranging from computer vision (Gregor et al., 2015; Larsen et al., 2015) to natural language processing (Mnih & Gregor, 2014; Miao et al., 2015; Bowman et al., 2015; Serban et al., 2016b). Furthermore, these models naturally incorporate a Bayesian modeling perspective, by enabling the integration of problem-dependent knowledge in the form of a prior on the generating distribution.
However, the majority of models proposed assume an extremely simple prior in the form of a multivariate Gaussian distribution in order to maintain mathematical and computational tractability. Although this assumption on the prior has lead to favorable results on several tasks, it is clearly a restrictive and often unrealistic assumption. First, it imposes a strong uni-modal structure on the latent variable space; latent samples from the generating model (prior distribution) all cluster around a single mean. Second, it encourages local smoothness on the latent variables; the similarity between two latent variables decreases exponentially as their distance increase. Thus, for complex, multi-modal distributions — such as the distribution over topics in a text corpus, or natural language responses in a dialogue system — the uni-modal Gaussian prior inhibits the model’s ability to extract and represent important structure in the data. To learn more powerful and expressive models — in particular, models with multi-modal latent variable structures for natural language processing applications — we seek a suitable and flexible prior than can be automatically adapted to model multiple modes of a target distribution.
∗First two authors contributed equally.
In this paper, we propose the multi-modal variational encoder-decoder framework, introducing an efficient, flexible prior distribution that is suitable for distributions such as those found in natural language text. We demonstrate the effectiveness of our multi-modal variational architectures in two representative tasks: document modeling and dialogue modeling. We find that our prior is able to capture elements of a target distribution that simpler priors — such as the uni-modal Gaussian — cannot model, thus allowing neural latent variable models to extract richer structure from data. In particular, we achieve state-of-the-art results on several document modeling tasks.
2 RELATED WORK
The idea of using an artificial neural network to approximate an inference model dates back to the 90s (Hinton & Zemel, 1994; Hinton et al., 1995; Dayan & Hinton, 1996). However, initial attempts at such an approach were hindered by the lack of low-bias, low-variance estimators of parameter gradients. Traditionally, researchers resorted to Markov chain Monte Carlo methods (MCMC) (Neal, 1992) which do not scale well and mix slowly, or to variational approaches which require a tractable, factored distribution to approximate the true posterior distribution, usually under-fitting it (Jordan et al., 1999). Others have since proposed using feed-forward inference models to efficiently initialize the mean-field inference algorithm for incrementally training Boltzmann architectures (Salakhutdinov & Larochelle, 2010; Ororbia II et al., 2015b). However, these approaches are limited by the mean-field inference’s inability to model structured posteriors. Recently, Mnih & Gregor (2014) proposed the neural variational inference and learning (NVIL) approach to match the true posterior directly without resorting to approximate inference. NVIL allows for the joint training of an inference network and directed generative model, maximizing a variational lower-bound on the data log-likelihood and facilitating exact sampling of the variational posterior. Simultaneously with this work, the variational autoencoder framework was proposed by Kingma & Welling (2013) and Rezende et al. (2014). This framework is the motivation of this paper, and will be discussed in detail in the next section.
With respect to document modeling, it has recently been demonstrated that neural architectures can outperform well-established, standard topic models such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003). For example, it has been demonstrated that models based on the Boltzmann machine, which learn semantic binary vectors (binary latent variables), perform very well (Hofmann, 1999). Work involving discrete latent variables include the constrained Poisson model (Salakhutdinov & Hinton, 2009), the Replicated Softmax model (Hinton & Salakhutdinov, 2009) and the Over-Replicated Softmax model (Srivastava et al., 2013), as well as similar, auto-regressive neural architectures and deep directed graphical models (Larochelle & Lauly, 2012; Uria et al., 2014; Lauly et al., 2016; Bornschein & Bengio, 2014). In particular, Mnih & Gregor (2014) showed that using NVIL yields better generative models of documents than these previous approaches. The success of these discrete latent variable models — which are able to partition probability mass into separate regions — serve as the main motivation for investigating models with continuous multi-modal latent variables for document modeling. More recently, Miao et al. (2015) have proposed continuous latent variable representations for document modeling, which has achieved state-of-the-art results. This model will be described later.
With respect to dialogue modeling, latent variable models were investigated by Bangalore et al. (2008), Crook et al. (2009) as well as others. More recently, Zhai & Williams (2014) have proposed three models combining hidden Markov models and topic models. The success of these discrete latent variable models also motivates our investigation into dialogue models with multi-modal latent variables. Most related to our work is the Variational Hierarchical Recurrent Encoder-Decoder (VHRED) model by Serban et al. (2016b), which is a neural architecture with latent multivariate Gaussian variables. This model will be described later.
There has been some work exploring alternative distributions for the latent variables in the variational autoencoder framework, including multi-modal distributions. Rezende & Mohamed (2015) propose an approach called normalizing flows which computes a more complex, potentially multimodal distribution, by projecting standard Gaussian variables through a sequence of non-linear transformations. This approach is similar to the inverse auto-regressive flow proposed by Kingma et al. (2016). Unfortunately, both normalizing flows and auto-regressive flow are only applicable to the approximate posterior distribution; typically these approaches require fixing the prior distri-
bution to a uni-modal multivariate Gaussian. Furthermore, to the best of our knowledge, neither of these approaches have been investigated in the context of larger scale text processing tasks, such as the document modeling and dialogue modeling tasks we evaluate on. A complementary approach is to combine variational inference with MCMC sampling (Salimans et al., 2015; Burda et al., 2015), however this is computationally expensive and therefore difficult to scale up to many real-world tasks. Enriching the latent variable distributions has also been investigated by Maaløe et al. (2016).
2.1 APPROACHES FOR LEARNING MULTI-MODAL LATENT VARIABLES
Mixture of Gaussians Perhaps the most direct and naive approach to learning multi-modal latent variables is to parametrize the latent variable prior and approximate posterior distributions as a mixture of Gaussians. However, the KL divergence between two mixtures of Gaussian distributions cannot be computed in closed form (Durrieu et al., 2012). To train such a model, one would have to either resort to MCMC sampling, which may slow down and hurt the training process due to the high variance it incurs, or resort to approximations of the KL divergence, which may also hurt the training process.1
Deep Directed Models An alternative to a mixture of Gaussians parametrization is to construct a deep directed graphical model composed of multiple layers of uni-modal latent variables (e.g. multivariate Gaussians) (Rezende et al., 2014). Such models have the potential to capture highly complex, multi-modal latent variable representations through the marginal distribution of the toplayer latent variables. However, this approaches has two major drawbacks. First, the variance of the gradient estimator grows with the number of layers. This makes it difficult to learn highly multimodal latent representations. Second, it is not clear how many modes such models can represent or how their inductive biases will affect their performance on tasks containing multi-modal latent structure. The piecewise constant latent variables we propose do not suffer from either of these two drawbacks; the piecewise constant variables incur low variance in the gradient estimator, and can, in principle, represent a number of modes exponential in the number of latent variables.
Discrete Latent Variables A third approach for learning multi-modal latent representations is to instead use discrete latent variables as discussed above. For example, the learning procedure proposed by Mnih & Gregor (2014) for discrete latent variables can easily be combined with the variational autoencoder framework to learn models with both discrete and continuous latent variables. However, the major drawback of discrete latent variables is the high variance in the gradient estimator. Without further approximations, it might be difficult to scale up models with discrete latent variables for real-world tasks.
3 THE MULTI-MODAL VARIATIONAL ENCODER-DECODER FRAMEWORK
We start by describing the general neural variational learning framework. Then we present our proposed prior model aimed at enhancing the model’s ability to learn multiple modes of data distributions. We focus on modeling discrete output variables in the context of natural language processing applications. However, the framework can easily be adapted to handle continuous output variables, such as images, video and audio.
3.1 NEURAL VARIATIONAL LEARNING
Let w1, . . . , wN be a sequence of N words conditioned on a continuous latent variable z. In the general framework, the distribution over the variables follows the directed graphical model:
Pθ(w1, . . . , wN , z) = ∫ N∏ n=1 Pθ(wn|w<n, z)Pθ(z)dz, (1)
where θ are the model parameters. The model first generates the higher-level, continuous latent variable z, and then, conditioned on this, generates the word sequence. The document modeling
1Our lab has previously investigated incorporating mixture of Gaussian models into the autoencoder framework, but without any success. This work has not been published.
task further simplifies the model by assuming the words are independent of each other:
Pθ(w1, . . . , wN , z) = ∫ N∏ n=1 Pθ(wn|z)Pθ(z)dz. (2)
Following the variational autoencoder (VAE) framework (Kingma & Welling, 2013), the parameters can be learned using the variational lower-bound: logPθ(w1, . . . , wN , z) ≥ Ez∼Qψ(z|w1,...,wN )[logPθ(wn|w<n, z)]− KL [Qψ(z|w1, . . . , wN )||Pθ(z)] , (3) where Qψ(z|w1, . . . , wN ) is the approximation to the posterior for z, called the encoder, or sometimes the recognition model or inference model, with parameters ψ. The distribution Pθ(z) is the prior model for z. The variational autoencoder model further makes use of the re-parametrization trick, which allows one to move the derivative of the lower-bound to inside the expectation. To accomplish this, we need to parametrize z as a transformation from a fixed (parameter-less) random distribution:
z = fθ( ), (4) where is drawn from a random distribution, e.g. a standard Gaussian distribution (with zero mean and unit standard deviation) or a uniform distribution in the interval [0, 1], and f is some transformation of this variable, also parametrized by θ.
The majority of work on VAEs that uses the re-parametrization trick propose to parametrize z — both the prior and approximate posterior (encoder) — as a multivariate Gaussian variable. However, the multivariate Gaussian is a uni-modal distribution and can therefore only represent one mode in latent space. This means the mapping from latent variable to outputs — i.e. the conditional distribution Pθ(wn|z) — has to be highly non-linear in order to capture additional modes. However, in general, it is difficult to learn such non-linear mappings with existing stochastic optimization methods, such as mini-batch stochastic gradient descent and its variants. Learning such a non-linear mapping is particularly difficult using the variational bound in eq. (3), because it incurs additional variance from sampling the latent variable z. Consequently, such a model is very likely to converge on a solution which does not model multi-modality which then leads to a poor approximation of the output distribution.
3.2 THE PIECEWISE-CONSTANT PRIOR FOR LATENT VARIABLES
In this work, we overcome the uni-modal restriction by parametrizing z using a piecewise constant probability density function (PDF). This parametrization will allow z to represent complex aspects of the data distribution in latent variable space, such as multiple modes and highly non-smooth regions of probability mass. From a manifold learning perspective, this extension translates into expanding the set of manifolds representable by the model parameters to include more non-linear manifolds – in particular, manifolds where there exists separate clusters of probability mass.
Let n ∈ N be the number of piecewise constant components. We assume z is drawn from the PDF:
P (z) = 1
K n∑ i=1 1( i− 1 n ≤z≤ i n
)ai, (5) where 1(x) is the indicator function (which is one whenever x is true and otherwise zero), ai > 0 for i = 1, . . . , n are the distribution parameters (which will be learned during training), and K is the normalization constant:
K = n∑ i=1 Ki, where K0 := 0,Ki := ai n for i = 1, . . . , n. (6)
To train the model using the re-parametrization trick, we need to generate z = f( ) where ∼ Uniform(0, 1). To do so, we employ inverse transform sampling (Devroye, 1986), which requires finding the inverse of the cumulative distribution function (CDF). We first derive the CDF of eq. (5):
φ(z) = 1
K n∑ i=1 1( i n ≤z )Ki + 1( i− 1 n ≤z≤ i n )(z − i− 1 n ) ai. (7)
Next, we derive its inverse:
φ−1( ) = n∑ i=1 1( 1 K ∑i−1 j=0Kj≤ ≤ 1 K ∑i j=0Kj
)  i− 1
n + K
ai
 − 1 K i−1∑ j=0 Kj  (8) Armed with the inverse CDF, we can now draw a sample z:
z = φ−1( ), where ∼ Uniform(0, 1). (9)
In addition to sampling, we need to compute the Kullback-Leibler (KL) divergence between the prior and approximate posterior distributions of the piecewise constant variables. We assume both the prior and the posterior are piecewise constant distributions. We use prior to denote prior parameters and post to denote posterior parameters (encoder model parameters). The KL divergence between the prior and posterior can be computed using a sum of integrals, where each integral inside the sum corresponds to one constant segment:
KL [Qψ(z|w1, . . . , wN )||Pθ(z)] = ∫ 1 0 Qψ(z|w1, . . . , wN ) log ( Qψ(z|w1, . . . , wN ) Pθ(z) ) dz (10)
= n∑ i=1 ∫ 1/n 0 aposti Kpost log ( aposti /K post apriori /K prior ) dz (11)
(12)
= 1
n n∑ i=1 aposti Kpost log
( aposti /K post
apriori /K prior
) (13)
= 1
n
1
Kpost n∑ i=1 aposti ( log(aposti )− log(a prior i ) ) (14)
+ log(Kprior)− log(Kpost)
In order to train the model, we take partial derivatives of the variational bound in eq. (3) w.r.t. each parameter in θ and ψ. These expressions involve derivatives of the indicator functions, which have derivatives zero everywhere except for the changing points where the derivative is undefined. However, the probability of sampling such that an indicator function is exactly at its changing point is effectively zero. Therefore, we fix their derivatives to zero.2 A similar approach is used for training neural networks with rectified linear units. Figure 1 illustrates how the piecewise constant latent variables can work with Gaussian latent variables in order to model multi-modality.
4 LATENT VARIABLE PARAMETRIZATIONS
The latent variable parametrizations are crucial to modeling the data effectively. In this section, we will develop the parametrizations for both the Gaussian variable and our proposed piecewise latent variable.
For all parametrizations, let c be the conditioning information for the prior. In document modeling there is no conditioning information available to the prior, so c = ∅. In dialogue modeling c is the vector representation of the dialogue context, namely all previous utterances until the current time step. Let x be the current output sequence (observation), which the model must generate (e.g. w1, . . . , wN for document modeling).
4.1 GAUSSIAN PARAMETRIZATION
Let µprior and σ2,prior be the prior mean and variance, and let µpost and σ2,post be the posterior mean and variance. For Gaussian latent variables, the prior distribution mean and variances are encoded using linear transformations of a hidden state. In particular, the prior distribution covariance is
2We thank Christian A. Naesseth for pointing out this assumption.
encoded as a diagonal covariance matrix using a softplus function:
µprior = Hpriorµ Enc(c) + b prior µ , (15)
σ2,prior = diag(log(1 + exp(Hpriorσ Enc(c) + b prior σ ))), (16)
where Enc(c) is an embedding/encoding of the context c (e.g. given by a bag-of-words encoder or an LSTM encoder applied to c), which is shared across all latent variable dimensions. The parameters Hpriorµ , b prior µ , H prior σ , b prior σ are to be learned.
For the posterior distribution, our preliminary experiments have shown that it is much better to parametrize the posterior distribution by interpolating between the prior distribution mean and variance and a new estimate of the mean and variance. This interpolation is controlled by a gating mechanism, which makes it easy for the model to learn how to turn on/off latent dimensions:
µpost = (1− αµ)µprior + αµ ( Hpostµ Enc(c, x) + b post µ ) , (17)
σ2,post = (1− ασ)σ2,prior + ασdiag(log(1 + exp(Hpostσ Enc(c, x) + bpostσ ))), (18)
where Enc(c, x) is an encoding/embedding of both c and x, and where the parameters are Hpostµ , b post µ , H post σ , b post σ , αµ, ασ . The interpolation mechanism is controlled by αµ and ασ , which are initialized to zero (i.e. initialized such that the posterior is equal to the prior).3
4.2 PIECEWISE CONSTANT PARAMETRIZATION
Similar to the Gaussian variances, we propose to parametrize the piecewise constant prior parameters using an exponential function applied to a linear transformation of the context embedding/encoding:
apriori = exp(H prior a,i Enc(c) + b prior a,i ), i = 1, . . . , n, (19)
where Hpriora and b prior a are the parameters to be learned.
We may also constrain the piecewise constant posterior parameters to be an interpolation between the prior parameters and a new estimated parameter:
aposti = (1− αa,i)a prior i + αa,i exp(H post a,i Enc(c, x) + b post a,i ), i = 1, . . . , n, (20)
3We experimented with more sophisticated mechanisms for controlling the gating variables, including defining αµ and ασ to be a linear function of the encoder. However, we found that simpler was often better and thus do not report these results using more advanced mechanisms.
whereHposta , b post a , αa are the parameters. However, we found that this interpolation hurt performance and therefore fixed αa = 1.
To take advantage of the properties of both priors, the Gaussian and piecewise constant variables may be combined, as was suggested in Section 3.2. In this work, we primarily experimented with their concatenation to create a hybrid model.
5 VARIATIONAL TEXT MODELING
We now present two probabilistic models, the NVDM and the VHRED, which are extended to incorporate the latent variable parametrization and used for the document modeling and the dialogue modeling experiments described below.
5.1 NEURAL VARIATIONAL DOCUMENT MODEL (NVDM)
The NVDM framework (Mnih & Gregor, 2014; Miao et al., 2015) collapses the recurrent neural encoder into a simpler bag-of-words model (since no symbol order is taken into account), which may be defined as a multi-layer perceptron (MLP) for Enc(c = ∅, x) = Enc(x). Let V be the vocabulary. Let W represent a document matrix, where row wi is the 1-of-|V | binary encoding of the i’th word in the document. Enc(W ) is trained to compress a document vector into a continuous distributed representation upon which the posterior model is built.
The NVDM parametrization requires only learning the parameters bpriora ,W post a , b post a for the piecewise variables, and learning the parameters bpriorµ , b prior σ ,W post µ , b post µ ,W post σ , b post σ for the Gaussian variables. We initialize the bias parameters to zero, in order for the NVDM to start with a centered Gaussian prior. This prior will be adapted by the parametric encoder as learning progresses, while also learning to turn on/off latent dimensions controlled through the gating mechanism. It is important to note that our particular instantiation of the NVDM is different from that of Mnih & Gregor (2014) and Miao et al. (2015); we jointly learn the prior mean and variance whereas in previous work it has been assumed to be a standard Gaussian. Furthermore, our models learn to interpolate between the generated prior and posterior models to calculate a new posterior.
Based on preliminary experiments, we choose the encoder to be a 2-hidden layer perceptron, defined by parameters {E0, b0, E1, b1}. The decoder is defined by parameters {R, c}. For example, in the case of the hybrid VAE we use eq. (15)–(20) to generate the distribution parameters. In this case, to draw a sample from the Gaussian prior, we draw a standard Gaussian variable and then multiply it by the standard deviation and add the mean of the Gaussian prior. To draw a sample from the piecewise prior, we use eq. (8). As such, the complete architecture is:
π(W ) = f0(E0W + b0),
Enc(W ) = f1(E1π(W ) + b1),
zGaussian = µ post + √ σ2,post ⊗ 0,
zPiecewise = φ −1,post( 1),
z = 〈zGaussian, zPiecewise〉, Dec(w, z) = g(−wTRz),
where⊗ is the Hadamard product, 〈◦, ◦〉 is an operator that combines the Gaussian and the Piecewise variables and Dec(w, z) is the decoder model. 4 As a result of using the re-parametrization trick and choice of prior, we calculate the latent variable z through the two samples, 0 and 1. f(◦) is a non-linear activation function. We choose it to be the softsign function, or f(v) = v/(1 + |v|). The decoder model Dec(z) outputs a probability distribution over words conditioned on z. In this case, we define g(◦) as the softmax function (omitting the bias term c for clarity) computed as:
Dec(w, z) = Pθ(w|z) = exp (−wTRz)∑ w′ exp (−wTRz) ,
4Operations include vector concatenation, summation, or averaging.
The decoder’s output is used to calculate the first term in the variational lower-bound: logPθ(W |z). The prior and posterior distributions are used to compute the KL term in the variational lower-bound. The lower-bound defined becomes:
L = EQψ(z|W ) [ N∑ i=1 logPθ(wi|z) ] − KL [Qψ(z|W )||Pθ(z)] ,
where the KL term is the sum of the Gaussian and piecewise KL-divergence measures:
KL [Q(z|W )||P (z)] = KLGaussian [Q(z|W )||P (z)] + KLPiecewise [Q(z|W )||P (z)] .
The KL-terms may be interpreted as regularizers of the parameter updates for the encoder model (Kingma & Welling, 2013). These terms encourage the posterior distributions to be similar to their corresponding prior distributions, by limiting the amount of information the encoder model transmits regarding the output. For example, it encourages the uni-modal Gaussian posterior to move its mean close to the mean of the Gaussian prior, which makes it difficult for the Gaussian posterior to represent different modes conditioned on the observation. Similarly, this encourages the piecewise constant posterior to be similar to the piecewise constant prior. However, since the piecewise constant posterior is multi-modal, it may be able to shift some of its probability mass towards the prior distribution while keeping other probability mass on one or several modes dependent upon the output observation (e.g. if the prior distribution is a uniform distribution and the true posterior concentrates all its probability mass in several small regions, then the approximate posterior could interpolate between the prior and the true posterior).
5.2 VARIATIONAL HIERARCHICAL RECURRENT ENCODER-DECODER (VHRED)
The VHRED model is an extension of the hierarchical recurrent encoder-decoder model (HRED) for dialogue (Serban et al., 2016b;a). The model decomposes dialogues using a two-level hierarchy: sequences of utterances (e.g. sentences), and sub-sequences of tokens (words). Let wn be the n’th utterance in a dialogue with N utterances. Let wn,m be the m’th word in the n’th utterance from vocabulary V , and let Mn be the number of words in the n’th utterance. In addition to this, VHRED has a latent multivariate continuous variable zn for each utterance n = 1, . . . , N . The probability distribution of the generative model factorizes as:
Pθ(w1, . . . ,wN ) = N∏ n=1 Pθ(wn|w<n, zn)Pθ(zn|w<n),
= N∏ n=1 Mn∏ m=1 Pθ(wn,m|wn,<m,w<n, zn)Pθ(zn|w<n), (21)
where θ are the model parameters. VHRED uses three RNN modules: an encoder RNN, a context RNN and a decoder RNN. First, each utterance is encoded into a vector by the encoder RNN:
hencn,0 = 0, h enc n,m = f enc θ (h enc n,m−1, wn,m) ∀m = 1, . . . ,Mn,
where f encθ is either a GRU or a bidirectional GRU function. The last hidden state of the encoder RNN is given as input to the context RNN. Then, the context RNN updates its internal hidden state to reflect all the information up until that utterance:
hcon0 = 0, h con n = f con θ (h con n−1, h enc n,Mn),
where f conθ is a GRU function taking as input two vectors. This state is used to compute the prior distribution over the latent variable zn:
Pθ(zn | w<n) = f priorθ (h con n−1), (22)
where f prior is a PDF parametrized by both θ and hconn . Next, a sample is drawn from this distribution: zn ∼ Pθ(zn|w<n). The sample and context state are given as input to the decoder RNN:
hdecn,0 = 0, h dec n,m = f dec θ (h dec n,m−1, h con n−1, zn, wn,m)
∀m = 1, . . . ,Mn,
where f decθ is the LSTM gating function taking as input four vectors. The output distribution is computed by passing hdecn,m through an MLP f mlp θ , an affine transformation and a softmax function:
Pθ(wn,m+1|wn,≤m,w<n, zn) = e(Own,m+1) Tfmlpθ (h dec n,m)∑
w′ e (Ow′)Tfmlpθ (h dec n,m)
, (23)
whereO ∈ R|V |×d is the word embedding matrix for the output distribution with embedding dimensionality d ∈ N. The model is trained by maximizing the variational lower-bound, which factorizes into independent terms for each sub-sequence (utterance):
logPθ(w1, . . . ,wN ) ≥ N∑ n=1 − KL [Qψ(zn | w1, . . . ,wn)||Pθ(zn | w<n)]
+ EQψ(zn|w1,...,wn) [logPθ(wn | zn,w<n)] , (24) where distribution Qψ is the approximate posterior distribution with parameters ψ, which is computed similar to the prior distribution but further conditioned on the future encoder RNN hidden state:
Qψ(zn | w≤n) = f postψ (h con n−1, h enc n,Mn), (25)
where f post is a PDF. More details are given by Serban et al. (2016b).
The original VHRED model as described by Serban et al. (2016b) used only Gaussian latent variables. We will refer to this model as Gaussian-VHRED (G-VHRED). The VHRED model with both Gaussian and piecewise constant latent variables will be referred to as Hybrid-VHRED (HVHRED). In this case, we combine the Gaussian and piecewise latent variables by concatenating them into one vector.5
6 EXPERIMENTS
In order to validate the ability of our piecewise latent variables to capture complex aspects of data distributions, we conduct experiments with both the NVDM and VHRED models.
All models are trained using back-propagation to obtain parameter gradients with respect to the variational lower-bound on the log-likelihood or the exact log-likelihood. We used a standard firstorder gradient-descent optimizer, Adam (Kingma & Ba, 2015), for both models, where only hyperparameter choices varied depending on the task. The specifics of the design of the encoder and decoder differed between the two tasks (as described in Sections 5.1 and 5.2). For all models that used piecewise latent variables, we chose to fix αai = 1, meaning the piecewise prior and posterior models are kept separate (instead of having the posterior be an interpolation between another distribution and the prior), since we found this to perform better6
6.1 DOCUMENT MODELING
For our experiments in document modeling, we make use of the 20 News-Groups dataset. We follow the pre-processing and set-up of Hinton & Salakhutdinov (2009). In addition, we make use of the Reuters corpus (RCV1-V2), using a version that contained a selected 5,000 term vocabulary. 7 Note that the features are a log(1 + TF ) transform of the original frequency vectors. To test our document models on text from another language (in this case, Brazilian Portuguese), we make use of the CADE12 dataset (stop-word removed and stemmed) Cardoso-Cachopo (2007), where we further filtered terms that occurred less than 130 times to obtain a vocabulary of 3,736 terms (over 26,991 training and 13,486 test documents). For all datasets, we track the validation bound on a subset of 100 vectors randomly drawn from each training corpus.
5Before concatenation, we transform the piecewise constant latent variables to lie within the interval [−1, 1]: z′ = 2z− 1. This ensures the input to the decoder RNN has mean zero at the beginning of training.
6We believe that if αai = 0 for a long period of time, then the posterior receives no gradient signal. Without a gradient signal, the estimated posterior becomes increasingly disconnected from the rest of the model and, thus, less effective. This might be due to the choice of non-linearities, which affect the piecewise latent variables moreso than the Gaussian latent variables.
7We will make the code and scripts used to create the final document input vectors and vocabulary files publicly available upon publication.
For the Gaussian NVDM (G-NVDM), we constrain the interpolated posterior variance to lie in the range of [0.01, 10.0]. For the hybrid NVDMs (H-NVDM) 8, we vary the number of components used in the PDF, investigating the effect that 3 and 5 pieces had on the final quality of the model. Parameter updates for all models were estimated using mini-batches of 100 samples drawn randomly without replacement from the training data over 150 epochs. A learning rate of 0.002 was used. Model selection and early stopping (the only additional form of regularization employed for this set of experiments) were conducted using the validation lower-bound, estimated using five stochastic samples per validation example. We rescale large gradients by their norm (Pascanu et al., 2012). Inference networks made use of 50 units in each hidden layer for 20 News-Groups and CADE and 100 for RCV1, while all performed best with 50 latent variables (chosen via preliminary experimentation with smaller models). On the 20 News-Groups, since we were able to use the same set-up (especially vocabulary) as Hinton & Salakhutdinov (2009), we also report the perplexities of a topic model (LDA, Hinton & Salakhutdinov (2009)), the Replicated Softmax (RSM, Hinton & Salakhutdinov (2009)), the document neural auto-regressive estimator (docNADE, Larochelle & Lauly (2012)), a sigmoid belief network (SBN, Mnih & Gregor (2014)), a deep auto-regressive neural network (fDARN, Mnih & Gregor (2014)), and a neural variational document model with a fixed standard Gaussian prior (NVDM, lowest reported perplexity, Miao et al. (2015)).
In Table 1, we report the test document perplexity (under the Sampled column), calculated using the standard formula, exp(− 1D ∑ n 1 Ln
logPθ(xn). Note that logPθ(xn), or the log-probability of a particular document, was approximated with an estimate of the variational lower-bound using 10 samples, as was done in Mnih & Gregor (2014). The second score (or column SGD-Inf ), refers to the model’s test-perplexity when the lower-bound is tightened using iterative inference to search for the optimal latent variable per document. In this paper, our iterative inference procedure consisted of simple stochastic gradient descent (no more than 100 steps), with a learning rate of 0.1 and the same
8We ultimately found that averaging the variables, as opposed to using concatenation, yielded best perplexity and thus report these results.
gradient rescaling used in training, using early-stopping (for 20 News-Groups, the lookahead was 10, while on Reuters and CADE the lookahead was 5). The parameters of the model, as well as the well as the generated prior, are fixed, and the gradients of the variational lower bound with respect to generated posterior model parameters (i.e., the mean and variance of the Gaussian variables, and the piecewise components, ai) are used to update the posterior model for each document (using a freshly drawn sample each step).
First and foremost, we note that the best baseline model (i.e., the NVDM) is more competitive when both the prior and posterior models are learnt together (i.e., the G-NVDM), as opposed to the fixed prior of Miao et al. (2015). However, we observe that integrating our proposed piecewise variables yields even better results in our document modeling experiments, substantially improving over the baselines. More importantly, in some cases, as in the 20 News-Groups and Reuters datasets, increasing the number of pieces from 3 to 5 can further reduce perplexity. Thus, we have achieved a new state-of-the-art perplexity on 20 News-Group task and — to the best of our knowledge – better perplexities on the CADE12 and RCV1 tasks compared to using a state-of-the-art model like the GNVDM. Furthermore, we observe iterative inference yields yet a further boost in performance since the bound estimated is tighter, however, this form of inference is expensive and requires additional meta-parameters (e.g., a step-size, an early-stopping criterion, etc.). We remark a simpler, and more accurate, approach to inference would be to use importance sampling.
In Table 2, we examine the top ten highest ranked words given a query term, using the decoder parameter matrix (since the decoder is directly affected by the latent variables in our document models). It appears that the piecewise variables affect what is uncovered by the model with respect to the data, as each model returns different, but relevant results with respect to the query word. In our current examples, it appears that the H-NVDM with 5 pieces returns more general words. For example, in the case of “government”, the baseline seems to value the plural form of the word (which is largely based on morphology) while the hybrid model actually pulls out meaningful terms such as “federal”, “policy”, and “administration”. The case of “space” is interesting–the hybrid with 5 pieces seems to value two senses of the word–one related to “outer space” (e.g., “sun”, “world”, etc.) and another related to the dimensions of depth, height, and width within which things may exist and move (e.g., “area”, “form”, “scale”, etc.).
6.2 DIALOGUE MODELING
We experiment with VHRED for dialogue modeling. This is a difficult problem, extensively studied in the recent literature (Ritter et al., 2011; Lowe et al., 2015; Sordoni et al., 2015; Li et al., 2016; Serban et al., 2016a). Related systems for dialogue response generation have recently gained a significant amount of attention from industry, with high-profile projects such as Google’s SmartReply system (Kannan et al., 2016) and Microsoft’s chatbot Xiaolice (Markoff & Mozur, 2015). Even more recently, Amazon has announced the Alexa Prize Challenge for the research community with the goal of developing a natural and engaging chatbot system (Farber, 2016).
We focus on non-goal-driven dialogue modeling and use the Twitter Dialogue Corpus (Ritter et al., 2011) based on public Twitter conversations. The dataset is split into training, validation, and test sets, containing respectively 749,060, 93,633 and 9,399 dialogues each. On average, each dialogue contains about 6 utterances (dialogue turns) and about 94 words. The dataset is the same as used by Serban et al. (2016b), but further pre-processed using byte-pair encoding (Sennrich et al., 2016) using a vocabulary consisting of 5000 sub-words.9 The dialogues are substantially longer than recent large-scale language modeling corpora, such as the 1 Billion Word Language Model Benchmark (Chelba et al., 2014), which usually focus on modeling single sentences.
Parameter optimization was conducted with a learning rate of 0.0002 and mini-batches of size 40 or 80.10 We use a variant of truncated back-propagation and apply gradient clipping (Pascanu et al., 2012). Model selection and early stopping — the only additional form of regularization employed for this set of experiments — are conducted using the validation lower-bound, estimated using one stochastic sample per validation example.
9In addition to applying byte-pair encoding, we filtered out 601 test dialogues so that no test dialogue context overlapped with the training or validation sets.
10We had to vary the mini-batch size to make the training fit on GPU architectures with low memory.
Similar to Serban et al. (2016b), we use a bidirectional GRU RNN encoder, where the forward and backward RNNs each have 1000 hidden units. We experiment with context RNN encoders with 500 and 1000 hidden units, and find that that 1000 hidden units reach better performance w.r.t. the variational lower-bound on the validation set. The encoder and context RNNs use layer normalization (Ba et al., 2016). We experiment with decoder RNNs with 1000, 2000 and 4000 hidden units (LSTM cells), and find that 2000 hidden units reach better performance. For the G-VHRED model, we experiment with latent multivariate Gaussian variables with 100 and 300 dimensions, and find that 100 dimensions reach better performance. For the H-VHRED model, we experiment with latent multivariate Gaussian and piecewise constant variables each with 100 and 300 dimensions, and find that 100 dimensions reach better performance. We follow the training procedure of Serban et al. (2016b): we drop words in the decoder with a fixed drop rate of 25% and multiply the KL terms in the variational lower-bound by a scalar, which starts at zero and linearly increases to 1 over the first 60,000 training batches.
We also experiment with an LSTM baseline model and a HRED baseline model (Serban et al., 2016a). For the LSTM model, we experiment with number of hidden units (LSTM cells) equal to 1000, 2000 and 4000 and find that 4000 hidden units perform best w.r.t. validation perplextiy. For the HRED model, we use the same encoder and context RNN architectures as the G-VHRED and H-VHRED models described earlier. We set the encoder RNN to have 1000 hidden units. We experiment with a context RNN with 500 and 1000 hidden units, and find that 1000 hidden units reach better performance. For the decoder RNN, we experiment with 1000 and 2000 hidden units (LSTM cells) and find that 2000 hidden units perform better.
Approximate Posterior Analysis Our hypothesis is that the piecewise constant latent variables are able to capture multi-modal aspects of the dialogue. Therefore, we evaluate the models by analyzing what information they have learned to represent in the latent variables. For each test dialogue with n utterances, we condition each model on the first n − 1 utterances and compute the latent posterior distributions using all n utterances. We then compute the gradients of the KL terms of the multivariate Gaussian and piecewise constant latent variables w.r.t. each word in the dialogue. Since the words vectors are discrete, we compute the sum of the squared gradients w.r.t. each word embedding. The higher the sum of the squared gradients of a word is, the more influence it will have
on the posterior approximation (encoder model). For every test dialogue, we count the top 5 words with highest squared gradients separately for the multivariate Gaussian and piecewise constant latent variables.11
The results are shown in Table 3. The piecewise constant latent variables clearly capture different aspects of the dialogue compared to the Gaussian latent variables. The piecewise constant variable approximate posterior encodes words related to time (e.g. weekdays and times of day) and events (e.g. parties, concerts, Easter). On the other hand, the Gaussian variable approximate posterior encodes words related to sentiment (e.g. laughter and appreciation) and acronyms, punctuation marks and emoticons (i.e. smilies). We also conduct a similar analysis on the document models evaluated in Sub-section 6.1, the results of which may be found in the Appendix.
Response Evaluation Non-goal-driven dialogue models are typically evaluated by asking humans to rate the quality of different responses. We follow the approach by Liu et al. (2016) by conducting an Amazon Mechanical Turk experiment to compare the G-VHRED and H-VHRED models. For each test dialogue, we use TF-IDF to extract 100 candidate responses (Lowe et al., 2015). We then rank the responses according to the G-VHRED model and H-VHRED model using the variational lowerbound.12 We ask three human evaluators to rate model responses for 45 dialogues on a Likert-type scale 1 − 5, with 1 representing an inappropriate response and 5 representing a highly appropriate response.13 For each dialogue, we show the human evaluators the top two responses ranked by the G-VHRED and H-VHRED models. We choose to evaluate the re-ranked responses for two reasons. First, it reduces variance in the output because it uses the approximate posterior model, compared to using beam search with samples from the high-entropy prior. Second, it decreases the number of generic responses, which are extremely common among generative models and which human evaluators tend to prefer despite not advancing the dialogue (Li et al., 2016).
The results are as follows. The G-VHRED model achieves scores 1.88 and 2.13 for the first and second ranked responses on average, and the H-VHRED model achieves scores 1.93 and 2.04 on average. In other words, H-VHRED performs nominally better on the first ranked response while G-VHRED performs nominally better on the second ranked response. In conclusion, if there exists a difference between the two models, naive human evaluators cannot see it.
Although naive human evaluators cannot distinguish between the model responses, based on our previous analysis we know that the two models encode different aspects of dialogue conversations. Therefore, we further investigate the probability of different responses to dialogue contexts related to time and events. Two examples are shown in Figure 2, where the dialogue contexts are “when do you want to meet this weekend?” and “where are you going tomorrow?”. H-VHRED assigns substantially more probability mass to relevant words compared to the G-VHRED as well as an LSTM baseline and HRED baseline. This confirms the ability of the piecewise constant latent variable to generate responses related to time and events.
Finally, we also evaluate the diversity of the G-VHRED and H-VHRED model outputs w.r.t. the top ranked FF-IDF candidate responses. We measure the average word entropy (Serban et al., 2016b) as well as number of unique words for each response and unique words across all test responses, but did not find a significant difference between the two models. This indicates that the Gaussian latent variables alone are able to increase response diversity, while the piecewise constant latent variables instead help encode specific aspects of the dialogue such as time and events.
11Our approach is equivalent to counting the top 5 words with the highest L2 gradient norms. We also did some experiments using L1 gradient norms, which showed similar patterns.
12We use one stochastic sample. 13Human evaluators are only given a minimal description of the task, without any examples, before beginning
the evaluation.
7 CONCLUSIONS
In this paper, we have proposed the multi-modal variational encoder-decoder framework. In order to capture complex aspects of unknown data distributions, we developed the piecewise constant prior, which can be efficiently and flexibly adjusted to capture distributions with many modes, such as those over topics. In experiments on document modeling and dialogue modeling, we have shown the effectiveness of our framework in building models capable of learning richer structure from data. In particular, we have demonstrated new state-of-the-art results on several document modeling tasks.
Future work should focus on exploring other natural language processing tasks, where multimodality plays an important role such as modeling technical help dialogues (Lowe et al., 2015) and online debates (Rosenthal & McKeown, 2015), and where additional information is available, such as in semi-supervised document categorization (Ororbia II et al., 2015a). Furthermore, the piecewise variables proposed in this work could prove useful in uncovering interesting and novel information in lesser-explored corpora.
APPENDIX A: ANALYSIS OF DOCUMENT MODEL PIECEWISE VARIABLES
We present an additional analysis of the learned 20 News-Groups document models in order to explore what each set of latent variables might be capturing. To calculate the gradient of the KL terms needed to formulate word scores, we follow the approach described in Sub-section 6.2, however, conditioning only on the (training) document bag-of-words to compute the latent posterior to then calculate the gradient of the KL-terms with respect to each word in the document.
In Table 4, we observe results similar to those of Sub-section 6.2–the piecewise variables capture different aspects of the document data. It is worth noting, in this experiment, that the Gaussian variables alone were originally were sensitive to some of these words. However, in the hybrid model, nearly all of the temporal words that the Gaussian variables were once more sensitive to now more strongly affect the piecewise variables, which themselves also capture all of the words that were originally missed. This might indicate a shift in responsibility in which latent variables the document model decide are more suitable to capture certain aspects of the data. This effect appears to be even stronger in the case of certain nationality-based adjectives (e.g., “american”, “israeli”, etc.). While the G-NVDM could model multi-modality in the data to some degree, this work would be primarily done in the model’s decoder. In the H-NVDM, the piecewise variables provide an explicit mechanism for capturing modes in the unknown target distribution, so it makes sense that the model would learn to use the piecewise variables instead, thus freeing up the Gaussian variables to capture other aspects of the data, as we found was the case with names (e.g., “jesus”, “kent”, etc.).

The authors introduce some new prior and approximate posterior families for variational autoencoders, which are compatible with the reparameterization trick, as well as being capable of expressing multiple modes. They also introduce a gating mechanism between prior and posterior. They show improvements on bag of words document modeling, and dialogue response generation. The original abstract is overly strong in its assertion that a unimodal latent prior p(z) cannot fit a multimodal marginal int_z p(x|z)p(x)dz with a DNN response model p(x|z) ("it cannot possibly capture more complex aspects of the data distribution", "critical restriction", etc).
While the assertion that a unimodal latent prior is necessary to model multimodal observations is false, there are sensible motivations for the piecewise constant prior and posterior. For example, if we think of a VAE as a sort of regularized autoencoder where codes are constrained to "fill up" parts of the prior latent space, then there is a sphere-packing argument to be made that filling a Gaussian prior with Gaussian posteriors is a bad use of code space. Although the authors don't explore this much, a hypercube-based tiling of latent code space is a sensible idea.
As stated, I found the message of the paper to be quite sloppy with respect to the concept of "multi-modality." There are 3 types of multimodality at play here: multimodality in the observed marginal distribution p(x), which can be captured by any deep latent Gaussian model, multimodality in the prior p(z), which makes sense in some situations (e.g. a model of MNIST digits could have 10 prior modes corresponding to latent codes for each digit class), and multimodality in the posterior z for a given observation x_i, q(z_i|x_i). The final type of multimodality is harder to argue for, except in so far as it allows the expression of flexibly shaped distributions without highly separated modes. I believe flexible posterior approximations are important to enable fine-grained and efficient tiling of latent space, but I don't think these need to have multiple strong modes. I would be interested to see experiments demonstrating otherwise for real world data.
I think this paper should be more clear about the different types of multi-modality and which parts of their analysis demonstrate which ones. I also found it unsatisfactory that the piecewise variable analysis did not show different components of the multi-modal prior corresponding to different words, but rather just a separation between the Gaussian and the piecewise variables.
As I mention in my earlier questions, I found it surprising that the learned variance and mean for the Gaussian prior helps so dramatically with G-NVDM likelihood when the powerful networks transforming to and from latent space should make it scale-invariant. Explicitly separating out the contributions of a reimplemented base model, prior-posterior interpolation and the learned prior parameters would strengthen these experiments. Overall, the very strong improvements on the text modeling task over NVDM seem hard to understand, and I would like to see an ablation analysis of all the differences between that model and the proposed one.
The fact that adding more constant components helps for document modeling is interesting, and it would be nice to see more qualitative analysis of what the prior modes represent. I also would be surprised if posterior modes were highly separated, and if they were it would be interesting to explore if they corresponded to e.g. ambiguous word-senses.
The experiments on dialog modeling are mostly negative results, quantitatively. The observation that the the piecewise constant variables encode time-related words and the Gaussian variables encode sentiment is interesting, especially since it occurs in both sets of experiments. This is actually quite interesting, and I would be interested in seeing analysis of why this is the case. As above, I would like to see an analysis of the sorts of words that are encoded in the different prior modes and whether they correspond to e.g. groups of similar holidays or days.
In conclusion, I think the piecewise constant variational family is a good idea, although it is not well-motivated by the paper. The experimental results are very good for document modeling, but without ablation analysis against the baseline it is hard to see why they should be with such a small modification in G-NVDM. The fact that H-NVDM performs better is interesting, though. This paper should better motivate the need for different types of multi-modality, and demonstrate that those sorts of things are actually being captured by the model. As it is, the paper introduces an interesting variational family and shows that it performs better for some tasks, but the motivation and analysis is not clearly focused. To demonstrate that this is a broadly applicable family, it would also be good to do experiments on a more standard datasets like MNIST. Even without an absolute log-likelihood improvement, if the method yielded interpretable multiple modes this would be a valuable contribution.