This work explores hypernetworks: an approach of using one network, also known as a hypernetwork, to generate the weights for another network. We apply hypernetworks to generate adaptive weights for recurrent networks. In this case, hypernetworks can be viewed as a relaxed form of weight-sharing across layers. In our implementation, hypernetworks are are trained jointly with the main network in an end-to-end fashion. Our main result is that hypernetworks can generate non-shared weights for LSTM and achieve state-of-the-art results on a variety of sequence modelling tasks including character-level language modelling, handwriting generation and neural machine translation, challenging the weight-sharing paradigm for recurrent networks.
1 INTRODUCTION
In this work, we consider an approach of using a small network (called a “hypernetwork") to generate the weights for a larger network (called a main network). The behavior of the main network is the same as with any usual neural network: it learns to map some raw inputs to their desired targets; whereas the hypernetwork takes a set of inputs that contain information about the structure of the weights and generates the weights for that layer.
The focus of this work is to use hypernetworks to generate weights for recurrent networks (RNN). In this case, the weights Wt for the main RNN at step t is a function of the input to the hidden state of the main RNN at the previous step ht−1 and the input at the current time step xt. This weight-generation scheme allows approximate weight-sharing across layers of the main RNN.
We perform experiments to investigate the behaviors of hypernetworks in a range of contexts and find that hypernetworks mix well with other techniques such as batch normalization and layer normalization. Our main result is that hypernetworks can generate non-shared weights for LSTM that work better than the standard version of LSTM (Hochreiter & Schmidhuber, 1997). On language modelling tasks with character Penn Treebank, Hutter Prize Wikipedia datasets, hypernetworks for LSTM achieve near state-of-the-art results. On a handwriting generation task with IAM handwriting dataset, hypernetworks for LSTM achieves good quantitative and qualitative results. On machine translation, hypernetworks for LSTM also obtain state-of-the-art performance on the WMT’14 en→fr benchmark.
2 RELATED WORK
Our approach is inspired by methods in evolutionary computing, where it is difficult to directly operate in large search spaces consisting of millions of weight parameters. A more efficient method is to evolve a smaller network to generate the structure of weights for a larger network, so that the search is constrained within the much smaller weight space. An instance of this approach is the work on the HyperNEAT framework (Stanley et al., 2009). In the HyperNEAT framework, Compositional Pattern-Producing Networks (CPPNs) are evolved to define the weight structure of the much larger main network. Closely related to our approach is a simplified variation of HyperNEAT, where the structure is fixed and the weights are evolved through Discrete Cosine Transform (DCT), called Compressed Weight Search (Koutnik et al., 2010). Even more closely related to our
∗Work done as a member of the Google Brain Residency program (g.co/brainresidency).
approach are Differentiable Pattern Producing Networks (DPPNs), where the structure is evolved but the weights are learned (Fernando et al., 2016), and ACDC-Networks (Moczulski et al., 2015), where linear layers are compressed with DCT and the parameters are learned. Most reported results using these methods, however, are in small scales, perhaps because they are both slow to train and require heuristics to be efficient. The main difference between our approach and HyperNEAT is that hypernetworks in our approach are trained end-to-end with gradient descent together with the main network, and therefore are more efficient.
Another closely related idea to hypernetworks is the concept of fast weights Schmidhuber (1992; 1993) in which one network can produce context-dependent weight changes for a second network. Small scale experiments were conducted to demonstrate fast weights for feed forward networks at the time, but perhaps due to the lack of modern computational tools, the recurrent network version was mentioned mainly as a thought experiment (Schmidhuber, 1993). A subsequent work demonstrated practical applications of fast weights (Gomez & Schmidhuber, 2005), where a generator network is learnt through evolution to solve an artificial control problem.
The focus of this work is to apply our method to recurrent networks. In this context, our method has a connection to second-order or multiplicative networks (Goudreau et al., 1994; Sutskever et al., 2011; Wu et al., 2016), where the hidden state of the last step and the input vector of the current time step interact in a multiplicative fashion. The key difference between our approach and second-order networks is that our approach is more memory efficient because we only learn the scaling factors in the interaction matrix. Furthermore, in second-order or multiplicative networks, the weights of the RNN are not fixed, but a linear function of the previous hidden state. In our work, we explore the use of a smaller RNN, rather than a linear function, to produce the weights of the main RNN.
The concept of a network interacting with another network is central to the work of (Jaderberg et al., 2016; Andrychowicz et al., 2016), and especially (Denil et al., 2013; Yang et al., 2015; Bertinetto et al., 2016; De Brabandere et al., 2016), where certain parameters in a convolutional network are predicted by another network. These studies however did not explore the use of this approach to recurrent networks, which is a main contribution of our work.
3 METHODS
Though it is possible to use hypernetworks to generate weights for feedforward or convolutional networks, in this paper, our focus is on using hypernetworks with recurrent networks. As can be seen below, when they are applied to recurrent networks, hypernetworks can be seen as a form of relaxed weight-sharing in the time dimension.
3.1 HYPERRNN
Our hypernetworks can be used to generate weights for the RNN and LSTM. When a hypernetwork is used to generate the weights for an RNN, we refer to it as the HyperRNN. At every time step t, a HyperRNN takes as input the concatenated vector of input xt and the hidden states of the main RNN ht−1, it then generates as output the vector ĥt. This output vector is then used to generate the weights for the main RNN at the same timestep. Both the HyperRNN and the main RNN are trained jointly with backpropagation and gradient descent. In the following, we will give a more formal description of the model.
The standard formulation of a Basic RNN is given by:
ht = φ(Whht−1 +Wxxt + b) (1)
where ht is the hidden state, φ is a non-linear operation such as tanh or relu, and the weight matrices and bias Wh ∈ RNh×Nh ,Wx ∈ RNh×Nx , b ∈ RNh is fixed each timestep for an input sequence X = (x1, x2, . . . , xT ).
In HyperRNN, we allow Wh and Wx to float over time by using a smaller hypernetwork to generate these parameters of the main RNN at each step (see Figure 1). More concretely, the parameters Wh,Wx, b of the main RNN are different at different time steps, so that ht can now be computed as:
ht = φ ( Wh(zh)ht−1 +Wx(zx)xt + b(zb) ) , where
Wh(zh) = 〈Whz, zh〉 Wx(zx) = 〈Wxz, zx〉 b(zb) =Wbzzb + b0
(2)
Where Whz ∈ RNh×Nh×Nz ,Wxz ∈ RNh×Nx×Nz ,Wbz ∈ RNh×Nz , b0 ∈ RNh and zh, zx, zz ∈ RNz . We use a recurrent hypernetwork to compute zh, zx and zb as a function of xt and ht−1:
x̂t = ( ht−1 xt ) ĥt = φ(Wĥĥt−1 +Wx̂x̂t + b̂)
zh =Wĥhĥt−1 + bĥh
zx =Wĥxĥt−1 + bĥx
zb =Wĥbĥt−1
(3)
Where Wĥ ∈ R Nĥ×Nĥ ,Wx̂ ∈ RNĥ×(Nh+Nz), b ∈ RNĥ , and Wĥh,Wĥx,Wĥb ∈ R Nz×Nĥ and bĥh, bĥx ∈ R Nz . This HyperRNN cell has Nĥ hidden units. Typically Nĥ is much smaller than Nh.
As the embeddings zh, zx and zb are of dimensions Nz , which is typically smaller than the hidden state size Nĥ of the HyperRNN cell, a linear network is used to project the output of the HyperRNN cell into the embeddings in Equation 3. After the embeddings are computed, they will be used to generate the full weight matrix of the main RNN.
The above is a general formulation of HyperRNN. However, Equation 2 is not practical because the memory usage becomes too large for real problems. We modify the HyperRNN described in Equation 2 so that it can be more memory efficient. We will use an intermediate hidden vector d(z) ∈ RNh to parametrize each weight matrix, where d(z) will be a linear function of z. To dynamically modify a weight matrix W , we will allow each row of this weight matrix to be scaled linearly by an element in vector d. We refer d as a weight scaling vector. Below is the modification to W (z):
W (z) =W ( d(z) ) =  d0(z)W0d1(z)W1... dNh(z)WNh  (4) While we sacrifice the ability to construct an entire weight matrix from a linear combination of Nz matrices of the same size, we are able to linearly scale the rows of a single matrix withNz degrees of
freedom. We find this change to have a good trade-off, as this formulation of converting W (z) into W (d(z)) decreases the amount of memory required by the HyperRNN. Rather than requiring Nz times the memory of a Basic RNN, we will only be using memory in the order Nz times the number of hidden units, which is an acceptable amount of extra memory usage that is often available in many applications. In addition, the row-level operation in Equation 4 can be shown to be equivalent to an element-wise multiplication operator and hence computationally much more efficient in practice. Below is the more memory efficient version of the setup of Equation 2:
ht = φ ( dh(zh) Whht−1 + dx(zx) Wxxt + b(zb) ) , where
dh(zh) =Whzzh
dx(zx) =Wxzzx
b(zb) =Wbzzb + b0
(5)
In our experiments, we focus on the use of hypernetworks with the Long Short-Term Memory (LSTM) architecture (Hochreiter & Schmidhuber, 1997) because LSTM often works better than the Basic RNN. In such case, an LSTM will have more weight matrices and biases, and thus our main change is to have many more d’s, each d is being associated with each weight matrix or bias.
3.2 RELATED APPROACHES
The formulation of the HyperRNN in Equation 5 has similarities to Recurrent Batch Normalization (Cooijmans et al., 2016) and Layer Normalization (Ba et al., 2016). The central idea for the normalization techniques is to calculate the first two statistical moments of the inputs to the activation function, and to linearly scale the inputs to have zero mean and unit variance. After the normalization, an additional set of fixed parameters are learned to unscale the inputs if required.
Since the HyperRNN cell can indirectly modify the rows of each weight matrix and also the bias of the main RNN, it is implicitly also performing a linear scaling to the inputs of the activation function. The difference here is that the linear scaling parameters will be learned by the HyperRNN cell, and not based on statistical-moments. We note that the existing normalization approaches can work together with the HyperRNN approach, where the HyperRNN cell will be tasked with discovering a better dynamical scaling policy to complement normalization. We also explore this combination in our experiments.
The element-wise operation also has similarities to the Multiplicative RNN and its extensions (mRNN, mLSTM) (Sutskever et al., 2011; Krause et al., 2016) and Multiplicative Integration RNN (MI-RNN) (Wu et al., 2016). In the case of the mRNN, the hidden-to-hidden weight matrix is replaced with a factorized matrix, to allow the weights to be input dependent. The factorization is described below in Equation 6 (Krause et al., 2016).
ht = φ(Whm(Wmxxt) (Wmhht−1) +Wxxt + b) (6)
For the MI-RNN approach, a second order term is added to the Basic RNN formulation, along with scaling vectors for each term, as described in Equation 7. The addition of the scaling vectors allow parameters to be shared more efficiently.
ht = φ(α Wxxt Whht−1 + β1 Whht−1 + β2 Wxxt + b) (7)
In the HyperRNN approach, the weights are also input dependent. However, unlike mRNN, both weight matrices and also the bias term will be dependent to not only to the inputs, but also to the hidden states. In the MI-RNN approach, the weights are also be augmented by both the input and hidden states, via the second order term in Equation 7. In both mRNN and MI-RNN approaches, the weight augmentation terms are produced by a linear operation, while in the HyperRNN approach, the weight scaling vectors d are dynamically produced by another RNN with its own hidden states and non-linearities.
4 EXPERIMENTS
In the following experiments, we will benchmark the performance of HyperLSTM on language modelling with Penn Treebank, and Hutter Prize Wikipedia. We will also benchmark the method on the tasks of handwriting generation with IAM On-Line Handwriting Database, and machine translation with WMT’14 en→fr.
4.1 CHARACTER-LEVEL PENN TREEBANK LANGUAGE MODELLING
We first evaluation the HyperLSTM model on a character level prediction task with the Penn Treebank corpus (Marcus et al., 1993) using the train/validation/test split outlined in (Mikolov et al., 2012). As the dataset is quite small, we apply dropout on both input and output layers with a keep probability of 90%. Unlike previous approaches (Graves, 2013; Ognawala & Bayer, 2014) of applying weight noise during training, we instead also apply dropout to the recurrent layer (Henaff et al., 2016) with the same dropout probability.
We compare our model to the basic LSTM cell, stacked LSTM cells (Graves, 2013), and LSTM with layer normalization applied. In addition, we also experimented with applying layer normalization to HyperLSTM. Using the setup in (Graves, 2013), we use networks with 1000 units and train the network to predict the next character. In this task, the HyperLSTM cell has 128 units and an embedding size of 4. As the HyperLSTM cell has more trainable parameters compared to the basic LSTM cell, we also experimented with an LSTM cell with 1250 units.
For character-level Penn Treebank, we use mini-batches of size 128, to train on sequences of length 100. We train the model using Adam (Kingma & Ba, 2015) with a learning rate of 0.001 and gradient clipping of 1.0. During evaluation, we generate the entire sequence, and do not use information about previous test errors for prediction, e.g., dynamic evaluation (Graves, 2013; Rocki, 2016b). As mentioned earlier, we apply dropout to the input and output layers, and also apply recurrent dropout with a keep probability of 90%. For baseline models, orthogonal initialization (Henaff et al., 2016) is used for all weights.
We also experiment with a version of the model using a larger embedding size of 16, and also with a lower dropout keep probability of 85%, and report results with this “Large Embedding" model in Table 1. Lastly, we stack two layers of this “Large Embedding" model together to measure the benefits of a multi-layer version of HyperLSTM, with a dropout keep probability of 80%.
Our results are presented in Table 1. The key observation here is that 1) HyperLSTM outperforms standard LSTM and 2) HyperLSTM also achieves similar improvements compared to Layer Normalization. The combination of Layer Normalization and Hyper LSTM achieves the best test perplexity so far on this dataset.
4.2 HUTTER PRIZE WIKIPEDIA LANGUAGE MODELLING
We train our model on the larger and more challenging Hutter Prize Wikipedia dataset, also known as enwik8 (Hutter, 2012) consisting of a sequence of 100M characters composed of 205 unique characters. Unlike Penn Treebank, enwik8 contains some foreign words (Latin, Arabic, Chinese), indented XML, metadata, and internet addresses, making it a more realistic and practical dataset to test character language models.
Our setup is similar in the previous experiment, using the same mini-batch size, learning rate, weight initialization, gradient clipping parameters and optimizer. We do not use dropout for the input and output layers, but still apply recurrent dropout with a keep probability of 90%. Similar to (Chung et al., 2015), we train on the first 90M characters of the dataset, use the next 5M as a validation set for early stopping, and the last 5M characters as the test set.
As enwik8 is a bigger dataset compared to Penn Treebank, we will use 1800 units for our networks. We also perform training on sequences of length 250. Our normal HyperLSTM cell consists of 256 units, and we use an embedding size of 64. To improve results, we also experiment with a larger model where both HyperLSTM and main network both have 2048 hidden units. The HyperLSTM cell consists of 512 units with an embedding size of 64. We also apply recurrent dropout to this larger model, with dropout keep probability of 85%, and train on a longer sequence length of 300.
The results are summarized in Table 2. As can be seen from the table, HyperLSTM is once again competitive to Layer Norm LSTM, and if we combine both techniques, the Layer Norm HyperLSTM achieves respectable results. The large version of HyperLSTM with normalization that uses 2048 hidden units achieve near state-of-the-art performance for this task. In addition, HyperLSTM converges more quickly compared to LSTM and Layer Norm LSTM (see Figure 2).
We perform additional analysis to understand the behavior of HyperLSTM by visualizing how the weight scaling vectors of the main LSTM change during the character sampling process. In Figure 3, we examine a sample text passage generated by HyperLSTM after training on enwik8 along with the weight differences below the text. We see that in regions of low intensity, where the weights of the main LSTM are relatively static, the types of phrases generated seem more deterministic. For example, the weights do not change much during the words Europeans, possessions and reservation. The regions of high intensity is when the HyperLSTM cell is making relatively large changes to the weights of the main LSTM.
1We do not compare against methods that use dynamic evaluation. 2Our implementation. 3Based on results of version 2 at the time of writing. http://arxiv.org/abs/1609.01704v2 4This method uses information about test errors during inference for predicting the next characters, hence
it is not directly comparable to other methods that do not use this information.
 In 1955-37 most American and Europeans signed into the sea. An absence of [[Japan (Korea city)|Japan]], the Mayotte like Constantino
 ple (in its first week, in [[880]]) that served as the mother of emperors, as the Corinthians, Bernard on his continued sequel toget
 her ordered [[Operation Moabili]]. The Gallup churches in the army promulgated the possessions sitting at the reservation, and [[Mel
 ito de la Vegeta Provine|Felix]] had broken Diocletian desperate from the full victory of Augustus, cited by Stephen I. Alexander Se
 nate became Princess Cartara, an annual ruler of war (777-184) and founded numerous extremiti of justice practitioners.
Figure 3: Example text generated from HyperLSTM model. We visualize how four of the main RNN’s weight matrices (W ih, W g h , W f h , W o h ) effectively change over time by plotting the norm of the changes below each generated character. High intensity represent large changes being made to weights of main RNN.
4.3 HANDWRITING SEQUENCE GENERATION
In addition to modelling discrete sequential data, we want to see how the model performs when modelling sequences of real valued data. We will train our model on the IAM online handwriting database (Liwicki & Bunke, 2005) and have our model predict pen strokes as per Section 4.2 of (Graves, 2013). The dataset has contains 12179 handwritten lines from 221 writers, digitally recorded from a tablet. We will model the (x, y) coordinate of the pen location at each recorded time step, along with a binary indicator of pen-up/pen-down. The average sequence length is around 700 steps and the longest around 1900 steps, making the training task particularly challenging as the network needs to retain information about both the stroke history and also the handwriting style in order to predict plausible future handwriting strokes.
We will use the same model architecture described in (Graves, 2013) and use a Mixture Density Network layer (Bishop, 1994) to generate a mixture of bi-variate Gaussian distributions to model at each time step to model the pen location. We normalize the data and use the same train/validation split as per (Graves, 2013) in this experiment. We remove samples less than length 300 as we found these samples contain a lot of recording errors and noise. After the pre-processing, as the dataset is small, we introduce data augmentation of chosen uniformly from +/- 10% and apply a this random scaling a the samples used for training.
For model training, will apply recurrent dropout and also dropout to the output layer with a keep probability of 0.95. The model is trained on mini-batches of size 32 containing sequences of variable length. We trained the model using Adam (Kingma & Ba, 2015) with a learning rate of 0.0001 and gradient clipping of 5.0. Our HyperLSTM cell consists of 128 units and a signal size of 4. For baseline models, orthogonal initialization (Henaff et al., 2016) is performed for all weights.
1Our implementation, to replicate setup of (Graves, 2013). 2Our implementation, with data augmentation, dropout and recurrent dropout.
The results are summarized in Table 3. Our main result is that HyperLSTM with 900 units, without Layer Norm, achieves best log loss on the validation set across all models in published work and in our experiment. HyperLSTM also converges more quickly compared to other models (see Figure 2).
Similar to the earlier character generation experiment, we show a generated handwriting sample from the HyperLSTM model in Figure 4, along with a plot of how the weight scaling vectors of the main RNN is changing over time below the sample. For a more detailed interactive demonstration of handwriting generation using HyperLSTM, visit http://blog.otoro.net/2016/09/28/ hyper-networks/.
We observe that the regions of high intensity is concentrated at many discrete instances, rather than slowly varying over time. This implies that the weights experience regime changes rather than gradual slow adjustments. We can see that many of these weight changes occur at the boundaries between words, and between characters. While the LSTM model alone already does a decent job of generating time-varying parameters of a Mixture Gaussian distribution used to generate realistic handwriting samples, the ability to go one level deeper, and to dynamically generate the generative model is one of the key advantages of HyperLSTM over a normal LSTM.
4.4 NEURAL MACHINE TRANSLATION
Finally, we experiment with the Neural Machine Translation task using the same experimental setup outlined in (Wu et al., 2016). Our model is the same wordpiece model architecture with a vocabulary size of 32k, but we replace the LSTM cells with HyperLSTM cells. We benchmark both models on WMT’14 En→Fr using the same test/validation set split described in the GNMT paper (Wu et al., 2016). The GNMT network has 8 layers in the encoder, 8 layers in the decoder. The first layer of the encoder has bidirectional connections. The attention module is a neural network with 1 hidden layer. When a LSTM cell is used, the number of hidden units in each layer is 1024. The model is trained in a distributed setting with a parameter sever and 12 workers. Additionally, each worker uses 8 GPUs and a minibatch of 128.
Our experimental setup is similar to that in the GNMT paper (Wu et al., 2016), with two simplifications. First, we use only Adam without SGD at the end. Adam was used with the same same hyperparameters described in the GNMT paper: learning rate of 0.0002 for 1M training steps.
We apply the HyperLSTM cell with Layer Norm to the GNMT architecture that uses a vocabulary of 32K wordpieces. We keep the same number of hidden units, which means that our model will have 16% more parameters.
The results are reported in Table 4, which shows that the HyperLSTM cell improves the performance of the existing GNMT model, achieving state-of-the-art single model results for this dataset. In addition, we demonstrate the applicability of hypernetworks to large-scale models used in production systems.
5 CONCLUSION
In this paper, we presented a method to use one network to generate weights for another neural network. Our hypernetworks are trained end-to-end with backpropagation and therefore are efficient and scalable. We focused on applying hypernetworks to generate weights for recurrent networks. On language modelling and handwriting generation, hypernetworks are competitive to or sometimes better than state-of-the-art models. On machine translation, hypernetworks achieve a significant gain on top of a state-of-the-art production-level model.
ACKNOWLEDGMENTS
We thank Jeff Dean, Geoffrey Hinton, Mike Schuster and the Google Brain team for their help with the project.
A APPENDIX
A.1 EXAMPLES OF GENERATED WIKIPEDIA TEXT
A.2 EXAMPLES OF RANDOMLY CHOSEN GENERATED HANDWRITING SAMPLES
A.3 EXAMPLES OF RANDOMLY CHOSEN MACHINE TRANSLATION SAMPLES
We randomly selected translation samples generated from both LSTM baseline and HyperLSTM models from the WMT’14 En→Fr Test Set. Given an English phrase, we can compare between the correct French translation, the LSTM translation, and the HyperLSTM translation.
English Input
I was expecting to see gnashing of teeth and a fight breaking out at the gate .
French (Ground Truth)
Je m’ attendais à voir des grincements de dents et une bagarre éclater à la porte .
LSTM Translation
Je m’ attendais à voir des larmes de dents et un combat à la porte .
HyperLSTM Translation
Je m’ attendais à voir des dents grincer des dents et une bataille éclater à la porte .
English Input
Prosecuting , Anne Whyte said : " If anyone should know not to the break the law , it is a criminal solicitor . "
French (Ground Truth)
Le procureur Anne Whyte a déclaré : « Si quelqu’ un doit savoir qu’ il ne faut pas violer la loi , c’ est bien un avocat pénaliste . »
LSTM Translation
Prosecuting , Anne Whyte a dit : « Si quelqu’ un doit savoir qu’ il ne faut pas enfreindre la loi , c’ est un solicitor criminel .
HyperLSTM Translation
En poursuivant , Anne Whyte a dit : « Si quelqu’ un doit savoir ne pas enfreindre la loi , c’ est un avocat criminel .
English Input
According to her , the CSRS was invited to a mediation and she asked for an additional period for consideration .
French (Ground Truth)
Selon elle , la CSRS a été invitée à une médiation et elle a demandé un délai supplémentaire pour y réfléchir .
LSTM Translation
Selon elle , le SCRS a été invité à une médiation et elle a demandé un délai supplémentaire .
HyperLSTM Translation
Selon elle , le SCRS a été invité à une médiation et elle a demandé une période de réflexion supplémentaire .
English Input
Relations between the US and Germany have come under strain following claims that the NSA bugged Chancellor Angela ’s Merkel ’s phone .
French (Ground Truth)
Les relations entre les États-Unis et l’ Allemagne ont été mises à rude épreuve à la suite de plaintes selon lesquelles la NSA avait mis sur écoute le téléphone portable de la chancelière allemande Angela Merkel .
LSTM Translation
Les relations entre les Etats-Unis et l’ Allemagne ont été mises à rude épreuve suite aux affirmations selon lesquelles la NSA aurait pris le téléphone de Merkel de la chancelière Angela .
HyperLSTM Translation
Les relations entre les États-Unis et l’ Allemagne ont été mises à rude épreuve après que la NSA a attaqué le téléphone de la chancelière Angela Angela .
English Input
Germany ’s BfV advises executives to consider using simple prepaid mobiles when on foreign trips because of the risk that smart phones are compromised .
French (Ground Truth)
Le BfV d’ Allemagne conseille à ses dirigeants d’ envisager d’ utiliser de simples téléphones portables prépayés lors de leurs voyages à l’ étranger en raison du risque d’ atteinte à l’ intégrité des smartphones .
LSTM Translation
Le BfV allemand conseille aux dirigeants d’ envisager l’ utilisation de mobiles prépayés simples lors de voyages à l’ étranger en raison du risque de compromission des téléphones intelligents .
HyperLSTM Translation
Le BfV allemand conseille aux dirigeants d’ envisager l’ utilisation de téléphones mobiles prépayés simples lors de voyages à l’ étranger en raison du risque que les téléphones intelligents soient compromis .
English Input
I was on the mid-evening news that same evening , and on TV the following day as well .
French (Ground Truth)
Le soir-même , je suis au 20h , le lendemain aussi je suis à la télé .
LSTM Translation
J’ étais au milieu de l’ actualité le soir même , et à la télévision le lendemain également .
HyperLSTM Translation
J’ étais au milieu de la soirée ce soir-là et à la télévision le lendemain .

*** Paper Summary ***
The paper proposes to a new neural network architecture. The layer weights of a classical network are computed as a function of a latent representation associated with the layer. Two instances are presented (i) a CNN where each layer weight is computed from a lower dimensional layer embedding vector; (ii) an RNN where each layer weight is computed from a secondary RNN state.
*** Review Summary ***
Pros:
- I like the idea of bringing multiplicative RNNs and their predecessors back into the spotlight.
- LM and MT results are excellent.
Cons:
- The paper could be better written. It is too long for the conference format and need refocussing.
- On related work, the relation with multiplicative RNN and their generic tensor product predecessor (Order 2 networks, wrt C. Lee Giles definition) should be mentioned in the related work section and the differences with earlier research need to be explained and motivated (by the way it is better to say that something is revisiting an old idea or training it at modern scale/on modern tasks than ommitting it).
- on focus, it is not clear if your goal is to achieve better performance or more compact networks. In the RNN section you lean toward the former, in the CNN section you seem to lean toward the latter.
I would suggest to make the paper shorter and clearer possibly leaving the CNN results for latter publication. The relation with multiplicative/order 2 networks and eventual differences need to be explained.
*** Detailed Review ***
Multiplicative networks are an extremely powerfull architecture and bringing them back into the spotlight is excellent. This paper has excellent results but suffer poor presentation, lack of a clear focus. It spends time on details and ommit important points. In its current form, it is much too long to long and his not self contained without the appendices.
Spending more time on multiplicative RNNs, order 2 networks at the begining of the paper would be excellent. This will let you highlight the difference between this paper and earlier work. It would also be necessary to spend a little time on why multiplicative RNN were less used than gated RNN: it seems that the optimization problem their training involve is tricker and it would be helpful to explain whether you had a harder time tweaking optimization parameters or whether you needed longer training sessions compared to LSTMs, regular CNN. On name, I am not sure that "hypernetwork" help the reader understand better what the proposed architecture compared to multiplicative interactions.
In section 3.2, you seem to imply that there are different settings of hypernetworks that allow to vary from an RNN to a CNN, this is not clear to me, maybe you could show how this would work on a simple temporal problem with equations.
The work on CNN and RNN are rather disconnected to me: for CNN, you seem to be interested in a low rank structure of the weights, showing that similar performance can be achieved with less weights. It is not clear to me why to pursue that goal. Do you expect speedups? less memory for embedded applications? In that case you should compare with alternative strategies, e.g. model compression (Caruana et al 2006, aka Dark Knowledge, Hinton et al 2014) or hashed networks (Chen et al 2015).
For RNN, you seem to target better perplexity/BLEU and model compactness is not a priority. Instead of making the weights have a simpler structure, you make them richer, i.e. dependent over time. It seems in that case models might be bigger and take longer to train. You might want to comment on training time, inference time, memory requirement in that case, as you highlight it might be an important goal in the CNN section. Overall, I am not sure it helps to have this mixed message. I would rather see the paper fit in the conference format with the RNN results alone and a clearer explanation and defers the publications of the CNN results when a proper comparison with memory concerned methods is performed.
Some of the discussions are not clear to me, I am not sure what message the reader should get from Figure 2 or from the discussion on saturation statistics (p10, Figure 5). Similarly, I am not sure if Figure 4 is showing anything: everything should change more drastically at word boundaries even in a regular LSTM (states, gates units should look very different before/after a space); without such a comparison it is hard to see if this is unique to your network.
The results on handwriting generation are harder to compare for me. Log-loss are hard to understand, I have no sense whether the difference between models is significant (what would be the variance in this metric under boostrap sampling of the training set?). I am not sold either on qualitative metric were human can assess quality but human cannot evaluate if the network is repeating the training set. Did you thing at precision/recall metric for ink, possibly with some spatial tolerance ? (e.g. evaluation of segmentation tasks in vision).
The MT experiments are insufficiently discussed in the main text.
Overall, I would suggest to make the paper shorter and clearer possibly leaving the CNN results for latter publication. You need to properly discuss the relation to multiplicative/order 2 networks and highlight the differences. Unclear discussion can be eliminated to make the experimental setup and the results presentation clearer in the main text.
*** References ***
M.W. Goudreau, C.L. Giles, S.T. Chakradhar, D. Chen, "First-Order Vs. Second-Order Single Layer Recurrent Neural Networks,"IEEE Trans. on Neural Networks, 5 (3), p. 511, 1994.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil, "Model Compression," The Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2006), August 2006, pp. 535-541.
Dark knowledge, G Hinton, O Vinyals, J Dean 2014
W. Chen, J. Wilson, S. Tyree, K. Weinberger and Y. Chen, Compressing Neural Networks with the Hashing Trick, Proc. International Conference on Machine Learning (ICML-15)


Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem. Unfortunately, existing automatic evaluation metrics are biased and correlate very poorly with human judgements of response quality (Liu et al., 2016). Yet having an accurate automatic evaluation procedure is crucial for dialogue research, as it allows rapid prototyping and testing of new models with fewer expensive human evaluations. In response to this challenge, we formulate automatic dialogue evaluation as a learning problem. We present an evaluation model (ADEM) that learns to predict human-like scores to input responses, using a new dataset of human response scores. We show that the ADEM model’s predictions correlate significantly, and at level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and system-level. We also show that ADEM can generalize to evaluating dialogue models unseen during training, an important step for automatic dialogue evaluation.
1 INTRODUCTION
Learning to communicate with humans is a crucial ability for intelligent agents. Among the primary forms of communication between humans is natural language dialogue. As such, building systems that can naturally and meaningfully converse with humans has been a central goal of artificial intelligence since the formulation of the Turing test (Turing, 1950). Research on one type of such systems, sometimes referred to as non-task-oriented dialogue systems, goes back to the mid-60s with Weizenbaum’s famous program ELIZA: a rule-based system mimicking a Rogerian psychotherapist by persistently either rephrasing statements or asking questions (Weizenbaum, 1966). Recently, there has been a surge of interest in the research community towards building large-scale non-task-oriented dialogue systems using neural networks (Sordoni et al., 2015b; Shang et al., 2015; Vinyals & Le, 2015; Serban et al., 2016a; Li et al., 2015). These models are trained in an end-to-end manner to optimize a single objective, usually the likelihood of generating the responses from a fixed corpus. Such models have already had a substantial impact in industry, including Google’s Smart Reply system (Kannan et al., 2016), and Microsoft’s Xiaoice chatbot (Markoff & Mozur, 2015), which has over 20 million users. More recently, Amazon has announced the Alexa Prize Challenge: a research competition with the goal of developing a natural and engaging chatbot system (Farber, 2016).
One of the challenges when developing such systems is to have a good way of measuring progress, in this case the performance of the chatbot. The Turing test provides one solution to the evaluation of dialogue systems, but there are limitations with its original formulation. The test requires live human interactions, which is expensive and difficult to scale up. Furthermore, the test requires carefully designing the instructions to the human interlocutors, in order to balance their behaviour and expectations so that different systems may be ranked accurately by performance. Although unavoidable, these instructions introduce bias into the evaluation measure. The more common approach of having humans evaluate the quality of dialogue system responses, rather than distinguish them from human responses, induces similar drawbacks in terms of time, expense, and lack of
∗The second and third authors contributed equally.
scalability. In the case of chatbots designed for specific conversation domains, it may also be difficult to find sufficient human evaluators with appropriate background in the topic (e.g. Lowe et al. (2015)).
Despite advances in neural network-based models, evaluating the quality of dialogue responses automatically remains a challenging and under-studied problem in the non-task-oriented setting. The most widely used metric for evaluating such dialogue systems is BLEU (Papineni et al., 2002), a metric measuring word overlaps originally developed for machine translation. However, it has been shown that BLEU and other word-overlap metrics are biased and correlate poorly with human judgements of response quality (Liu et al., 2016). There are many obvious cases where these metrics fail, as they are often incapable of considering the semantic similarity between responses (see Figure 1). Despite this, many researchers still use BLEU to evaluate their dialogue models (Ritter et al., 2011; Sordoni et al.,
2015b; Li et al., 2015; Galley et al., 2015; Li et al., 2016a), as there are few alternatives available that correlate with human judgements. While human evaluation should always be used to evaluate dialogue models, it is often too expensive and time-consuming to do this for every model specification (for example, for every combination of model hyperparameters). Therefore, having an accurate model that can evaluate dialogue response quality automatically — what could be considered an automatic Turing test — is critical in the quest for building human-like dialogue agents.
To make progress towards this goal, we first collect a dataset of human scores to various dialogue responses, and we use this dataset to train an automatic dialogue evaluation model, which we call ADEM. The model is trained in a semi-supervised manner using a hierarchical recurrent neural network (RNN) to predict human scores. We show that ADEM scores correlate significantly, and at a level much higher than BLEU, with human judgement at both the utterance-level and system-level. Crucially, we also show that ADEM can generalize to evaluating new models, whose responses were unseen during training, without a drop in performance, making ADEM a strong first step towards effective automatic dialogue response evaluation.1
2 A DATASET FOR DIALOGUE RESPONSE EVALUATION
To train a model to predict human scores to dialogue responses, we first collect a dataset of human judgements (scores) of Twitter responses using the crowdsourcing platform Amazon Mechanical Turk (AMT).2 The aim is to have accurate human scores for a variety of conversational responses — conditioned on dialogue contexts – which span the full range of response qualities. For example, the responses should include both relevant and irrelevant responses, both coherent and non-coherent responses and so on. To achieve this variety, we use candidate responses from several different models. Following Liu et al. (2016), we use the following 4 sources of candidate responses: (1) a response selected by a TF-IDF retrieval-based model, (2) a response selected by the Dual Encoder (DE) (Lowe et al., 2015), (3) a response generated using the hierarchical recurrent encoder-decoder (HRED) model (Serban et al., 2016a), and (4) human-generated responses. It should be noted that the human-generated candidate responses are not the
reference responses from a fixed corpus, but novel human responses that are different from the reference. In addition to increasing response variety, this is necessary because we want our evaluation model to learn to compare the reference responses to the candidate responses.
1We will provide open-source implementations of the model upon publication. 2All data collection was conducted in accordance with the policies of the host institutions’ ethics board.
We conducted two rounds of AMT experiments. We first asked AMT workers to provide a reasonable continuation of a Twitter dialogue (i.e. generate the next response given the context of a conversation). Each survey contained 20 questions, including an attention check question. Workers were instructed to generate longer responses, in order to avoid simple one-word responses. In total, we obtained approximately 2,000 human responses.
Second, we filtered these human-generated responses for potentially offensive language, and combined them with approximately 1,000 responses from each of the above models into a single set of responses. We then asked AMT workers to rate the overall quality of each response on a scale of 1 (low quality) to 5 (high quality). Each user was asked to evaluate 4 responses from 50 different contexts. We included four additional attention-check questions and a set of five contexts was given to each participant for assessment of inter-annotator agreement. We removed all users who either failed an attention check question or achieved a κ inter-annotator agreement score lower than 0.2 (Cohen, 1968). The remaining evaluators had a median κ score of 0.63, indicating moderate agreement. This is consistent with results from Liu et al. (2016). Dataset statistics are provided in Table 1.
Measurement κ score Overall 0.63 Topicality 0.57 Informativeness 0.31 Background 0.05
To train evaluation models on human judgements, it is crucial that we obtain scores of responses that lie near the distribution produced by state-of-the-art models. This is why we use the Twitter Corpus (Ritter et al., 2011), as such models are pre-trained and readily available. Further, the set of topics discussed is quite broad — as opposed to the very specific Ubuntu Dialogue Corpus — and therefore the model should generalize better to other domains involving chit-chat. Finally, since it does not require domain specific knowledge (e.g. technical knowledge), it should be easy for AMT workers to annotate.
3 TECHNICAL BACKGROUND
3.1 RECURRENT NEURAL NETWORKS
Recurrent neural networks (RNNs) are a type of neural network with time-delayed connections between the internal units. This leads to the formation of a hidden state ht, which is updated for every input: ht = f(Whhht−1 +Wihxt), where Whh and Wih are parameter matrices, f is a smooth non-linear activation function such as tanh, and xt is the input at time t. The hidden state allows for RNNs to better model sequential data, such as natural language.
In this paper, we consider RNNs augmented with long-short term memory (LSTM) units (Hochreiter & Schmidhuber, 1997). LSTMs add a set of gates to the RNN that allow it to learn how much to update the hidden state. LSTMs are one of the most well-established methods for dealing with the vanishing gradient problem in recurrent networks (Hochreiter, 1991; Bengio et al., 1994).
3.2 WORD-OVERLAP METRICS
One of the most popular approaches for automatically evaluating the quality of dialogue responses is by computing their word overlap with the reference response. In particular, the most popular metrics are the BLEU and METEOR scores used for machine translation, and the ROUGE score used for automatic summarization. While these metrics tend to correlate with human judgements in their target domains, they have recently been shown to highly biaqsed and correlate very poorly with
human judgements for dialogue response evaluation (Liu et al., 2016). We briefly describe BLEU here, and provide a more detailed summary of word-overlap metrics in the Appendix.
BLEU BLEU (Papineni et al., 2002) analyzes the co-occurrences of n-grams in the ground truth and the proposed responses. It computes the n-gram precision for the whole dataset, which is then multiplied by a brevity penalty to penalize short translations. For BLEU-N , N denotes the largest value of n-grams considered (usually N = 4).
Drawbacks One of the major drawbacks of word-overlap metrics is their failure in capturing the semantic similarity between the model and reference responses when there are few or no common words. This problem is less critical for machine translation; since the set of reasonable translations of a given sentence or document is rather small, one can reasonably infer the quality of a translated sentence by only measuring the word-overlap between it and one (or a few) reference translations. However, in dialogue, the set of appropriate responses given a context is much larger (Artstein et al., 2009); in other words, there is a very high response diversity that is unlikely to be captured by word-overlap comparison to a single response.
Further, word-overlap scores are computed directly between the model and reference responses. As such, they do not consider the context of the conversation. While this may be a reasonable assumption in machine translation, it is not the case for dialogue; whether a model response is an adequate substitute for the reference response is clearly context-dependent. For example, the two responses in Figure 1 are equally appropriate given the context. However, if we simply change the context to: “Have you heard of any good movies recently?”, the model response is no longer relevant while the reference response remains valid.
4 AN AUTOMATIC DIALOGUE EVALUATION MODEL (ADEM)
To overcome the problems of evaluation with word-overlap metrics, we aim to construct a dialogue evaluation model that: (1) captures semantic similarity beyond word overlap statistics, and (2) exploits both the context of the conversation and the reference response to calculate its score for the model response. We call this evaluation model ADEM.
ADEM learns distributed representations of the context, model response, and reference response using a hierarchical RNN encoder. Given the dialogue context c, reference response r, and model response r̂, ADEM first encodes each of them into vectors (c, r̂, and r, respectively) using the RNN encoder. Then, ADEM computes the score using a dot-product between the vector representations of c, r, and r̂ in a linearly transformed space: :
score(c, r, r̂) = (cTM r̂ + rTN r̂− α)/β (1)
where M,N ∈ Rn are learned matrices initialized to the identity, and α, β are scalar constants used to initialize the model’s predictions in the range [0, 5]. The model is shown in Figure 2.
The matrices M and N can be interpreted as linear projections that map the model response r̂ into the space of contexts and reference responses, respectively. The model gives high scores to responses that have similar vector representations to the context and reference response after this projection. The model is end-to-end differentiable; all the parameters can be learned by backpropagation. In our
implementation, the parameters θ = {M,N} of the model are trained to minimize the squared error between the model predictions and the human score, with L1-regularization:
L = ∑ i=1:K [score(ci, ri, r̂i)− human scorei]2 + γ||θ||1 (2)
where γ is a scalar constant. The simplicity of our model leads to both accurate predictions and fast evaluation time (see Appendix), which is important to allow rapid prototyping of dialogue systems.
The hierarchical RNN encoder in our model consists of two layers of RNNs (El Hihi & Bengio, 1995; Sordoni et al., 2015a). The lower-level RNN, the utterance-level encoder, takes as input words from the dialogue, and produces a vector output at the end of each utterance. The context-level encoder takes the representation of each utterance as input and outputs a vector representation of the context. This hierarchical structure is useful for incorporating information from early utterances in the context (Serban et al., 2016a). Following previous work, we take the last hidden state of the context-level encoder as the vector representation of the input utterance or context.
An important point is that the ADEM procedure above is not a dialogue retrieval model. The fundamental difference between ADEM and a dialogue model is that ADEM has access to the reference response. Thus, ADEM can compare a model’s response to a known good response, which is significantly easier than inferring response quality from solely the context.
Pre-training with VHRED We would like an evaluation model that can make accurate predictions from few labeled examples, since these examples are expensive to obtain. We therefore employ semi-supervised learning, and use a pre-training procedure to learn the parameters of the encoder. In particular, we train the encoder as part of a neural dialogue model; we attach a third decoder RNN that takes the output of the encoder as input, and train it to predict the next utterance of a dialogue conditioned on the context.
The dialogue model we employ for pre-training is the latent variable hierarchical recurrent encoderdecoder (VHRED) model (Serban et al., 2016b). The VHRED model is an extension of the original hierarchical recurrent encoder-decoder (HRED) model (Serban et al., 2016a) with a turn-level stochastic latent variable. The dialogue context is encoded into a vector using our hierarchical encoder, and the VHRED then samples a Gaussian variable that is used to condition the decoder (see Appendix for further details). After training VHRED, we use the last hidden state of the context-level encoder, when c, r, and r̂ are fed as input, as the vector representations for c, r, and r̂, respectively. We use representations from the VHRED model as it produces more diverse and coherent responses compared to its HRED counterpart.
Maximizing the likelihood of generating the next utterance in a dialogue is not only a convenient way of training the encoder parameters; it is also an objective that is consistent with learning useful representations of the dialogue utterances. Two context vectors produced by the VHRED encoder are similar if the contexts induce a similar distribution over subsequent responses; this is consistent with the formulation of the evaluation model, which assigns high scores to responses that have similar vector representations to the context. VHRED is also closely related to the skip-thought-vector model (Kiros et al., 2015), which has been shown to learn useful representations of sentences for many tasks, including semantic relatedness and paraphrase detection. The skip-thought-vector model takes as input a single sentence and predicts the previous sentence and next sentence. On the other hand, VHRED takes as input several consecutive sentences and predicts the next sentence. This makes it particularly suitable for learning long-term context representations.
5 EXPERIMENTS
5.1 EXPERIMENTAL PROCEDURE
In order to reduce the effective vocabulary size, we use byte pair encoding (BPE) (Gage, 1994; Sennrich et al., 2015), which splits each word into sub-words or characters. We also use layer normalization (Ba et al., 2016) for the hierarchical encoder, which we found worked better at the task of dialogue generation than the related recurrent batch normalization (Ioffe & Szegedy, 2015; Cooijmans et al., 2016). To train the VHRED model, we employed several of the same techniques found in Serban et al. (2016b) and Bowman et al. (2016): we drop words in the decoder with a fixed
rate of 25%, and we anneal the KL-divergence term linearly from 0 to 1 over the first 60,000 batches. We use Adam as our optimizer (Kingma & Ba, 2014).
For training VHRED, we use a context embedding size of 2000. However, we found the ADEM model learned more effectively when this embedding size was reduced. Thus, after training VHRED, we use principal component analysis (PCA) (Pearson, 1901) to reduce the dimensionality of the context, model response, and reference response embeddings to n. While our results are robust to n, we found experimentally that n = 7 provided slightly improved performance. We provide other hyperparameter values in the Appendix.
When evaluating our models, we conduct early stopping on a separate validation set to obtain the best parameter setting. For the evaluation dataset, we split the train/ validation/ test sets such that there is no context overlap (i.e. the contexts in the test set are unseen during training).
5.2 RESULTS
Utterance-level correlations We first present new utterance-level correlation results3 for existing word-overlap metrics, in addition to results with embedding baselines and ADEM, in Table 3. The
3We present both the Spearman correlation (computed on ranks, depicts monotonic relationships) and Pearson correlation (computed on true values, depicts linear relationships) scores.
baseline metrics are evaluated on the entire dataset of 4,104 responses.4 We measure the correlation for ADEM on the validation and test sets (616 responses each).
We also conduct an additional analysis of the response data from Liu et al. (2016), where the preprocessing is standardized by removing ‘<first speaker>’ tokens at the beginning of each utterance. The results are detailed in Table 10 of Appendix D. We can observe from both this data, and the new data in Table 3, that the correlations for the word-overlap metrics are even lower than estimated in previous studies (Liu et al., 2016; Galley et al., 2015). In particular, this is the case for BLEU-4, which has frequently been used for dialogue response evaluation (Ritter et al., 2011; Sordoni et al., 2015b; Li et al., 2015; Galley et al., 2015; Li et al., 2016a).
We can see from Table 3 that ADEM correlates far better with human judgement than the word-overlap baselines. This is further illustrated by the scatterplots in Figure 3. We also compare with ADEM using tweet2vec embeddings for c, r, and r̂, which are computed at the character-level with a bidirectional GRU (Dhingra et al., 2016), and obtain comparable but slightly inferior performance compared to using VHRED embeddings.
System-level correlations We show the system-level correlations for various metrics in Table 4, and present it visually in Figure 4. Each point in the scatterplots represents a dialogue model; humans give low scores to TFIDF and DE responses, higher scores to HRED and the highest scores to other human responses. It is clear that existing word-overlap metrics are incapable of capturing this relationship for even 4 models. This renders them completely deficient for dialogue evaluation. However, ADEM produces the exact same model ranking as humans, achieving a significant Pearson correlation of 0.98.5 Thus, ADEM correlates well with humans both at the response and system level.
Generalization to previously unseen models When ADEM is used in practice, it will take as input responses from a new model that it has not seen during training. Thus, it is crucial that ADEM correlates with human judgements for new models. We test ADEM’s generalization ability by performing a leave-one-out evaluation. For each dialogue model that was the source of response data for training ADEM (TF-IDF, Dual Encoder, HRED, humans), we conduct an experiment where we train on all model responses except those from the chosen model, and test only on the model that was unseen during training.
The results are given in Table 5. Overall, we observe that the ADEM model is very robust, and is capable of generalizing to new models in all cases. When testing the correlation on the entire test set, the model achieves comparable correlations to the ADEM model that was trained on 25%
4Note that our word-overlap correlation results in Table 3 are also lower than those presented in Galley et al. (2015). This is because Galley et al. measure corpus-level correlation, i.e. correlation averaged across different subsets (of size 100) of the data, and pre-filter for high-quality reference responses.
5For comparison, BLEU achieves a system-level correlation of 0.99 on 5 models in the translation domain (Papineni et al., 2002).
less data selected at random. This is particularly surprising for the HRED model; in this case, ADEM was trained only on responses that were written by humans (from retrieval models or humangenerated), but is able to generalize to responses produced by a generative neural network model. This demonstrates ADEM’s ability to accurately score new neural network-based dialogue models.
Qualitative Analysis To illustrate some strengths and weaknesses of ADEM, we show human and ADEM scores for each of the responses to various contexts in Table 7. There are several instances where ADEM predicts accurately: in particular, ADEM is often very good at assigning low scores to poor responses. This seen in the first two contexts, where most of the responses given a score of 1 from humans are given scores less than 2 by ADEM. The single exception in response (4) for the second context seems somewhat appropriate and should perhaps have been scored higher by the human evaluator. There are also several instances where the model assigns high scores to suitable responses, as in the first two contexts.
One drawback we observed is that ADEM tends to be too conservative when predicting response scores. This is the
case in the third context, where the model assigns low scores to most of the responses that a human rated highly (although response (2) is arguably not relevant to the context). This behaviour is likely due to the squared error loss used to train ADEM; since the model receives a large penalty for incorrectly predicting an extreme value, it learns to predict scores closer to the average human score.
Correlation with response length One implicit assumption in the ADEM model is that the human evaluations of model responses is absolutely correct, including the biases that humans exhibit when evaluating dialogues. For example, it has been shown that humans have a tendency to give a higher rating to shorter responses than to longer responses (Serban et al., 2016b), as shorter responses are often more generic and thus are more likely to be suitable to the context. This affects dialogue response evaluation: we calculated the test set correlation between response length and the human score, and obtained a significant Pearson correlation of 0.27, and a Spearman correlation of 0.32. If the assumption that human evaluators are absolutely correct is not accurate, it may be desirable to remove human biases in an automatic evaluation model to improve the model’s generalization capabilities. This is an important direction for future work.
Improvement over word-overlap metrics Next, we analyze more precisely how ADEM outperforms traditional word-overlap metrics such as BLEU-2 and ROUGE. We first normalize the metric scores to have the same mean and variance as human scores, clipping the resulting scores to the range [1, 5] (we assign raw scores of 0 a normalized score of 1). We indicate normalization with vertical bars around the metric. We then select all of the good responses that were given low scores by word-overlap metrics (i.e. responses which humans scored as 4 or higher, and which |BLEU-2| and |ROUGE| scored as 2 or lower). The results are summarized in Table 6: of the 237 responses that humans scored 4 or higher, most of them (147/237) were ranked very poorly by both BLEU-2 and ROUGE. This quantitatively demonstrates what we argued qualitatively in Figure 1; a major failure of word-overlap metrics is the inability to consider reasonable responses that have no word-overlap with the reference response. We can also see that, in almost half (60/147) of the cases where both BLEU-2 and ROUGE fail, |ADEM| is able to correctly assign a score greater than 4. For comparison, there are only 42 responses where humans give a score of 4 and |ADEM| gives a score less than 2, and only 14 of these are assigned a score greater than 4 by either |BLEU-2| or |ROUGE|.
To provide further insight, we give specific examples of responses that are scored highly (> 4) by both humans and |ADEM|, and poorly (< 2) by both |BLEU-2| and |ROUGE| in Table 9. We draw 3 responses randomly (i.e. no cherry-picking) from the 60 test set responses that meet this criteria. We can observe that ADEM is able to recognize short responses that are appropriate to the context, without word-overlap with the reference response. This is even the case when the model and reference responses have very little semantic similarity, as in the first and third examples in Table 9.
Finally, we show the behaviour of ADEM when there is a discrepancy between the lengths of the reference and model responses. In (Liu et al., 2016), the authors show that word-overlap metrics such as BLEU-1, BLEU-2,
and METEOR exhibit a bias in this scenario: they tend to assign higher scores to responses that are
closer in length to the reference response.6 However, humans do not exhibit this bias; in other words, the quality of a response as judged by a human is roughly independent of its length. In Table 8, we show that ADEM also does not exhibit this bias towards similar-length responses.
6 RELATED WORK
Related to our approach is the literature on novel methods for the evaluation of machine translation systems, especially through the WMT evaluation task (Callison-Burch et al., 2011; Machácek & Bojar, 2014; Stanojevic et al., 2015). In particular, Gupta et al. (2015) have recently proposed to evaluate machine translation systems using Tree-LSTMs. Their approach differs from ours as, in the dialogue domain, we must additionally condition our score on the context of the conversation, which is not necessary in translation.
Several recent approaches use hand-crafted reward features to train dialogue models using reinforcement learning (RL). For example, Li et al. (2016b) use features related to ease of answering and information flow, and Yu et al. (2016) use metrics related to turn-level appropriateness and conversational depth. These metrics are based on hand-crafted features, which only capture a small set of relevant aspects; this inevitably leads to sub-optimal performance, and it is unclear whether such objectives are preferable over retrieval-based cross-entropy or word-level maximum log-likelihood objectives. Furthermore, many of these metrics are computed at the conversation-level, and are not available for evaluating single dialogue responses. The metrics that can be computed at the response-level could be incorporated into our framework, for example by adding a term to equation 1 consisting of a dot product between these features and a vector of learned parameters.
There has been significant work on evaluation methods for task-oriented dialogue systems, which attempt to solve a user’s task such as finding a restaurant. These methods include the PARADISE framework (Walker et al., 1997) and MeMo (Möller et al., 2006), which consider a task completion signal. Our models do not attempt to model task completion, and thus fall outside this domain.
7 DISCUSSION
We use the Twitter Corpus to train our models as it contains a broad range of non-task-oriented conversations and has has been used to train many state-of-the-art models. However, our model could easily be extended to other general-purpose datasets, such as Reddit, once similar pre-trained models become publicly available. Such models are necessary even for creating a test set in a new domain, which will help us determine if ADEM generalizes to related dialogue domains. We leave investigating the domain transfer ability of ADEM for future work.
The evaluation model proposed in this paper favours dialogue models that generate responses that are rated as highly appropriate by humans. It is likely that this property does not fully capture the desired end-goal of chatbot systems. For example, one issue with building models to approximate human judgements of response quality is the problem of generic responses. Since humans often provide high scores to generic responses due to their appropriateness for many given contexts, a model trained to predict these scores will exhibit the same behaviour. An important direction for future work is modifying ADEM such that it is not subject to this bias. This could be done, for example, by censoring ADEM’s representations (Edwards & Storkey, 2016) such that they do not contain any information about length. Alternatively, one could build a second evaluation model that assigns a score based on how easy it is to distinguish the dialogue model responses from human responses. In this case, a model that generates generic responses will easily be distinguishable and obtain a low score.
An important direction of future research is building models that can evaluate the capability of a dialogue system to have an engaging and meaningful interaction with a human. Compared to evaluating a single response, this evaluation is arguably closer to the end-goal of chatbots. However, such an evaluation is extremely challenging to do in a completely automatic way. We view the evaluation procedure presented in this paper as an important step towards this goal; current dialogue systems are incapable of generating responses that are rated as highly appropriate by humans, and we believe our evaluation model will be useful for measuring and facilitating progress in this direction.
6Note that, for our dataset, BLEU-2 almost exclusively assigns scores near 0 for both ∆w ≤ 6 and ∆w > 6, resulting in a p-value >0.05.
ACKNOWLEDGEMENTS
We’d like to thank Casper Liu for his help with the correlation code, Laurent Charlin for helpful discussions on the data collection, Jason Weston for suggesting improvements in the experiments, and Jean Harb and Emmanuel Bengio for their debugging skills. We gratefully acknowledge support from the Samsung Institute of Advanced Technology, the National Science and Engineering Research Council, and Calcul Quebec. We’d also like to thank the developers of Theano (Team et al., 2016).
APPENDIX A: FURTHER NOTES ON CROWDSOURCING DATA COLLECTION
Before conducting the primary crowdsourcing experiments to collect the dataset in this paper, we ran a series of preliminary experiments to see how AMT workers responded to different questions. Unlike the primary study, where we asked a small number of overlapping questions to determine the κ score and filtered users based on the results, we conducted a study where all responses (40 in total from 10 contexts) were overlapping. We did this for 18 users in two trials, resulting in 153 pair-wise correlation scores per trial.
In the first trial, we asked the following questions to the users, for each response:
1. How appropriate is the response overall? (overall, scale of 1-5) 2. How on-topic is the response? (topicality, scale of 1-5) 3. How specific is the response to some context? (specificity, scale of 1-5) 4. How much background information is required to understand the context? (background,
scale of 1-5)
Note that we do not ask for fluency, as the 3/4 responses for each context were written by a human (including retrieval models). We also provided the AMT workers with examples that have high topicality and low specificity, and examples with high specificity and low topicality. The background question was only asked once for each context.
We observed that both the overall scores and topicality had fairly high inter-annotator agreement (as shown in Table 2), but were strongly correlated with each other (i.e. participants would often put the same scores for topicality and overall score). Conversely, specificity (κ = 0.12) and background (κ = 0.05) had very low inter-annotator agreements.
To better visualize the data, we produce scatterplots showing the distribution of scores for different responses, for each of the four questions in our survey (Figure 5). We can see that the overall and topicality scores are clustered for each question, indicating high agreement. However, these clusters are most often in the same positions for each response, which indicates that they are highly correlated with each other. Specificity and background information, on the other hand, show far fewer clusters, indicating lower inter-annotator agreement. We conjectured that this was partially because the terms ‘specificity’ and ‘background information’, along with our descriptions of them, had a high cognitive load, and were difficult to understand in the context of our survey.
To test this hypothesis, we conducted a new survey where we tried to ask the questions for specificity and background in a more intuitive manner. We also changed the formulation of the background question to be a binary 0-1 decision of whether users understood the context. We asked the following questions:
1. How appropriate is the response overall? (overall, scale of 1-5) 2. How on-topic is the response? (topicality, scale of 1-5) 3. How common is the response? (informativeness, scale of 1-5) 4. Does the context make sense? (context, scale of 0-1)
We also clarified our description for the third question, including providing more intuitive examples. Interestingly, the inter-annotator agreement on informativeness κ = 0.31 was much higher than that for specificity in the original survey. Thus, the formulation of questions in a crowdsourcing survey has a large impact on inter-annotator agreement. For the context, we found that users either agreed highly (κ > 0.9 for 45 participants), or not at all (κ < 0.1 for 113 participants).
We also experimented with asking the overall score on a separate page, before asking questions 2-4, and found that this increased the κ agreement slightly. Similarly, excluding all scores where participants indicated they did not understand the context improved inter-annotator agreement slightly.
Due to these observations, we decided to only ask users for their overall quality score for each response, as it is unclear how much additional information is provided by the other questions in the context of dialogue. We hope this information is useful for future crowdsourcing experiments in the dialogue domain.
APPENDIX B: METRIC DESCRIPTION
BLEU BLEU (Papineni et al., 2002) analyzes the co-occurrences of n-grams in the ground truth and the proposed responses. It first computes an n-gram precision for the whole dataset:
Pn(r, r̂) =
∑ k min(h(k, r), h(k, r̂i))∑
k h(k, ri)
where k indexes all possible n-grams of length n and h(k, r) is the number of n-grams k in r. Note that the min in this equation is calculating the number of co-occurrences of n-gram k between the ground truth response r and the proposed response r̂, as it computes the fewest appearances of k in either response. To avoid the drawbacks of using a precision score, namely that it favours shorter (candidate) sentences, the authors introduce a brevity penalty. BLEU-N, where N is the maximum length of n-grams considered, is defined as:
BLEU-N := b(r, r̂) exp( N∑ n=1 βn logPn(r, r̂))
βn is a weighting that is usually uniform, and b(·) is the brevity penalty. The most commonly used version of BLEU assigns N = 4. Modern versions of BLEU also use sentence-level smoothing, as the geometric mean often results in scores of 0 if there is no 4-gram overlap (Chen & Cherry, 2014). Note that BLEU is usually calculated at the corpus-level, and was originally designed for use with multiple reference sentences.
METEOR The METEOR metric (Banerjee & Lavie, 2005) was introduced to address several weaknesses in BLEU. It creates an explicit alignment between the candidate and target responses. The alignment is based on exact token matching, followed by WordNet synonyms, stemmed tokens, and then paraphrases. Given a set of alignments, the METEOR score is the harmonic mean of precision and recall between the proposed and ground truth sentence.
Given a set of alignments m, the METEOR score is the harmonic mean of precision Pm and recall Rm between the candidate and target sentence.
Pen = γ( ch
m )θ (3)
Fmean = PmRm
αPm + (1− α)Rm (4)
Pm = |m|∑ k hk(ci)
(5)
Rm = |m|∑
k hk(sij) (6)
METEOR = (1− Pen)Fmean (7) The penalty term Pen is based on the ‘chunkiness’ of the resolved matches. We use the default values for the hyperparameters α, γ, and θ.
ROUGE ROUGE (Lin, 2004) is a set of evaluation metrics used for automatic summarization. We consider ROUGE-L, which is a F-measure based on the Longest Common Subsequence (LCS) between a candidate and target sentence. The LCS is a set of words which occur in two sentences in the same order; however, unlike n-grams the words do not have to be contiguous, i.e. there can be other words in between the words of the LCS. ROUGE-L is computed using an F-measure between the reference response and the proposed response.
R = max j
l(ci, sij)
|sij | (8)
P = max j fracl(ci, sij)|cij | (9)
ROUGEL(ci, Si) = (1 + β2)RP
R+ β2P (10)
where l(ci, sij) is the length of the LCS between the sentences. β is usually set to favour recall (β = 1.2).
APPENDIX C: LATENT VARIABLE HIERARCHICAL RECURRENT ENCODER-DECODER (VHRED)
The VHRED model is an extension of the original hierarchical recurrent encoder-decoder (HRED) model (Serban et al., 2016a) with an additional component: a high-dimensional stochastic latent variable at every dialogue turn. The dialogue context is encoded into a vector representation using the utterance-level and context-level RNNs from our encoder. Conditioned on the summary vector at each dialogue turn, VHRED samples a multivariate Gaussian variable that is provided, along with the context summary vector, as input to the decoder RNN, which in turn generates the response word-by-word. We use representations from the VHRED model as it produces more diverse and coherent responses compared to its HRED counterpart.
The VHRED model is trained to maximize a lower-bound on the log-likelihood of generating the next response:
L = logPθ̂(w1, . . . ,wN )
≥ N∑ n=1 −KL [ Qψ(zn | w1, . . . ,wn)||Pθ̂(zn | w<n) ] + EQψ(zn|w1,...,wn) [ logPθ̂(wn | zn,w<n) ] ,
where KL[Q||P ] is the Kullback-Leibler (KL) divergence between distributions Q and P . The distribution Qψ(zn | w1, . . . ,wN ) = N (µposterior(w1, . . . ,wn),Σposterior(w1, . . . ,wn)) is the approximate posterior distribution (or recognition model) which approximates the intractable true posterior distribution Pψ(zn | w1, . . . ,wN ). The posterior mean µposterior and covariance Σposterior (as well as that of the prior) are computed using a feed-forward neural network, which takes as input the concatenation of the vector representations of the past utterances and that of the current utterance.
The multivariate Gaussian latent variable in the VHRED model allows modelling ambiguity and uncertainty in the dialogue through the latent variable distribution parameters (mean and variance). This provides a useful inductive bias, which helps VHRED encode the dialogue context into a real-valued embedding space even when the dialogue context is ambiguous or uncertain, and it helps VHRED generate more diverse responses.
APPENDIX D: EXPERIMENTS & RESULTS
HYPERPARAMETERS
When evaluating our model, we conduct early stopping on an external validation set to obtain the best parameter setting. We similarly choose our hyperparameters (PCA dimension n, L1 regularization penalty γ, learning rate a, and batch size b) based on validation set results. Our best ADEM model used γ = 0.02, a = 0.01, and b = 16. For ADEM with tweet2vec embeddings, we did a similar hyperparameter searched, and used n = 150, γ = 0.01, a = 0.01, and b = 16.
ADDITIONAL RESULTS
New results on Liu et al. (2016) data In order to ensure that the correlations between word-overlap metrics and human judgements were comparable across datasets, we standardized the processing of the evaluation dataset from Liu et al. (2016). In particular, the original data from Liu et al. (2016) has a token (either ‘<first speaker>’, ‘<second speaker>’, or ‘<third speaker>’) at the beginning of each utterance. This is an artifact left-over by the processing used as input to the hierarchical recurrent encoder-decoder (HRED) model (Serban et al., 2016a). Removing these tokens makes sense for establishing the ability of word-overlap models, as they are unrelated to the content of the tweets.
We perform this processing, and report the updated results for word-overlap metrics in Table 10. Surprisingly, almost all significant correlation disappears, particularly for all forms of the BLEU score. Thus, we can conclude that the
word-overlap metrics were heavily relying on these tokens to form bigram matches between the model responses and reference responses.
Evaluation speed An important property of evaluation models is speed. We show the evaluation time on the test set for ADEM on both CPU and a Titan X GPU (using Theano, without cudNN) in Table 11. When run on GPU, ADEM is able to evaluate responses in a reasonable amount of time (approximately 2.5 minutes). This includes the time for encoding the contexts, model responses, and reference responses into vectors with the hierarchical RNN, in addition to computing the PCA projection, but does not include pre-training with VHRED. For comparison, if run on a test set of 10,000 responses, ADEM would take approximately 45 minutes.
This is significantly less time consuming than setting up human experiments at any scale. Note that we have not yet made any effort to optimize the speed of the ADEM model.
Learning curves To show that our learning procedure for ADEM really is necessary, and that the embeddings produced by VHRED are not sufficient to evaluate dialogue systems, we plot the Spearman and Pearson correlations on the test set as a function of the number of epochs in Figure 7. It is clear that, at the beginning of training, when the matrices M and N have been initialized to the identity, the model is incapable of accurately predicting human scores, and its correlation is approximately 0.
Failure analysis We now conduct a failure analysis of the ADEM model. In particular, we look at two different cases: responses where both humans and (normalized) ROUGE or BLEU-2 score highly (a score of 4 out of 5 or greater) while ADEM scores poorly (2 out of 5 or lower), and the converse, where ADEM scores the response highly while humans and either ROUGE or BLEU-2 score it poorly. We randomly sample (i.e. without cherry picking) three examples of each case, which are shown in Tables 12-13.
From Table 12, the cases where ADEM misses a good response, we can see that there are a variety of reasons for this cause of failure. In the first example, ADEM is not able to match the fact that the model response talks about sleep to the reference response or context. This is possibly because the utterance contains a significant amount of irrelevant information: indeed, the first two sentences are not related to either the context or reference response. In the second example, the model response does not seem particularly relevant to the context — despite this, the human scoring this example gave it 4/5. This illustrates one drawback of human evaluations; they are quite subjective, and often have some noise. This makes it difficult to learn an effective ADEM model. Finally, ADEM is unable to score the third response highly, even though it is very closely related to the reference response.
We can observe from the first two examples in Table 13, where the ADEM model erroneously ranks the model responses highly, that ADEM is occasionally fooled into giving high scores for responses that are completely unrelated to the context. This may be because both of the utterances are short, and short utterances are ranked higher by humans in general since they are often more generic (as detailed in Section 5). In the third example, the response actually seems to be somewhat reasonable given the context; this may be an instance where the human evaluator provided a score that was too low.
Data efficiency How much data is required to train ADEM? We conduct an experiment where we train ADEM on different amounts of training data, from 5% to 100%. The results are shown in Table 14. We can observe that ADEM is very data-efficient, and is capable of reaching a Spearman correlation of 0.4 using only half of the available training data (1000 labelled examples). ADEM correlates significantly with humans even when only trained on 5% of the original training data (100 labelled examples).

This paper propose a new evaluation metric for dialogue systems, and show it has a higher correlation with human annotation. I agree the MT based metrics like BLEU are too simple to capture enough semantic information, but the metric proposed in this paper seems to be too compliciated to explain.
On the other hand, we could also use equation 1 as a retrieval based dialogue system. So what is suggested in this paper is basically to train one dialogue model to evaluate another model. Then, the high-level question is why we should trust this model? This question is also relevant to the last item of my detail comments.
Detail comments:
- How to justify what is captured/evaluated by this metric? In terms of BLEU, we know it actually capture n-gram overlap. But for this model, I guess it is hard to say what is captured. If this is true, then it is also difficult to answer the question like: will the data dependence be a problem?
- why not build model incrementally? As shown in equation (1), this metric uses both context and reference to compute a score. Is it possible to show the score function using only reference? It will guarantee this metric use the same information source as BLEU or ROUGE.
- Another question about equation (1), is it possible to design the metric to be a nonlinear function. Since from what I can tell, the comparison between BLEU (or ROUGE) and the new metric in Figure 3 is much like a comparison between the exponential scale and the linear scale.
- I found the two reasons in section 5.2 are not convincing if we put them together. Based on these two reasons, I would like to see the correlation with average score. A more reasonable way is to show the results both with and without averaging.
- In table 6, it looks like the metric favors the short responses. If that is true, this metric basically does the opposite of BLEU, since BLEU will panelize short sentences. On the other hand, human annotators also tends to give short respones high scores, since long sentences will have a higher chance to contain some irrelevant words. Can we eliminate the length factor during the annotation? Otherwise, it is not surprise that the correlation.