HIERARCHICAL BEHAVIOR MODELING
We propose a framework for detecting action patterns from motion sequences and modeling the sensory-motor relationship of animals, using a generative recurrent neural network. The network has a discriminative part (classifying actions) and a generative part (predicting motion), whose recurrent cells are laterally connected, allowing higher levels of the network to represent high level behavioral phenomena. We test our framework on two types of tracking data, fruit fly behavior and online handwriting. Our results show that 1) taking advantage of unlabeled sequences, by predicting future motion, significantly improves action detection performance when training labels are scarce, 2) the network learns to represent high level phenomena such as writer identity and fly gender, without supervision, and 3) simulated motion trajectories, generated by treating motion prediction as input to the network, look realistic and may be used to qualitatively evaluate whether the model has learnt generative control rules.
1 INTRODUCTION
Behavioral scientists strive to decode the functional relationship between sensory input and motor output of the brain (Tinbergen, 1963; Moore, 2002). In particular, ethologists study the natural behavior of animals while neuroscientists and psychologists study behavior in a controlled environment, manipulating neural activations and environmental stimuli. These studies require quantitative measurements of behavior to discover correlations or causal relationships between behaviors over time or between behavior and stimuli; automating this process allows for more objective and precise measurements, and significantly increased throughput (Dell et al., 2014; Anderson & Perona, 2014). Many industries are also concerned with automatic measurement and prediction of human behavior, for applications such as surveillance, assisted living, sports analytics, and self driving vehicles.
Behavior is complex and may be perceived at different time-scales of resolution: position, trajectory, action, activity. While position and trajectory are geometrical notions, action and activity are semantic in nature. The analysis of behavior may therefore be divided into two steps: (a) detection and tracking, where the pose of the body over time is estimated, and (b) action/activity detection and classification, where motion is segmented into meaningful intervals, each one of which is associated with a goal or a purpose. Our work focuses on going from (a) to (b), that is to detect and classify actions from motion trajectories. We use data for which tracking and pose estimation is relatively simple, which lets us focus on modeling the temporal dynamics of pose trajectories without worrying about errors stemming from low level feature extraction.
Supervised learning is a powerful tool for learning action classifiers from expert-labeled examples (Jhuang et al., 2010; Burgos-Artizzu et al., 2012; Kabra et al., 2013; Eyjolfsdottir et al., 2014). However, it has two drawbacks. First, it requires a lot of training labels which involves time consuming and painstaking annotation. Second, behavior measurement is limited to actions that a human can perceive and believes to be important. We propose a framework that takes advantage of both labeled and unlabeled sequences; it simultaneously learns to predict future motion and detect actions, allowing the system to learn from fewer expert labels and discover unbiased behavior representations.
The framework models the sensory-motor relationship of an agent, predicting motion based on its sensory input and motion history. It can be used to simulate an agent by iteratively feeding motion predictions as input to the network and updating sensory inputs accordingly. A model that can simulate realistic behavior has learnt to emulate the generative control laws underlying behavior, which could be a useful tool for behavior analysis (Simon, 1996; Braitenberg, 1984).
Our model is constructed with the goal that it will learn to represent and discover behaviors at different semantic scales, offering an unbiased way of measuring behavior with minimal human input. Recent work by Berman et al. (2014) and Wiltschko et al. (2015) shows promising results towards unsupervised behavior representation. Compared to their work our framework offers three advantages. Our model learns a hierarchical embedding of behavior, can be trained semi-supervised to learn specific behaviors of interest, and our sensory-motor representation enables the model to learn interactive behavior of an agent with other agents and with its environment.
Our experiments focus mainly on the behavior of fruit flies, Drosophila Melanogaster, a popular model organism for the study of behavior (Siwicki & Kravitz, 2009). To explore the generality of our approach we also test our model on online handwriting data, an interesting human behavior that produces two dimensional trajectories.
To summarize our contributions:
1) We propose a framework that simultaneously models the sensory-motor relationship of an agent and classifies its actions, and can be trained with partially labeled sequences.
2) We show that motion prediction is a good auxiliary task for action classification, especially when training labels are scarce.
3) We show that simulated motion trajectories resemble trajectories from the data domain and can be manipulated by activating discriminative cell units.
4) We show that the network learns to represent high level information, such as gender or identity, at higher levels of the network and low level information, such as velocity, at lower levels.
5) We test our framework on the spontaneous and sporadic behavior of fruit flies, and the intentional and structured behavior of handwriting.
2 BACKGROUND
Hidden Markov models (HMMs) have been extensively used for sequence classification. The motivating assumption for HMMs is that there exists a process that transitions with some probability between discrete states, each of which emits observations according to some distribution, and the objective is to learn these functions given a sequence of observations and states. This model is limited in that its transition functions are linear, state space is discrete, and emission distribution is generally assumed to be Gaussian, although generalizations of the model that fall under the category of dynamic Bayesian networks are more expressive (Murphy, 2002).
Recurrent neural networks (RNNs) have recently been shown to be extremely successful in classifying time series data, especially with the popularization of long short term memory cells (Hochreiter & Schmidhuber, 1997), in applications such as speech recognition (Graves et al., 2013). RNNs have also been used for generative sequence prediction of handwriting (Graves, 2013) as well as speech synthesis (Chung et al., 2015).
Imitation learning involves learning to map a state to an action, from demonstrated sequences of actions. This is a supervised learning technique which, when implemented as an RNN, can be trained via backpropagation using action-error computed at every time step. The problem with this approach is that the domain of states that an agent is trained on consists only of states that the demonstrators encounter, and when an agent makes a mistake it finds itself in a situation never experienced during training. Reinforcement learning handles this by letting an agent explore the domain using an action policy, and updating the policy based on a goal-specific penalty or reward which may be obtained after taking several actions. This exploration can be extremely expensive, and therefore it is common to precede reinforcement learning with imitation learning to start the agent off with a reasonable policy. This strategy is used in (Mnih et al., 2015) where an agent is trained to play Atari games, and in (Silver et al., 2016) for mastering the game of GO.
Autoencoders (Rumenlhart et al., 1986) have been used in semi-supervised classification to pretrain a network on an auxiliary task, such as denoising, to prevent overfitting on a small number of labeled data (Baldi, 2012). Recent work in this area (Rasmus et al., 2015) proposes to train on the primary and auxiliary task concurrently and using lateral connections (Valpola, 2015) between encoding and decoding layers to allow higher layers of the network to focus on high level features.
Our framework takes inspiration from each of the works described here.
3 MODEL
Our model is a recurrent neural network, with long short term memory, that simultaneously classifies actions and predicts future motion of agents (insects, animals, and humans). Rather than actions being a function of the recurrent state, as is common practice, our model embeds actions in recurrent state units. This way the recurrent function encodes action transition probabilities and motion prediction is a direct function of actions, similar to an HMM. The network takes as input an agent’s motion and sensory input at every time step, and outputs the agent’s next move according to a policy, which is effectively learnt via imitation learning. Similar to autoencoders, our model has a discriminative path, used to embed high level information, and a generative path used to reconstruct the input domain, in our case filling in the future motion. Each discriminative recurrent cell is fully connected with its corresponding generative cell, allowing higher level states to represent higher level information, similar to the idea of Ladder networks (Valpola, 2015).
3.1 ARCHITECTURE
The model can be thought of as two parallel recurrent networks: The discriminative network takes as input an agent’s motion, x, and environmental sensory input, v, and propagates them up through its hidden states which encode high level information, including action labels, y. The generative network decodes the states of the discriminative network, propagating information down to predict the agent’s motion at the next time step, x̂. The two networks have the same number of layers and are connected diagonally at each layer such that the information encoded in the hidden units of the discriminative network is propagated to the corresponding layer of the generative network at the next time step. Intuitively, these can be thought of as skip connections or “shortcuts” which let low level motion information propagate directly through lower levels of the network, leaving higher levels of the network free to represent high level phenomena, such as goals or individual characteristics. Our experiments confirm this intuition. The model can be trained without any action labels, in which case the hidden state may be used to discover high level information about the data, or with action labels for a subset of the data, in which case each action is assigned to a hidden state unit and will thus contribute to subsequent motion prediction and action classification.
The flow of information through the network and the cost associated with its classification and prediction is expressed by the following equations:
Discriminative Generative Cost
h1i = f([xi, vi], h 1 i−1) ĥ L i+1 = f(h L i , ĥ L i ) Cy = ∑T i=1 Ly(yi, ŷi)
hli = f(h l−1 i , h l i−1) ĥ l i+1 = f([̂h
l+1 i+1, h l i], ĥ l i) Cx = ∑T i=1 Lx(xi+1, x̂i+1)
ŷi = (h L i (1 : N) + 1)/2 x̂i+1 = g(ĥ 1 i+1) C = λCy + (1− λ)Cx
where f is a recurrent cell function, g is a transformation, Ly computes classification loss (on frames for which labels are provided) and Lx computes motion prediction loss. The total cost, C, combines the misclassification cost, Cy , and misprediction cost, Cx, using λ to trade off the two. N is the number of labeled action classes, L the number of levels, T the number of frames, l is the layer index and i the frame index. The first N units of state hL are forced to be classification units, they are scaled from [-1 1] to [0 1] (assuming f ’s activation function is tanh) and assigned to ŷ.
The model is presented as part of a general framework where f , g, and number of levels/units are architectural choices to be optimized for each dataset. For our experiments we found that 2-3 levels of recurrent cells with 100-200 units worked well, with f as a gated recurrent unit (GRU) cell (Cho et al., 2014) and g as linear transformation. The choice of loss functions depends on the target type; sigmoid cross entropy for multitask classification (where actions can co-occur), softmax cross entropy for multiclass classification (where actions are mutually exclusive), and sum of squared differences for regression (where outputs are real valued). The optimal value for λ depends both on the output domain of Ly and Lx and whether the primary goal is classification or simulation. Data-specific model- and training parameters are described in Section 5 and further training details are discussed in supplementary material1.
3.2 MULTIMODAL PREDICTION
Evidence suggests that animal behavior is nondeterministic (Roberts et al., 2016); thus, motion prediction may be better represented as a probability distribution than a function. When future motion is multimodal, the best regression model will pick the average motion of the different modes which may not lie within any of the actual modes (visualized in supplementary material). This observation has been made by others in the context of modeling real-valued sequences with RNNs, (Graves, 2013) model the output of an RNN as a Gaussian mixture model and (Chung et al., 2015) additionally model the hidden recurrent states as random variables. We take a nonparametric approach, making no assumption about the shape of the distribution. We discretize motion into bins and treat the task of predicting future motion as independent multiclass classification problems for each motion feature, which results in a probability distribution over all bins for each dimension. More concretely, each dimension of x is assigned n bins and the target for x̂i+1 becomes the binned version of xi+1, denoted as x̃i+1, which has exactly one nonzero entry for each dimension of x. The prediction x̂i+1 then becomes a discrete distribution over the bins for each feature dimension and the motion prediction loss becomes Lx(xi+1, x̂i+1) = ∑ d(crossentropy(x̃i+1, x̂i+1)), as opposed to the Euclidean distance in the case when x is a real valued vector. The number of bins determines the granularity of the motor control; a greater number of bins means more precise motion control but is also more expensive to train.
3.3 SIMULATION
Given a model that can predict an agent’s future motion from its current state, a virtual agent can be simulated by iteratively feeding predicted motion x̂i+1 as input xi+1 to the network. We pick a bin by sampling from the distribution given by x̂i+1 and assign a real value to xi+1 by sampling uniformly from the selected bin. An agent’s perception of the environment depends on the agent’s location, and therefore sensory features vi+1 must be updated for each forward simulation step to correspond to the agent’s perspective at time i + 1. When simulating multiple agents that interact with one another, each agent is moved according to its xi+1 and then vi+1 is computed for each agent based on the new configuration of all agents.
1www.vision.caltech.edu/˜eeyjolfs/behavior_modeling
4 DATA
Our framework is agent centric, it models the behavior of each agent individually based on how it moves and senses its surroundings, including other agents. It is applicable to any data that can be represented in terms of motor control (e.g. joystick controller) and sensory input that captures context from the environment (e.g. 1st person camera). We test our model on two types of data, fly behavior and online handwriting. Both can be thought of as a type of behavior represented in the form of trajectories, but the two are complementary. First, flies behave spontaneously, performing actions of interest sporadically and in response to their environment, while handwritten text is intensional and highly structured. Second, handwriting varies significantly between different writers in terms of size, speed, slant, and proportions, while inter-fly variation is relatively small. We use 4 datasets for our experiments (listed below) with the aim to answer the following questions: 1) does motion prediction improve action classification, 2) can the model generate realistic simulations (does it learn the sensory-motor control), and 3) can the model discover novel behavioral phenomena?
Fly-vs-fly (Eyjolfsdottir et al., 2014) contains pairs of fruit flies engaging in 10 labeled courtshipand aggressive behaviors. We include this dataset in our experiments to see how our model compares with our previous action detection work which relies on handcrafted window features.
FlyBowl is a video of 10 male and 10 female fruit flies interacting and is labeled with male wing extensions which is part of their courtship behavior. With this dataset we were particularly interested in whether our model could simulate a virtual fly in a complex, dynamic environment.
SynthFly is a synthetic dataset containing a single fly moving inside of a rectangular chamber with a stationary object located in the center. The fly is synthesized to move according to the control laws listed in Figure 2. The purpose of this dataset is to test whether our model could learn generative control rules, particularly ones that enforce non-deterministic behavior (see laws 4 and 5).
IAM-OnDB (Liwicki & Bunke, 2005) contains handwritten text from 195 different writers, acquired using a smart whiteboard that records a list of (x, y) coordinates for each pen stroke. The data is weakly labeled, with each sequence separated into short lines of transcribed text. For consistency with our framework we hand annotated strokes of 10 writers, marking the start and end of the 26 lower case characters, which we use along with data from 35 unlabeled writers for our experiments.
All data, along with details about training and test splits, will be available in supplementary material.
Published as a conference paper at ICLR 2017Fly-centric features
Nicolas Gompel gompel.org/drosophilidae
Fly-centric features
Nicolas Gompel gompel.org/drosophilidae
Fly representation: Motor control features, x, describe the locomotion of a fly. The flies are tracked from video using FlyTracker2 and from the tracked fly poses we extract motion features represented in the fly’s frame of reference. The 8 motion features, displayed on top of the close-up fly3 in Figure 3, are designed such that they can animate virtual fly agents. Sensory input features, v, are inspired by a fly’s compound eye which consist of 750 compactly aligned ommatidia. Approximating its vision as a one dimensional 360◦ view, we place 72 5◦ circular sectors around a fly agent, aligned with its orientation, and project flies that overlap with a sector onto its artificial retina with intensity inversely proportional to their distance to the agent. Thus, flies close to the agent yield high intensity in several pixels and flies that are far away take up few pixels with low intensity (compare scene in Figure 3 with v sensed by the agent). We represent chamber walls similarly, projecting them onto a separate channel decreasing intensity exponentially with distance to the agent. This representation is invariant of the shape of the chamber and the number of flies present in the chamber.
In order to compare our model with methods presented in Eyjolfsdottir et al. (2014), independently of feature representation, we use the 36 features provided with the Fly-vs-Fly dataset. We assign the first 8 dimensions (describing fly’s motion) as motor control x, and the remaining features (describing fly’s position relative to the other fly, and feature derivatives) as sensory input v.
Handwriting representation: We represent the motor control, x, as (dx, dy, z) where dx and dy are the x and y displacements from the previous pen recording and z is a binary variable denoting segment visibility. We normalize dx and dy for each writer, providing invariance to writing speed, but character size (number of points per character), slant, and other variations are not explicitly accounted for. As handwriting is not influenced by a changing environment, but rather a function of the internal state and current motion of the writer, we leave the sensory input, v, empty.
5 EXPERIMENTS AND ANALYSIS
We evaluate our framework on three objectives: classification, simulation, and discovery. For classification we show the benefit of motion prediction as an auxiliary task, compare our performance on Fly-vs-Fly with previous work, and analyze the performance on IAM-OnDB. We qualitatively show that simulation results for fly behavior and handwriting look convincing, and that the model is able to learn control laws used to generate the SynthFly dataset. For discovery we show that hidden states of the model, trained only to predict motion (without any action labels), cleanly capture high level phenomena that affect behavior, such as fly gender and writer identity.
Model details: We trained a separate model for each dataset, using a sequence length of 50, a batch size of 20, and 51 bins per dimension for motion prediction. For fly behavior data we used 2 levels of GRU cells (4 cells total) of 100 units each, and for handwriting we used 3 levels of GRU cells (6 cells total) of 200 units each. Parameters were determined using a rough parameter sweep on a subset of the training data. Further training details are described in supplementary material. Our model is implemented in Tensorflow (Abadi et al., 2015).
2www.vision.caltech.edu/Tools/FlyTracker 3Original photograph from gompel.org/drosophilidae
5.1 CLASSIFICATION
Action labeling involves recording the start frame, end frame, and class label, of each action interval, which we refer to as a bout. From a sequence of frame-wise classifications, consecutive frames of the same class prediction are consolidated into a single bout. To measure both duration and counting accuracy we use the performance measures described in Eyjolfsdottir et al. (2014), namely the F1 score (harmonic mean of precision and recall), on a per-frame and per-bout level. Bout-wise precision and recall is computed by assigning predicted bouts to ground truth bouts one-to-one, maximizing intersection over overlap. F* is the harmonic mean of the F1-frame and F1-bout scores.
Our goal for classification is to reduce the number of training labels without loss in performance. To measure the benefit of motion prediction as an auxiliary task we compare our model, which we will refer to as Behavior Embedding Sensory-motor Network (BESNet), with our model without its generative part (similar to a standard RNN but with action labels embedded in hidden states, shown in Figure 5), referred to as Behavior Embedding Network (BENet). We trained both models on each dataset using 3-100% of available labels. As BESNet is trained to predict future motion it makes use of unlabeled sequences during training whereas BENet does not. Figure 4 a) shows the frame-wise F1 score for each of the 36 trained models (3 datasets, 6 label fractions, 2 model types), averaged over all action classes in a dataset. This experiment shows that motion prediction as an auxiliary task significantly improves classification performance, especially when labels are scarce.
In Figure 4 b) we compare the performance of our network with the best performing method on Flyvs-Fly, a window based support vector machine (SVM) that uses hand crafted window features and fits an HMM to the output for smoother classification – outperforming sophisticated methods such as structured SVM. For this comparison we used the features published with the dataset as described in Section 4. Although recurrent networks implicitly enable smooth classification, different actions require different levels of smoothness. To avoid over segmentation of action intervals, we smooth the output of our network by applying a flat filter, of size equal to 10% of the mean duration of each class. Our results show that filtering significantly improves the bout-wise performance and that our performance on the Fly-vs-Fly test set is comparable with that of Eyjolfsdottir et al. (2014), using no handcrafting and no context of future frames (apart from smoothing).
We applied the same type of filtering to the classification output of IAM-OnDB as we did for Flyvs-Fly and obtained an F1-(frame, bout) of (0.445, 0.585) averaged over all classes, and (0.567, 0.690) averaged over all instances (weighted average of classes). Figure 4 c) demonstrates that at the beginning of some characters there tends to be more confusion in ŷ than towards the end, which is unsurprising as the beginning of these characters looks approximately the same.
5.2 MOTION PREDICTION
Before we look at simulation results, we quantitatively measure the accuracy of one-step predictions. We compute the log-likelihood of FlyBowl test sequences under the motion prediction model: loglik(x) = ∑T−1 i=1 ∑8 d=1 log(x̃ d i+1 · x̂di+1), where x̃di+1 is the ground truth indicator vector for bins of motion dimension d, and x̂di+1 is a probability distribution over the bins predicted by the model.
We compare our model with the following motion prediction policies: 1) uniform distribution over bins, 2) distribution over bins computed from training set, 3) constant motion policy that copies previous indicator vector as motion prediction, and 4) a smooth version of 3) filtered using an optimized Gaussian kernel. The results, shown in Figure 5, demonstrate that the recurrent models learn a significantly better policy. In addition, we compare variants of our model and a standard RNN within our framework (with the same sensory-motor representation, multimodal output, and GRU cells) which shows that recurrence is essential for good motion prediction and that diagonal connections provide a slight performance gain. In Section 5.4 we show the main benefit of the diagonals.
1
0.1 0.3 0.4 0.1 0.1xif^
xif~
xif~ = bin(xfi, n_bins)
L( , ) = sum_f(crossentropy ( , )xfi+1~ xfi+1^xi+1 xi+1^
ground truth
prediction
5.3 SIMULATION
One-step prediction performance does not clearly reveal whether a model has learnt the generative process underlying the training data. In order to get a better noti n of that we look at simulations produced by the learnt models, which can be thought of as very long term predictions. As motion prediction is probabilistic, comparing long term predictions with ground truth becomes difficult as the domain of probable positions becomes exponentially large. Qualitative inspection, however, gives a good intuition about whether the simulated agent has learnt reasonable control laws.
While the underlying generative process for the motion of real flies is unknown, simulations from the model trained to imitate them suggest that the model has learnt a reasonable policy. During simulation we place no physical constraints on how the flies can move but our results show that simulated FlyBowl agents avoid collisions with the chamber walls and with other flies, and that agents are attracted to other flies and occasionally engage in courtship-like behavior. This is shown in Figure 6 and better visualized as video in supplementary material.
Simulated handwriting is easier to visualize in an image and we are used to recognizing the structure it should produce. Figure 7 shows that the model trained on IAM-OnDB produces character-like trajectories in word-like combinations. Note that handwriting is generated one (dx, dy, z) vector at a time, and each character is composed of roughly 20 such points on average. On the right hand side of Figure 7 we show that we can increase the generation of specific characters by activating their classification units (forcing their values to 1 and others to 0) during simulation.
Figure 8 shows the output of two recurrent units of the SynthFly model that indicate that the model was able to learn control rules that were designed to ensure a multimodal motion prediction target. One unit fires in correlation with either left or right wing extension, and the other toggles between a negative and positive state as the agent turns left or right to avoid the object. In supplementary material we show a video of this simulation and compare it to a simulation from the model trained with deterministic motion prediction. This comparison clearly demonstrates the benefit of treating motion prediction as a distribution over bins, as the deterministic agent quickly becomes degenerate.
b) simulated trajectories c) real trajectoriesa) 10 x 20-frame lookaheads for test flies
generated text character injectionsoriginal text
generated text character injectionsoriginal text gen t d text charact injections
r r rl l l l l l l
r r rl l l l l l l
5.4 DISCOVERY
We motivated the structure of our network, specifically the diagonal connections between discriminative and generative cells, with the intuition that it would allow higher levels of the network to better represent high level phenomena. To verify this we train models to only predict future motion, with no classification target, and visualize what the hidden states capture. We apply the model to [x, v], obtaining hidden state vectors hl and ĥl, l ∈ {1, ..., L}, and prediction x̂, map the data points (time steps of each fly/writer) from each state to 2 dimensions using t-distributed stochastic neighbor embedding (tSNE, Maaten & Hinton (2008)), and plot them in colors based on known phenomena.
In Figure 9 we plot the data points of a 3 level (L=3) model trained on IAM-OnDB in this low dimensional embedding, and color code them according to three criteria: stroke length, character class, and writer identity. The results show that stroke length is well clustered at low levels but not at high levels, characters are best clustered at mid to top discriminative levels, and writer identity is extremely well clustered at the top generative level but not at low levels. We ran the same experiment for the model trained without diagonal connections (which without a classification target is effectively a standard RNN with 6 levels of GRU cells), which did not learn to represent writer identity in any of its hidden states. Intuitively this is because the network has to carry low level information through every state to predict low level information at the other end, whereas BESNet carries it directly through the low level diagonal connections leaving higher hidden states free to capture high level information. A visualization comparing both models is shown in supplementary material along with a quantitative measurement of our observation.
The same visualization for the model trained on FlyBowl, where data points are color coded by gender and left/right wing extension, shows (Figure 9, right) that gender is very mixed in the input and output states but well separated in the top generative state, while lower level information such as wing extension is well represented at lower levels of the network.
6 CONCLUSION
We have proposed a framework for modeling the behavior of animals, that simultaneously classifies their actions and predicts their motion. We showed empirically that motion prediction (a target that requires no labeling) is a good auxiliary task for training action classifiers, especially when labels are scarce. We also showed that the generative task can be used to simulate trajectories that look natural to the human eye, and that activating classification units increases the frequency of that action in the simulation. Finally, we showed that our model lends itself well to discovery of high level information from the data, by visualizing what is captured in its hidden states.
We tested the framework on two types of data, fly behavior and online handwriting, and we anticipate that it will scale to more complex data with appropriate tuning of hyperparameters and abstraction of visual input. For example, application to human motion capture with 1st person video as sensory input might require greater model complexity to account for the higher dimensional motor control and pre-processing of the sensory input, e.g. with a convolutional neural network, to extract a higher level sensory representation before feeding it to the dynamical system.
Moving forward, we are interested in working on hierarchical label embedding in the states, assigning higher order activities to units higher in the network. Along those lines, a discrete recurrent network could be trained separately on the wealth of available text, and be placed on top of a realvalued handwriting network. We also aim to explore how this framework can be used to understand the neural mechanisms underlying the generation of behavior in flies.
7 ACKNOWLEDGMENTS
We would like to thank David J. Anderson and Charless Fowlkes for insightful discussions, and acknowledge Google and The Simons Foundation for their financial support.

While my above review title is too verbose, it would be a more accurate title for the paper than the current one (an overall better title would probably be somewhere in between).
The overall approach is interesting: all three of the key techniques (aux. tasks, skip/diagonal connections, and the use of internal labels for the kind of data available) make a lot of sense.
I found some of the results hard to understand/interpret. Some of the explanation in the discussion below has been helpful (e.g. see my earlier questions about Fig 4 and 5); the paper would benefit from including more such explanations.
It may be worthwhile very briefly mentioning the relationship of "diagonal" connections to other emerging terms for similar ideas (e.g. skip connections, etc). "Skip" seems to me to be accurate regardless of how you draw the network, whereas "diagonal" only makes sense for certain visual layouts.
In response to comment in the discussion below: "leading to less over-segmentation of action bouts" (and corresponding discussion in section 5.1 of the paper): I would be like to have a bit more about this in the paper. I have assumed that "per-bout" refers to "per-action event", but now I am not certain that I have understood this correctly (i.e. can a "bout" last for a few minutes?): given the readership, I think it would not be inappropriate to define some of these things explicitly.
In response to comment about fly behaviours that last minutes vs milliseconds: This is interesting, and I would be curious to know how classification accuracy relates to the time-scale of the behaviour (e.g. are most of the mistakes on long-term behaviours? i realize that this would only tell part of the story, e.g. if you have a behaviour that has both a long-term duration, but that also has very different short-term characteristics than many other behaviours, it should be easy to classify accuractely despite being "long-term"). If easy to investigate this, I would add a comment about it; if this is hard to investigate, it's probably not worth it at this point, although it's something you might want to look at in future.
In response to comment about scaling to human behavior: I agree that in principle, adding conv layers directly above the sensory input would be the right thing to try, but seriously: there is usually a pretty big gap between what "should" work and what actually works, as I am sure the authors are aware. (Indeed, I am sure the authors have a much more experiential and detailed understanding of the limitations of their work than I do). What I see presented is a nice system that has been demonstrated to handle spatiotemporal trajectories. The claims made should correspond to this.
I would consider adjusting my rating to a 7 depending on future revisions.


ON THE QUANTITATIVE ANALYSIS OF DECODER- BASED GENERATIVE MODELS
The past several years have seen remarkable progress in generative models which produce convincing samples of images and other modalities. A shared component of many powerful generative models is a decoder network, a parametric deep neural net that defines a generative distribution. Examples include variational autoencoders, generative adversarial networks, and generative moment matching networks. Unfortunately, it can be difficult to quantify the performance of these models because of the intractability of log-likelihood estimation, and inspecting samples can be misleading. We propose to use Annealed Importance Sampling for evaluating log-likelihoods for decoder-based models and validate its accuracy using bidirectional Monte Carlo. The evaluation code is provided at https:// github.com/tonywu95/eval_gen. Using this technique, we analyze the performance of decoder-based models, the effectiveness of existing log-likelihood estimators, the degree of overfitting, and the degree to which these models miss important modes of the data distribution.
1 INTRODUCTION
In recent years, deep generative models have dramatically pushed forward the state-of-the-art in generative modelling by generating convincing samples of images (Radford et al., 2016), achieving state-of-the-art semi-supervised learning results (Salimans et al., 2016), and enabling automatic image manipulation (Zhu et al., 2016). Many of the most successful approaches are defined in terms of a process which samples latent variables from a simple fixed distribution (such as Gaussian or uniform) and then applies a learned deterministic mapping which we will refer to as a decoder network. Important examples include variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), generative adversarial networks (GANs) (Goodfellow et al., 2014), generative moment matching networks (GMMNs) (Li & Swersky, 2015; Dziugaite et al., 2015), and nonlinear independent components estimation (Dinh et al., 2014). We refer to this set of models collectively as decoder-based models, also known as density networks (MacKay & Gibbs, 1998).
While many decoder-based models are able to produce convincing samples (Denton et al., 2015; Radford et al., 2016), rigorous evaluation remains a challenge. Comparing models by inspecting samples is labor-intensive, and potentially misleading (Theis et al., 2016). While alternative quantitative criteria have been proposed (Bounliphone et al., 2016; Im et al., 2016; Salimans et al., 2016), log-likelihood of held-out test data remains one of the most important measures of a generative model’s performance. Unfortunately, unless the decoder is designed to be reversible (Dinh et al., 2014; 2016), log-likelihood estimation in decoder-based models is typically intractable. In the case of VAE-based models, a learned encoder network gives a tractable lower bound, but for GANs and GMMNs it is not obvious how even to compute a good lower bound. Even when lower bounds are available, their accuracy may be hard to determine. Because of the difficulty of log-likelihood
evaluation, it is hard to answer basic questions such as whether the networks are simply memorizing training examples, or whether they are missing important modes of the data distribution.
The most widely used estimator of log-likelihood for GANs and GMMNs is the Kernel Density Estimator (KDE) (Parzen, 1962). It estimates the likelihood under an approximation to the model’s distribution obtained by simulating from the model and convolving the set of samples with a kernel (typically Gaussian). Unfortunately, KDE is notoriously inaccurate for estimating likelihood in high dimensions, because it is hard to tile a high-dimensional manifold with spherical Gaussians (Theis et al., 2016).
In this paper, we propose to use annealed importance sampling (AIS; (Neal, 2001)) to estimate log-likelihoods of decoder-based generative models and to obtain approximate posterior samples. Importantly, we validate this approach using Bidirectional Monte Carlo (BDMC) (Grosse et al., 2015), which provably bounds the log-likelihood estimation error and the KL divergence from the true posterior distribution for data simulated from a model. For most models we consider, we find that AIS is two orders of magnitude more accurate than KDE, and is accurate enough to perform fine-grained comparisons between generative models. In the case of VAEs, we show that AIS can be further sped up by using the recognition network to determine the initial distribution; this yields an estimator which is fast enough to be run repeatedly during training.
Using the proposed method, we analyze several scientific questions central to understanding decoderbased generative models. First, we measure the accuracy of KDE and of the importance weighting bound which is commonly used to evaluate VAEs. We find that the KDE error is larger than the (quite significant) log-likelihood differences between different models, and that KDE can lead to misleading conclusions. The importance weighted bound, while reasonably accurate, can also yield misleading results in some cases.
Second, we compare the log-likelihoods of VAEs, GANs, and GMMNs, and find that VAEs achieve log-likelihoods several hundred nats higher than the other models (even though KDE considers all three models to have roughly the same log-likelihood). Third, we analyze the degree of overfitting in VAEs, GANs, and GMMNs. Contrary to a commonly proposed hypothesis, we find that GANs and GMMNs are not simply memorizing their training data; in fact, their log-likelihood gaps between training and test data are much smaller relative to comparably-sized VAEs. Finally, by visualizing (approximate) posterior samples obtained from AIS, we observe that GANs miss important modes of the data distribution, even ones which are represented in the training data.
We emphasize that none of the above phenomena can be measured using KDE or the importance weighted bound, or by inspecting samples. (See Fig. 1 for an example where it is tricky to compare models based on samples.) While log-likelihood is by no means a perfect measure, we find that the ability to accurately estimate log-likelihoods of decoder-based models yields crucial insight into their behavior and suggests directions for improving them.
2 BACKGROUND
2.1 DECODER-BASED GENERATIVE MODELS
In generative modelling, a decoder network is often used to define a generative distribution by transforming samples from some simple distribution (e.g. normal) to the data manifold. In this
paper, we consider three kinds of decoder-based generative models: Variational Autoencoder (VAE) (Kingma & Welling, 2014), Generative Adversarial Network (GAN) (Goodfellow et al., 2014), and Generative Moment Matching Network (GMMN) (Li & Swersky, 2015; Dziugaite et al., 2015).
2.1.1 VARIATIONAL AUTOENCODER
A variational autoencoder (VAE) (Kingma & Welling, 2014) is a probabilistic directed graphical model. It is defined by a joint distribution over a set of latent random variables z and the observed variables x: p(x, z) = p(x|z)p(z). The prior over the latent random variables, p(z), is usually chosen to be a standard Gaussian distribution. The data likelihood p(x|z) is usually a Gaussian or Bernoulli distribution whose parameters depend on z through a deep neural network, known as the decoder network. It also uses an approximate inference model called an encoder or recognition network, that serves as a variational approximation q(z|x) to the posterior p(z|x). The decoder network and the encoder networks are jointly trained to maximize the evidence lower bound (ELBO):
log p(x) ≥ Eq(z|x)[log p(x|z)]−KL(q(z|x)||p(z)) (1) In addition, the reparametrization trick is used to reduce the variance of the gradient estimate.
2.1.2 GENERATIVE ADVERSARIAL NETWORK (GAN)
A generative adversarial network (GAN) (Goodfellow et al., 2014) is a generative model trained by a game between a decoder network and a discriminator network. It defines the generative model by sampling the latent variable z from some simple prior distribution p(z) (e.g., Gaussian) followed through the decoder network. The discriminator network D(·) outputs a probability of a given sample coming from the data distribution. Its task is to distinguish samples from the generator distribution from real data. The decoder network, on the other hand, tries to produce samples as realistic as possible, in order to fool the discriminator into accepting its outputs as being real. The competition between the two networks results in the following minimax problem:
min G max D Ex∼pdata [logD(x)] + Ez∼p(z)[log(1−D(G(z))] (2)
Unlike VAE, the objective is not explicitly related to the log-likelihood of the data. Moreover, the generative distribution is a deterministic mapping, i.e., p(x|z) is a Dirac delta distribution, parametrized by the deterministic decoder. This can make data likelihood ill-defined, as the probability density of any particular point x can be either infinite, or exactly zero.
2.1.3 GENERATIVE MOMENT MATCHING NETWORK (GMMN)
Generative moment matching networks (GMMNs) (Li & Swersky, 2015; Dziugaite et al., 2015) adopt maximum mean discrepancy (MMD) as the training objective, a moment matching criterion where kernel mean embedding techniques are used to avoid unnecessary assumptions of the distributions. It has the same issue as GAN in that the log-likelihood is undefined.
2.2 ANNEALED IMPORTANCE SAMPLING
We are interested in estimating the probability p(x) = ∫ p(z)p(x|z) dz a model assigns to an observation x. This is equivalent to computing the normalizing constant of the unnormalized distribution f(z) = p(z, x). One naïve approach is likelihood weighting, where one samples {z(k)}Kk=1 ∼ p(z) and averages the conditional likelihoods p(x|z(k)). This is justified by the following identity:
p(x) =
∫ p(x, z)
p(z) p(z) dz = Ez∼p(z)[p(x|z)] (3)
Likelihood weighting can be viewed as simple importance sampling, where the proposal distribution is the prior p(z) and the target distribution is the posterior p(z|x). Unfortunately, importance sampling works well only when the proposal distribution is a good match for the target distribution. For the models considered in this paper, the (very broad) prior can be drastically different than the (highly concentrated) posterior, leading to inaccurate estimates of the likelihood.
Annealed importance sampling (AIS; Neal, 2001) is a Monte Carlo algorithm commonly used to estimate (ratios of) normalizing constants. Roughly speaking, it computes a sequence of importance
sampling based estimates, each of which is stable because it involves two distributions which are very similar. In particular, suppose one is interested in estimating the normalizing constant Z = ∫ f(z) dz of an unnormalized distribution f(z). (In the likelihood estimation setting, f(z) = p(z, x) and Z = p(x).) One must specify a sequence of distributions q1, ..., qT , where qt = ft/Zt, and fT = f is the target distribution. It is required that one can obtain one or more exact samples from the initial distribution q1. One must also specify a sequence of reversible MCMC transition operators T1, ..., TT , where Tt leaves qt invariant. AIS produces a (nonnegative) unbiased estimate of the ratio ZT /Z1 as follows: first, we sample a random initial state z1 ∼ q1 and set the initial weight w1 = 1. For every stage t ≥ 2 we update the weight w and sample the state zt according to
wt ← wt−1 ft(zt−1)
ft−1(zt−1) zt ∼ Tt(z|zt−1) (4)
As demonstrated by Neal (2001), this procedure produces a nonnegative weight wT such that E[wT ] = ZT /Z1. Typically, Z1 is known, so one computes multiple independent AIS weights {w(K)T }Kk=1 and obtains the unbiased estimate ẐT = Z1 1 K ∑K k=1 w (K) T . In the likelihood estimation setting, Z1 = 1 and ZT = p(x), so we denote this estimator as p̂(x). Typically, the unnormalized intermediate distributions are simply defined to be geometric averages ft(z) = f1(z) 1−βtfT (z) βt , where the βt are monotonically increasing parameters with β1 = 0 and βT = 1. For f1(z) = p(z) and fT (z) = p(z, x), this gives
ft(z) = p(z) p(x|z)βt . (5)
As shown by Neal (2001), under certain regularity conditions, the variance of ẐT tends to zero as the number of intermediate distributions is increased. AIS is very effective in practice, and has been used to estimate normalizing constants of complex high-dimensional distributions (Salakhutdinov & Murray, 2008).
2.3 BIDIRECTIONAL MONTE CARLO
AIS provides a nonnegative unbiased estimate p̂(x) of p(x). However, it is often more practical to estimate p(x) in the log space, i.e. log p(x), because of underflow problem of dealing with many products of probability measure. In general, we note that logarithm of a nonnegative unbiased estimate is a stochastic lower bound of the log estimand (Grosse et al., 2015). In particular, log p̂(x) is a stochastic lower bound on log p(x), satisfying E[log p̂(x)] ≤ log p(x) and Pr(log p̂(x) > log p(x) + b) < e−b.
Grosse et al. (2015) pointed out that if AIS is run in reverse starting from an exact posterior sample, it yields an unbiased estimate of 1/p(x), which (by the above argument) can be seen as a stochastic upper bound on log p(x). The combination of lower and upper bounds from forward and reverse AIS is known as bidirectional Monte Carlo (BDMC). In many cases, the combination of bounds can pinpoint the true value quite precisely. While posterior sampling is just as hard as log-likelihood estimation (Jerrum et al., 1986), in the case of log-likelihood estimation for simulated data, one has available a single exact posterior sample: the parameters and/or latent variables which generated the data. Because this trick is only applicable to simulated data, BDMC is most useful for measuring the accuracy of a log-likelihood estimator on simulated data.
Grosse et al. (2016) observed that BDMC can also be used to validate posterior inference algorithms, as the gap between upper and lower bounds is itself a bound on the KL divergence of approximate samples from the true posterior distribution.
3 METHODOLOGY
For a given generative distribution p(x, z) = p(z)p(x|z), our task is to measure the log-likelihood of test examples log p(xtest). We first discuss how we define the generative distribution for decoderbased networks. For VAE, the generative distribution is defined in the standard way, where p(z) is a standard normal distribution and p(x|z) is a normal distribution parametrized by mean µθ(z) and σθ(z), predicted by the generator given the latent code. However, the observation distribution for GANs and GMMNs is typically taken to be a delta function, so that the model’s distribution covers
only a submanifold of the space of observables. In order for the likelihood to be well-defined, we follow the same assumption made when evaluating using Kernel Density Estimator (Parzen, 1962): we assume a Gaussian observation model with a fixed variance hyperparameter σ2. We will refer to the distribution defined by this Gaussian observation model as pσ .
Observe that the KDE estimate is given by
p̂σ(x) = 1
K K∑ k=1 pσ(x|z(k)), (6)
where {z(k)}Kk=1 are samples from the prior p(z). This is equivalent to likelihood weighting for the distribution pσ, which is an instance of simple importance sampling (SIS). Because SIS is an unbiased estimator of the likelihood, log p̂σ(x) is a stochastic lower bound on log pσ(x) (Grosse et al., 2015). Unfortunately, SIS can result in very poor estimates when the evidence has low prior probability (i.e. the posterior is very dissimilar to the prior). This suggests that AIS might be able to yield much more accurate log-likelihood estimates under pσ. We note that KDE can be viewed as a special case of AIS where the number of intermediate distributions is set to 0.
We now describe specifically how we carry out evaluation using AIS. In most of our experiments, we choose the initial distribution of AIS to be p(z), the same prior distribution used in training decoderbased models. If the model provides an encoder network (e.g., VAE), we can take the approximated distribution predicted by the encoder q(z|x) as the initial distribution of the AIS chain. For continuous data, we define the unnormalized density of target distribution to be the joint generative distribution with the Gaussian noise model, pσ(x, z) = pσ(x|z)p(z). For the small subset of experiments done on the binary data, we define the observation model to be a Bernoulli model with mean predicted by the decoder. Our intermediate distributions are geometric averages of the prior and posterior, as in Eqn. 5. Since all of our experiments are done using continuous latent space, we use Hamiltonian Monte Carlo (Neal, 2010) as the transition operator for sampling latent samples along annealing. The evaluation code is provided at https://github.com/tonywu95/eval_gen.
4 RELATED WORK
AIS is known to be a powerful technique of estimating the partition function of the model. One influential example was the use of AIS to evaluate deep belief networks (Salakhutdinov & Murray, 2008). Although we used the same technique, the problem we consider is completely different. First of all, the model they consider is undirected graphical models, whereas decoder-based models are directed graphical models. Secondly, their model has a well-defined probabilistic density function in terms of energy function, whereas we need to consider different probabilistic model for one in which the the likelihood is ill-defined. In addition, we validate our estimates using BDMC.
Theis et al. (2016) give an in-depth analysis of issues that might come up in evaluating generative models. They also point out that a model that completely fails at modelling the proportion of modes of the distribution might still achieve a high likelihood score. Salimans et al. (2016) propose an image-quality measure which they find to be highly correlated with human visual judgement. They propose to feed the samples x of the model to the “inception” model to obtain a conditional label distribution p(y|x), and evaluate the score defined by expExKL(p(y|x)||p(y)), which is motivated by having a low entropy of p(y|x) but a large entropy of p(y). However, the measure is largely based on visual quality of the sample, and we argue that the visual quality can be a misleading way to evaluate a model.
5 EXPERIMENTS
5.1 DATASETS
All of our experiments were performed on the MNIST dataset of images of handwritten digits (LeCun et al., 1998). For consistency with prior work on evaluating decoder-based models, most of our experiments used the continuous inputs. We dequantized the data following Uria et al. (2013), by adding a uniform noise of 1256 to the data and rescaling it to be in [0, 1]
D after dequantization. We use the standard split of MNIST into 60,000 training and 10,000 test examples, and used 50,000 images from the training set for training, and remaining 10,000 images for validation. In addition,
some of our experiments used the binarized MNIST dataset with a Bernoulli observation model (Salakhutdinov & Murray, 2008).
5.2 MODELS
For most of our experiments, we considered two decoder architectures: a small one with 10 latent dimensions, and a larger one with 50 latent dimensions. We use standard Normal distribution as prior for training all of our models. All layers were fully connected, and the number of units in each layer was 10–64–256–256-1024–784 for the smaller architecture and 50–1024–1024–1024–784 for the larger one. We trained both architectures using the VAE, GAN, and GMMN objectives, resulting in six networks which we refer to as VAE-10, VAE-50, etc. In general, the larger architecture performed substantially better on both the training and test sets, but we analyze the smaller architecture as well because it better highlights some of the differences between the training criteria. Additional architectural details are given in Appendix A.1.
In order to enable a direct comparison between training criteria, all models used a spherical Gaussian observation model with fixed variance. This is consistent with previous protocols for evaluating GANs and GMMNs. However, we note that this observation model is a nontrivial constraint on the VAEs, which could instead be trained with a more flexible diagonal Gaussian observation model where the variances depend on the latent state. Such observation models can easily achieve much higher log-likelihood scores, for instance by noticing that boundary pixels are always close to 0. (E.g., we trained a VAE with the more general observation model which achieved a log-likelihood of at least 2200 nats on continuous MNIST.) Therefore, the log-likelihood values we report should not be compared directly against networks which have a more flexible observation model.
5.3 VALIDATION OF LOG-LIKELIHOOD ESTIMATES
Before we analyze the performance of the trained networks, we must first determine the accuracy of the log-likelihood estimators. In this section, we validate the accuracy of our AIS-based estimates using BDMC. We then analyze the error in the KDE and IWAE estimates and highlight some cases where these measures miss important phenomena.
5.3.1 VALIDATION OF AIS
We used AIS to estimate log-likelihoods for all models under consideration. Except where otherwise specified, all AIS estimates were obtained using 16 independent chains, 10,000 intermediate distributions of the form in Eqn. 5, and a transition operator consisting of one proposed HMC trajectory with 10 leapfrog steps.1 Following Ranzato et al. (2010), the HMC stepsize was tuned to achieve an acceptance rate of 0.65 (as recommended by Neal (2010)).
For all six models, we evaluated the accuracy of this estimation procedure using BDMC on data sampled from the model’s distribution on 1000 simulated examples. The gap between the loglikelihood estimates produced by forward AIS (which gives a lower bound) and reverse AIS (which gives an upper bound) bounds the error of the AIS estimates on simulated data. We refer to this gap as the BDMC gap. For five of the six networks under consideration, we found the BDMC gap to be less than 1 nat. For the remaining model (GAN-50), the gap was about 10 nats. Both gaps are much smaller than our measured log-likelihood differences between models. If these gaps are representative of the true error in the estimates on the real data, then this indicates AIS is accurate enough to make fine-grained comparisons between models and to benchmark other log-likelihood estimators. (The BDMC gap is not guaranteed to hold for the real data, although Grosse et al. (2016) found the behavior of AIS to match closely between real and simulated data.)
5.3.2 HOW ACCURATE IS KERNEL DENSITY ESTIMATION?
Kernel density estimation (KDE) (Parzen, 1962) is widely used to evaluate decoder-based models (Goodfellow et al., 2014; Li & Swersky, 2015), and a variant was proposed in the setting of evaluating Boltzmann machines (Bengio et al., 2013). Papers reporting KDE estimates often caution that the
1We used the HMC implementation from http://deeplearning.net/tutorial/ deeplearning.pdf
KDE is not meant to be applied in high-dimensional spaces and that the results might therefore be inaccurate. Nevertheless, KDE remains the standard protocol for evaluating decoder-based models. We analyzed the accuracy of the KDE estimates by comparing against AIS. Both estimates are stochastic lower bounds on the true log-likelihood (see Section 3), so larger values are guaranteed (with high probability) to be more accurate.
For each estimator, we varied one parameter influencing the computational budget; for AIS, this was the number of intermediate distributions (chosen from {100, 500, 1000, 2000, 10000}), and for KDE, it was the number of samples (chosen from {10000, 100000, 500000, 1000000, 2000000}). Using GMMN-10 for illustration, we plot both log-likelihood estimates 100 simulated examples as a function of evaluation time in Fig. 2(b). We also plot the upper bound of likelihood given by running AIS in reverse direction. We see that the BDMC gap approaches to zero, validating the accuracy of AIS. We also see that the AIS estimator achieves much more accurate estimates during similar evaluation time. Furthermore, the KDE estimates appear to level off, suggesting one cannot obtain accurate results even using orders of magnitude more samples.
The KDE estimation error also impacts the estimate of the observation noise σ, since a large value of σ is needed for the samples to cover the full distribution. We compared the log-likelihoods estimated by AIS and KDE with varying choices of σ on 100 training and validation examples of MNIST. We used 1 million simulated samples for KDE evaluation, which takes almost the same time as running AIS estimation. In Fig. 2(a), we show the log-likelihood of GAN-50 estimated by KDE and AIS as a function of σ. Because the accuracy of KDE declines sharply for small σ values, it creates a strong bias towards large σ.
5.3.3 HOW ACCURATE IS THE IWAE BOUND?
In principle, one could estimate VAE likelihoods using the VAE objective function (which is a lower bound on the true log-likelihood). However, it is more common to use importance weighting, where the proposal distribution is computed by the recognition network. This is provably more accurate than the VAE bound (Burda et al., 2016). Because the importance weighted estimate corresponds to the objective function used by the Importance Weighted Autoencoder (IWAE) (Burda et al., 2016), we will refer to it as the IWAE bound.
On continuous MNIST, the IWAE bound underestimated the true log-likelihoods by at least 33.2 nats on the training set and 187.4 nats on the test set. While this is considerably more accurate than KDE, the error is still significant. Interestingly, this result also suggests that the recognition network overfits the training data.
Since VAE and IWAE results have customarily been reported on binarized MNIST, we additionally trained an IWAE in this setting. The training details are given in Appendix A.2. To show the practicality of our method, we evaluated the IWAE on the full 10000 test using AIS and IWAE bound, with different choices of intermediate distribution and number of simulated samples, shown in Table 1. We also evaluate AIS with the initial distribution defined by encoders of VAEs, denoted as AIS+encoder. We find that the IWAE bound underestimates the true value by at least 1 nat, which is a large difference by the standards of binarized MNIST. (E.g., it represents about half of the gap between a state-of-the-art permutation-invariant model (Tran et al., 2016) and one which exploits structure (van den Oord et al., 2016).) The AIS and IWAE estimates are compared in terms of evaluation time in Fig. 2 (c).
5.4 SCIENTIFIC FINDINGS
Having validated the accuracy of AIS, we now use it to analyze the effectiveness of various training criteria. We also highlight phenomena which would not be observable using existing log-likelihood estimators or by inspecting samples. For all experiments in this section, we used 10,000 intermediate distributions for AIS, 1 million simulated samples for KDE, and 200,000 importance samples for the IWAE bound. (These settings resulted in similar computation time for all three estimators.)
5.4.1 MODEL LIKELIHOOD COMPARISON
We evaluated the trained models using AIS and KDE on 1000 test examples of MNIST; results are shown in Table 2. We find that for all three training criteria, the larger architectures consistently outperformed the smaller ones. We also find that for both the 10- and 50-dimensional architectures, the VAEs achieved substantially higher log-likelihoods than GANs or GMMNs. It is not surprising that the VAEs achieved higher likelihood, because they were trained using a likelihood-based objective while the GANs and GMMNs were not. However, it is interesting that the difference in log-likelihoods was so large; in the rest of this section, we attempt to analyze what exactly is causing this large difference.
We note that the KDE errors were of the same order of magnitude as the differences between models, indicating that it cannot be used reliably to compare log-likelihoods. Furthermore, KDE did not identify the correct ordering of models; for instance, it estimated a lower log-likelihood for VAE50 than for VAE-10, even though its true log-likelihood was almost 300 nats higher. KDE also underestimated by an order of magnitude the log-likelihood improvements that resulted from using the larger architectures. (E.g., it estimated a 15 nat difference between GMMN-10 and GMMN-50, even though the true difference was 247 nats as estimated by AIS.)
These differences are also hard to observe simply by looking at samples; for instance, we were unable to visually distinguish the quality of samples for GAN-10 and GAN-50 (see Fig. 1), even though their log-likelihoods differed by almost 300 nats on both the training and test sets.
5.4.2 MEASURING THE DEGREE OF OVERFITTING
One question that arises in evaluation of decoder-based generative models is whether they memorize parts of the training dataset. One cannot test this by looking only at model samples. The commonly reported nearest-neighbors from the training set can be misleading (Theis et al., 2016), and interpolation in the latent space between different samples can be visually appealing, but does not provide a quantitative measure of the degree of generalization.
To analyze the degree of overfitting, Fig. 3 shows training curves for three networks as measured by AIS, KDE, and the IWAE bound. We observe that GAN-50’s training and test log-likelihoods are nearly identical throughout training, disconfirming the hypothesis that it was memorizing training data. Both GAN-50 and GMMN-50 overfit less than VAE-50.
We also observed two phenomena which could not be measured using existing techniques. First, in the case of VAE-50, the IWAE lower bound starts to decline after 200 epochs, while the AIS estimates hold steady, suggesting it is the recognition network rather than the generative network which is overfitting most. Second, the GMMN-50 training and validation error continue to improve at 10,000 epochs, even though KDE erroneously indicates that performance has leveled off.
5.4.3 HOW APPROPRIATE IS THE OBSERVATION MODEL?
Appendix B addresses the questions of whether the spherical Gaussian observation model is a good fit and whether the log-likelihood differences could be an artifact of the observation model. We find that all of the models can be substantially improved by accounting for non-Gaussianity, but that this effect is insufficient to explain the gap between the VAEs and the other models.
5.4.4 ARE THE NETWORKS MISSING MODES?
It was previously observed that one of the potential failure modes of Boltzmann machines is to fail to generate one or more modes of a distribution or to drastically misallocate probability mass between modes (Salakhutdinov & Murray, 2008). Here we analyze this for decoder-based models.
First, we ask a coarse-grained version of this question: do the networks allocate probability mass correctly between the 10 digit classes, and if not, can this explain the difference in log-likelihood scores? In Fig. 1, we see that GAN-50’s distribution of digit classes was heavily skewed: out of 100 samples, it generated 37 images of 1’s, but only a single 2. This appears to be a large effect, but it does not explain the magnitude of the log-likelihood difference from VAEs. In particular, if the allocation of digit classes were off by a factor of 10, this effect by itself could cost at most log 10 ≈ 2.3 nats of log-likelihood. Since VAE-50 outperformed GAN-50 by 364 nats, this effect cannot explain the difference.
However, MNIST has many factors of variability beyond simply the 10 digit classes. In order to determine whether any of the models missed more fine-grained modes, we visualized posterior samples for each model conditioned on training and test images. In particular, for each image x under consideration, we used AIS to approximately sample z from the posterior distribution p(z|x), and then ran the decoder on z. While these samples are approximate, Grosse et al. (2016) point out that the BDMC gap also bounds the KL divergence of approximate samples from the true posterior. With the exception of GAN-50, our BDMC gaps were on the order of 1 nat, suggesting our approximate posterior samples are fairly representative. The results are shown in Fig. 4. Further posterior visualizations for digit class 2 (the most difficult for the models we considered) are shown in Appendix C.
Both VAEs’ posterior samples match the observations almost perfectly. (We observed a few poorly reconstructed examples on the test set, but not on the training set.) The GANs and GMMNs fail to
reconstruct some of the examples on both the training and validation sets, suggesting that they failed to learn some modes of the distribution.
ACKNOWLEDGMENTS
We like to thank Yujia Li for providing his original GMMN model and codebase, and thank Jimmy Ba for advice on training GANs. Ruslan Salakhutdinov is supported in part by Disney and ONR grant N000141310721. We also thank the developers of Lasagne (Battenberg et al., 2014) and Theano (Al-Rfou et al., 2016).
A NETWORK ARCHITECTURES/TRAINING
A.1 MODELS ON CONTINUOUS MNIST
The decoders have all fully connected layers, and the number of units in each layer was 10–64–256– 256-1024–784 for the smaller architecture and 50–1024–1024–1024–784 for the larger one. Other architecture details are summarized as follows.
• For GAN-10, we used a discriminator with the architecture 784-512-256-1, where each layer used dropout with parameter 0.5. For GAN-50, we used a discriminator with architecture 784-4096-4096-4096-4096-1. All hidden layers used dropout with parameter 0.8. All hidden layers in both networks used the tanh activation function, and the output layers used the logistic function.
• The larger model uses an encoder of an architecture 784-1024-1024-1024-100. We add dropout layer between each hidden layer, with a dropout rate of 0.2. The smaller model uses an encoder of an architecture 784-256-64-20. Generator’s hidden layers use tanh activation function, and the output layer uses sigmoid unit. Encoder’s hidden layers use tanh activation function, and the output layer uses linear activation.
• GMMN: The hidden layers use ReLU activation function, and the output layer uses sigomid unit.
For training GAN/VAE, we use our own implementation. We use Adam for optimization, and perform grid search of learning rate from {0.001, 0.0001, 0.00001}. For training GMMN, we take the implementation from https://github.com/yujiali/gmmn.git. Following the implementation, we use SGD with momentum for optimization, and perform grid search of learning rate from {0.1, 0.5, 1, 2}, with momentum 0.9.
A.2 MODELS ON BINARIZED MNIST
Its decoder has the architecture 50-200-200-784 with all tanh hidden layers and sigmoid output layer, and its encoder is symmetric in architecture, with linear output layer. We take the implementation at https://github.com/yburda/iwae.git for training the IWAE model.The IWAE bound was computed with 50 samples during training. We keep all the hyperparameter choices the same as in the implementation.
B HOW PROBLEMATIC IS THE GAUSSIAN OBSERVATION MODEL?
In this section, we consider whether the difference in log-likelihood between models could be an artifact of the Gaussian noise model (which we know to be a poor fit). In principle, the Gaussian noise assumption could be unfair to the GANs and GMMNs, because the VAE training uses the correct
observation model, while the GAN and GMMN objectives do not have any particular observation model built in.
To determine the size of this effect, we evaluated the models under a different regime where, instead of choosing a fixed value of the observation noise σ on a validation set, σ was tuned independently for each example.2 This is not a proper generative model, but it can be viewed as an upper bound on the log-likelihood that would be achievable with a heavy-tailed and radially symmetric noise model.3 Results are shown in Table 3. We see that adapting σ for each example results in a log-likelihood improvement between 30 and 100 nats for all of the networks. In general, the examples which show the largest performance jump are images of 1’s (which prefer smaller σ) and 2’s (which prefer larger σ). This is a significant effect, and suggests that one could significantly improve the log-likelihood scores by picking a better observation model. However, this effect is smaller in magnitude than the differences between VAE and GAN/GMMN log-likelihoods, so it fails to explain the likelihood difference.
C POSTERIOR VISUALIZATION OF DIGIT “2"
According to the log-likelihood evaluation, we find digit “2" is the hardest digit for modelling. In this section we investigate the quality of modelling “2" of each model. We randomly sampled a fixed set of 100 samples of digit “2" from training data and compare whether model capture this mode. We show the plots of “2" for GAN-10, GAN-50, VAE-10 and true data in the following figures for illustration. We see that GAN-10 fails at capturing many instances of digit “2" in the training data! We see instead of generating “2", it tries to generate digit “1", “7" “9", “4", “8" from reconstruction. GAN-50 does much better, its reconstruction are all digit “2" and there is only some style difference from the true data. VAE-10 totally dominates this competition, where it perfectly reconstructs all the samples of digit “2". We emphasize if directly sampling from each model, samples look visually indistinguishable (see Fig. 1), but we can clearly see differences in posterior samples.
2We pick the best variance parameter among {0.005, 0.01, 0.015, 0.02, 0.025} for each training/validation examples when evaluating GAN-50 and GMMN-50 and {0.015, 0.02, 0.025, 0.03, 0.035} when evaluating GAN-10 and GMMN-10.
3In particular, heavy-tailed radially symmetric distributions can be viewed as Gaussian scale mixtures (Wainwright & Simoncelli, 1999). I.e., one has a prior distribution on σ (possibly learned) and integrates it out for each test example. Clearly the probability under such a mixture cannot exceed the maximum value with respect to σ.

# Review
This paper proposes a quantitative evaluation for decoder-based generative models that use Annealed Importance Sampling (AIS) to estimate log-likelihoods. Quantitative evaluations are indeed much needed since for some models, like Generative Adversarial Networks (GANs) and Generative Moment Matching Networks (GMMNs), qualitative evaluation of samples is still frequently used to assess their generative capability. Even though, there exist quantitative evaluations like Kernel Density Estimation (KDE), the authors show how AIS is more accurate than KDE and how it can be used to perform fine-grained comparison between generative models (GAN, GMMs and Variational Autoencoders (VAE)).
The authors report empirical results comparing two different decoder architectures that were both trained, on the continuous MNIST dataset, using the VAE, GAN and GMMN objectives. They also trained an Importance Weighted Autoencoder (IWAE) on binarized MNIST and show that, in this case, the IWAE bound underestimates the true log-likelihoods by at least 1 nat (which is significant for this dataset) according to the AIS evaluation of the same model.
# Pros
Their evaluation framework is public and is definitely a nice contribution to the community.
This paper gives some insights about how GAN behaves from log-likelihood perspective. The authors disconfirm the commonly proposed hypothesis that GAN are memorizing training data. The authors also observed that GANs miss important modes of the data distribution.
# Cons/Questions
It is not clear for me why sometimes the experiments were done using different number of examples (100, 1000, 10000) coming from different sources (trainset, validset, testset or simulation/generated by the model). For instance, in Table 2 why results were not reported using all 10,000 examples of the testing set?
It is not clear why in Figure 2c, AIS is slower than AIS+encoder? Is the number of intermediate distributions the same in both?
16 independent chains for AIS seems a bit low from what I saw in the literature (e.g. in [Salakhutdinov & Murray, 2008] or [Desjardins etal., 2011], they used 100 chains). Could it be that increasing the number of chains helps tighten the confidence interval reported in Table 2?
I would have like the authors to give their intuitions as to why GAN50 has a BDMC gap of 10 nats, i.e. 1 order of magnitude compared to the others?
# Minor comments
Table 1 is not referenced in the text and lacks description of what the different columns represent.
Figure 2(a), are the reported values represents the average log-likelihood of 100 (each or total?) training and validation examples of MNIST (as described in Section 5.3.2).
Figure 2(c), I'm guessing it is on binarized MNIST? Also, why are there fewer points for AIS compared to IWAE and AIS+encoder?
Are the BDMC gaps mentioned in Section 5.3.1 the same as the ones reported in Table2 ?
Typo in caption of Figure 3: "(c) GMMN-10" but actually showing GMMN-50 according to the graph title and subcaption.