IN DEEP GENERATIVE MODELS
Deep generative models provide a powerful and flexible means to learn complex distributions over data by incorporating neural networks into latent-variable models. Variational approaches to training such models introduce a probabilistic encoder that casts data, typically unsupervised, into an entangled representation space. While unsupervised learning is often desirable, sometimes even necessary, when we lack prior knowledge about what to represent, being able to incorporate domain knowledge in characterising certain aspects of variation in the data can often help learn better disentangled representations. Here, we introduce a new formulation of semi-supervised learning in variational autoencoders that allows precisely this. It permits flexible specification of probabilistic encoders as directed graphical models via a stochastic computation graph, containing both continuous and discrete latent variables, with conditional distributions parametrised by neural networks. We demonstrate how the provision of dependency structures, along with a few labelled examples indicating plausible values for some components of the latent space, can help quickly learn disentangled representations. We then evaluate its ability to do so, both qualitatively by exploring its generative capacity, and quantitatively by using the disentangled representation to perform classification, on a variety of models and datasets.
1 INTRODUCTION
Reasoning in complex perceptual domains such as vision often requires the ability to effectively learn flexible representations of high-dimensional data, interpret the representations in some form, and understand how the representations can be used to reconstruct the data. The ability to learn representations is a measure of how well one can capture relevant information in the data. Being able to interpret the learned representations is a measure of extracting consistent meaning in an effort to make sense of them. Having the ability to reliably reconstruct the data, a tool for predictive synthesis, can aid in model diagnosis, enable successful transfer learning, and improve generality. Such tasks are typically best addressed by generative models, as they exhibit the flexibility required to satisfy all three facets. Discriminative models primarily attend to the first two, learning flexible representations and conforming to some interpretable space (e.g. classification domain) but don’t perform the predictive synthesis task.
Probabilistic graphical models (Koller & Friedman, 2009; Murphy, 2012) are a framework for generative modelling that enables specifying a joint probability distribution on a richly semantic representation space. As good a fit as they are for specification and representation, the learning process for both the analysis and synthesis tasks typically suffers in complex perceptual domains such as vision. This is because constructing a generative model requires explicitly specifying the conditional distribution of the observed data given latent variables of interest. In practice, designing such
likelihood functions by hand is incredibly challenging, and applying generative models to vision data often requires extensive and significant feature engineering to be successful. One approach to alleviate some of this hardship involves the development of deep generative models: generative models that employ neural networks to learn, automatically from data, the unknown conditional distribution in the model. They function as flexible feature learners, where the features are encoded in the posterior distribution over the latent variables in the model. Recent work exploring the effectiveness of such models (e.g. Kingma & Welling (2014); Kulkarni et al. (2015b); Goodfellow et al. (2014)) has shown considerable promise in being able to address the fundamental issues in performing this task. These models however are typically unsupervised, learning representations that are not directly amenable to human interpretation. Any interpretability or disentanglement of the learned representation is observed or extracted after learning has been performed, by exploring the latent space along its non-specific axes of variation. A more recent approach by Chen et al. (2016) involves imposition of information-theoretic constraints to better separate factors of variation, but here too, any interpretability is only established post facto.
While such approaches have considerable merit, particularly when faced with the absence of any information about the data, when there are aspects of variation in the data that can be characterised effectively, using and being able to express these can often be desirable. For example, when learning representations for images of house numbers, having an explicit “digit” latent variable helps capture a meaningful axis of variation, independent of other aspects. We also often want to interpret the same data in different ways depending on context: for a given image of a person, do we care about the identity, lighting, or indeed any other facets of the scene (c.f. Figure 1). In these situations, not being able to enforce context is something of a handicap.
In this paper, we seek to combine the best of both worlds: providing the facility to describe the structural constraints under which we would like to interpret the data, while using neural nets to capture variation for aspects we cannot, or choose not to, explicitly model. By structural constraints, we refer to the (arbitrary) dependencies one would like to employ in the recognition model, particularly in regard to there being consistent interpretable semantics of what the variables in the model represent. In particular, we set up our framework in the context of variational autoencoders (VAE Kingma & Welling (2014); Rezende et al. (2014)), as a means for semi-supervised learning in deep generative models (Kingma et al., 2014). We provide an alternate formulation of the variational objective and a modified training procedure which permits us to explore a wide space of recognition networks to use as probabilistic encoders. In particular we make no mean-field assumptions for our recognition networks, allowing arbitrary hierarchical and structured-graphical-model representations, employing both continuous and discrete latent variables that can be alternately observed, or left unobserved.
2 BACKGROUND AND RELATED WORK
Variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014) simultaneously train both a probabilistic encoder and decoder for a dataset x. The central idea is that an encoding z can be considered a latent variable which allows describing a decoder as a conditional probability density pθ(x|z). This is typically a distribution with parameters defined as the output of a deterministic multi-layer neural network (itself with parameters θ) which takes z as input. Placing a weak prior over z, the corresponding probabilistic encoder can be interpreted as the posterior distribution pθ(z | x) ∝ pθ(x | z)p(z). Estimating parameters θ in this model is challenging, as is performing the posterior inference necessary to encode data. The variational Bayes approach learns an approximate encoder qφ(z | x), called an “inference network” or a “recognition network”, which aims to approximate the posterior distribution pθ(z | x). Then, rather than fitting parameters θ by maximizing the marginal likelihood pθ(x), the variational approach maximizes an evidence lower bound (ELBO) L(φ, θ;x) ≤ log pθ(x), defined with respect to both decoder θ and encoder φ parameters.
L(φ, θ;x) = Eqφ(z|x)[log pθ(x, z)− log qφ(z | x)], (1)
One line of work to embed structure into the latent space z such that it exhibits disentangled features, is through partial supervision. This is either in terms of labelled data (Sohn et al., 2015),
or curriculum-learning schemes (Kulkarni et al., 2015b) which explicitly disentangle different factors. Kingma et al. (2014) explore semi-supervised learning in the VAE setting by factoring the latent space to learn a joint classification model qφ(y | x) and recognition model qφ(z | x). This is done by separating the latent space into structured, interpretable components y and unstructured components z, analytically marginalising variables out where discrete. Sohn et al. (2015) perform fully-supervised learning in VAEs by transforming an unconditional objective into one where the data conditions both the (unstructured) latent and the (structured) labels. In contrast to Kingma et al. (2014), the learning objective is a lower bound on the conditional marginal likelihood pθ(x | y), conditioning the learned VAE on the values of the labelled data. Both of these approaches effectively require the label space y to be discrete and finite. Kulkarni et al. (2015b) attend to weaklysupervised learning with VAEs through a novel training procedure that uses data clustered into equivalence classes along different axes of variation. They then constrain different parts of the latent space to account for changes along a single axis, by training with data from a particular equivalence class. An advantage of this approach is not requiring any explicit labels on the latent space, though it does require independence assumptions on structured components, as well as carefully curated data.
An alternative approach biases towards interpretable representations by introducing structure in the prior distribution over the latent space p(z). Johnson et al. (2016) explore the combination of graphical models and VAEs using classical conjugate exponential family statistical models as structured priors over the latent space. They consider relaxation of conjugacy constraints in the likelihood model using neural network approximations, with a training scheme resembling traditional meanfield coordinate ascent algorithms. The recognition network, rather than proposing values outright, proposes parameters of a conjugate-likelihood approximation to the true non-conjugate likelihood.
From a specific-instance perspective, Eslami et al. (2016) use a recurrent neural network (RNN) coupled with a spatial transformer network (STN, Jaderberg et al. (2015)) inducing a particular state-space representation with the approximation distribution of a VAE to parse images into scene constituents. Kulkarni et al. (2015a) also explore a specific instance related to a 3D graphics engine by having a programmatic description provide structure using neural networks as surrogates for the perceptual-matching problem. Andreas et al. (2016) explore a more general formulation of structure with compositional neural network models derived from linguistic dependency parses.
3 FRAMEWORK AND FORMULATION
Our method synthesises the semi-supervised and structured-graphical-model approaches. Like Johnson et al. (2016), we incorporate graphical model structures, however rather than placing them within the generative model pθ(z,x), we incorporate them into the encoder model qφ(z | x). For many perceptual problems in domains such as vision, complex dependencies arise in the posterior due to deterministic interactions during rendering. A mean-field approximation in qφ(z | x) is a poor fit, even in situations where all the interpretable latent variables are a priori independent. This is an important reason for our choice of where we embed structure. The use of a structured, multilevel probabilistic model to define the encoder can also be interpreted as a hierarchical variational model (Ranganath et al., 2015). Interpretability is enforced by occasionally supplying labels to latent variables expected to have a interpretable meaning in the final encoded representation.
Our framework provides an embedded domainspecific language (EDSL) in Torch (Collobert et al., 2011), that can be used to specify a wide variety of graphical models in the form of a stochastic computation graph (Schulman et al., 2015). An example is shown in Figure 2. These graphical models describe the structure of latent, observable, and partially observable random variables which exist in an idealized representation space. Specifically, we assume a model structure of the form pθ(x, z,y) = pθ(x | z,y)p(z,y) where the likelihood pθ(x | z,y) of the data x is conditioned on a set of structured variables y and unstructured variables z, for which we define some appropri-
ately structured prior p(z,y). The likelihood itself is typically unstructured (e.g. a multivariate normal distribution). This model structure allows us to optimize the parameters θ learning a likelihood function constrained by the structured latents, but crucially does not require that these latents completely explain the data. The approximation to the true posterior is nominally taken to be of the form of the prior distribution qφ(z,y | x), with parameters φ but can often include additional structure and alternate factorisations as appropriate. Models with such factoring are useful for situations where interpretability is required, or informative, for some axes of variation in the data. It is also useful when we wish to interpret the same data from different contexts and when we cannot conceivable capture all the variation in the data due to its complexity, settling for particular restrictions, as is often the case with real world data.
A particular challenge here lies in choosing a manner for incorporating labelled data for some of the y into a training scheme. For example, choosing qφ(z,y | x) = qφz (z | y,x)qφy (y | x), decomposes the problem into simultaneously learning a classifier qφy (y | x) alongside the generative model parameters θ and encoder qφz (z|x,y). In the fully unsupervised setting, the contribution of a particular data point xi to the ELBO can be expressed, with minor adjustments of Equation (1), as
L ( θ, φ;xi ) = Eqφ(z,y|xi) [ log pθ ( xi | z,y ) p(z,y)
qφz (z,y | xi)
] . (2)
a Monte Carlo approximation of which samples y ∼ qφy (y | x) and z ∼ qφz (z | y,x). By contrast, in the fully supervised setting the values y are treated as observed and become fixed inputs into the computation graph, instead of being sampled from qφ. When the label y is observed along with the data, for fixed (xi,yi) pairs, the lower bound on the conditional log-marginal likelihood log pθ(x | y) is
Lx|y ( θ, φz;x i,yi ) = Eqφz (z|xi,yi) [ log pθ ( xi | z,yi ) p ( z | yi ) qφz (z | xi,yi) ] . (3)
This quantity can be optimized directly to learn model parameters θ and φz simultaneously via SGD. However, it does not contain the encoder parameters φy . This difficulty was also encountered in a related context by Kingma et al. (2014). Their solution was to augment the loss function by including an explicit additional term for learning a classifier directly on the supervised points.
An alternative approach involves extending the model using an auxiliary variable ỹ. Defining p(ỹ,y, z | x) = p(ỹ | y)p(x,y, z) and q(ỹ,y, z | x) = p(ỹ | y)q(y, z | x), with likelihood p(ỹ | y) = δỹ(y), we obtain a model for which marginalization over ỹ reproduces the ELBO in Equation (2), and treating ỹ as observed gives the supervised objective
L ( θ, φ;xi )∣∣ ỹ=yi = Eqφy [ δyi(y)Eqφz [ log pθ ( xi | z,y ) p(z,y)
qφy (y | xi)qφz (z | y,xi)
]]
= qφy ( yi | xi ) Eqφz [ log pθ ( xi | z,yi ) p ( z,yi ) qφy (y i | xi)qφz (z | yi,xi) ]
= qφy ( yi | xi )[ Lx|y ( θ, φz ;x i,yi ) + log p ( yi ) − log qφy ( yi | xi )] . (4)
This formulation enables a range of capabilities for semi-supervised learning in deep generative models. To begin with, it extends the ability to partially-supervise latent variables to those that have continuous support. This effectively learns a regressor instead of a classifier in the same formulation. Next, it automatically balances the trade-off between learning a classifier/regressor and learning the parameters of the generative model and the remainder of the recognition network. This is due to the fact that the classifier qφy (y | x) is always present and learned, and is contrast to the hyperparameter-driven approach in Kingma et al. (2014). Finally, it allows for easy automatic implementation of a wide variety of models, separating out the labelled and unlabelled variables, to derive a unified objective over both the supervised and unsupervised cases. When unsupervised, the value of the label yi is sampled from qφy (y | x) and scored in that distribution, and when supervised, it is set to the given value, and scored in the same distribution. This is in the same spirit as a
number of approaches such as Automatic Differentiation (AD) and Probabilistic Program inference, where the choice of representation enables ease of automation for a great variety of different cases.
Supervision rate. While learning with this objective, we observe data in batches that are either wholly supervised, or wholly unsupervised. This typically obviates the need to construct complicated estimators for the partially observed cases, while also helping reduce variance in general over the learning and gradient computation (details of which are provided in the Appendix). Doing so also presents a choice relating to how often we observe labelled data in a complete sweep through the dataset, referred to as the supervision rate r. Practically, the rate represents a clear trade-off in learning the generative and recognition-network parameters under interpretability constraints. If the rate is too low, the supervision can be insufficient to help with disentangling representation in the recognition network, and if too high, the generative model can overfit to just the (few) supervised data points. The rate also has a natural relation to the variance of the objective function and its gradients. As can be seen from Equation (4), an evaluation of the objective for a given yi involves the unsupervised estimation of the conditional ELBO Lx|y . The rate implicitly affects the number of such estimations for any given yi and thus the variance of the objective with respect to that label yi. The same argument applies for the gradients of the objective.
Plug-in estimation for discrete variables. In targeting a general class of models, another particular difficulty is the ubiquity of discrete latent variables. To obtain a differentiable objective, one can either marginalize over discrete variables directly (as done by Kingma et al. (2014) and in the STAN probabilistic programming system (Stan Development Team, 2013)), which doesn’t scale over numbers of variables, or use a REINFORCE-style estimator (Williams, 1992; Mnih & Gregor, 2014), which tends to have high variance. A third approach, related to Bengio et al. (2013), is to represent discrete latent variables defined on a finite domain using a one-hot encoding, then relaxing them to a continuous probability simplex when used as an input to a recognition network. For example, when y is a one-hot encoding of a discrete value used in a recognition network which factors as qφ(y | x)qφ(z | y,x), then qφ(y | x) is itself a discrete distribution with a probability vector ρ = gφ(x) for some deterministic function gφ. The value y is itself an input to a second function hφ(x,y) producing the parameters for qφ(z | y,x). Instead of evaluating hφ(x,y) at a sampled value y (or enumerating over the entire domain), we simply evaluate it at the single point ρ, noting that ρ = Eqφ(y|x)[y]. This may seem a crude approximation, replacing integration with a single evaluation, claiming Eqφ(y|x)[hφ(x,y)] ≈ hφ(x,Eqφ(y|x)[y]), which is not true in general for hφ(·). However, if ρ is actually a one-hot encoding, i.e., when Eqφ(y|x)[y] has a single non-zero value, they are in fact equal. For our experiments we employ this plug-in estimator where applicable, although our framwork can express any of the above methods.
4 EXPERIMENTS
We evaluate our framework on along a number of different axes, pertaining to its ability to (i) learn disentangled representation from a little supervision, (ii) demonstrate capability at a relevant classification/regression task, (iii) successfully also learn the generative model, and (iv) admit the use of latent spaces of varying dimensionality Note that we do not set out to build the best possible classifier in these tasks. Instead, the classification task is a means to the end of demonstrating that the learned representation is indeed disentangled, often with minimal supervision. Also, details of neural network architectures, graphical models for the recognition networks, dataset characteristics, and hyper-parameter settings are provided in the Appendix.
4.1 MNIST AND SVHN
To begin with, we explore the facets of our model in the standard MNIST and Google Street-View House Numbers (SVHN) datasets. We use this example to highlight how the provision of even the slightest structure, coupled with minimal supervision, in often sufficient to induce the emergence of disentangled representations in the recognition network. Figure 3 shows the structure of the generative and recognition models for this experiment.
Figure 4(a) and (c) show the effect of first transforming a given input (leftmost column) into the disentangled latent space, and with the style latent variable fixed, manipulating the digit through the generative model to produce appropriately modified reconstructions. These were both derived with full supervision over a 50 and 100 dimensional Gaussian latent space for the styles, respectively. Figure 4(b) shows the transformation for a fixed digit, when the style latent is varied. This was derived with a simple 2D Gaussian latent space for the style. The last part, Figure 4(d) shows the ability of the network to begin disentangling the latent space with just 100 labelled samples per digit (training dataset size is 73000 points). Separation between style and class is clearly evident even with such little supervision.
We compute the classification accuracy of the label-prediction task with this model for both datasets, and the results are reported in the bottom of Figure 5. The results are compared to those reported in Kingma et al. (2014). For the MNIST dataset, we compare against model M2 as we run directly on the data, without performing a preliminary feature-extraction step. For the SVHN dataset, we compare against model M1+M2 even though we run directly on the data, using a CNN to simultaneously learn to extract features. Confidence estimates for both were computed off of 10 runs. We note that we fare comparably with these models, and in particular, when employing a CNN for feature extraction for the SVHN dataset, comfortably exceed them.
Figure 5 shows the effect of the supervision rate r on the error rate. As evident from the graph, the rate has a strong affect on how quickly one learns an effective classifier. This indicates that when labels are sparse or hard to come by, a training regime that runs largely unsupervised, even only occasionally looking at the supervised data, still learns to disentangle the latent-space representations.
4.2 INTRINSIC FACES
We next move to a harder problem involving a generative model of faces, attempting to highlight how the introduction of stronger dependency structures in the recognition model helps disentangle latents, particularly when the generative model assumes conditional independence between the latents. Here, we use the “Yale B” dataset as processed by Jampani et al. (2015) to train the models shown in Figure 6. The primary tasks we are interested in here are (i) the ability to manipulate the inferred latents to evaluate if they qualitatively achieve semantically meaningful disentangled representations, (ii) the classification of person identity, and (iii) the regression for lighting direction.
Figure 7 presents both qualitative and quantitative evaluation of the framework to jointly learn both the structured recognition model, and the generative model parameters. A particular point of note is that we explicitly encode “identity” as a categorical random variable since we have knowledge about the domain and the relevant axis to explore. Since we also learn the generative model, which in the domain of the actual dataset is simply the expression (n.l)× r+ , we can afford to weakly specify the structure allowing for some neural-network component to take up the requisite slack in order to reconstruct the input. This allows us to directly address the task of predicting identity, instead of approaching it through surrogate evaluation methods (e.g. nearestneighbour classification based on inferred reflectance).
While this formulation allows us to to perform the identity classification task, the fact that our recognition model never supervises the reflectance means that the variable can typically absorb some of the representational power of other, semi-supervised nodes. This is particularly the case when dealing with high-dimensional latent spaces as for reflectance and shading.
4.3 MULTI-MNIST
Finally, we run an experiment to test the ability of our framework to handle models that induce latent representations of variable dimension. We extend the simple model from the MNIST experiment by composing it with a stochastic sequence generator, to test its ability to count the number of digits in a given input image, given its ability to encode and reconstruct the digits in isolation. The graphical models employed are depicted in Figure 8.
We observe that we are indeed able to reliable learn to count, at least within the limits of upto 3 digits in the multi-mnist dataset. The dataset was generated directly from the MNIST dataset by manipulating the scale and positioning of the standard digits into a combined canvas, evenly balanced across the counts and digits. The results across different supervised set sizes and supervision rates are shown in the table in Figure 8.
5 DISCUSSION AND CONCLUSION
In this paper, we introduce a general framework for semi-supervised learning in the VAE setting that allows incorporation of graphical models to specify a wide variety of structural constraints on the recognition network. We demonstrate its flexibility by applying it to a variety of different tasks in the visual domain, and evaluate its efficacy at learning disentangled representations in a semi-supervised manner, showing strong performance.
This framework ensures that the recognition network learns to make predictions in an interpretable and disentangled space, constrained by the structure provided by the graphical model. The structured form of the recognition network also is typically a better fit for vision models, as it helps better capture complexities in the likelihood (usually the renderer). Given that we encode graphical models in the recognition network, and Johnson et al. (2016) encode it in the generative model in concert with VAEs, a natural extension would be the exploration of the ability to learn effectively when specifying structure in both by means of graphical models. This is a direction of future work we are interested in, particularly in context of semi-supervised learning.
The framework is implemented as a Torch library (Collobert et al., 2011), enabling the construction of stochastic computation graphs which encode the requisite structure and computation. This provides another direction to explore in the future – the extension of the stochastic computation graph framework to probabilistic programming (Goodman et al., 2008; Wingate et al., 2011; Wood et al., 2014). Probabilistic programs go beyond the presented framework to include stochastic inference and the ability to specify arbitrary models of computation. The combination of such frameworks with neural networks has recently been studied in Ritchie et al. (2016); Le et al. (2016), and indicates a promising avenue for further exploration.
APPENDIX
FORMULATION
Gradients of the Variational Objective: We consider the gradients of the form in Equation (4) with respect to θ, φz , and φy . In particular, note that for both θ and φz the gradient is the same as the gradient with respect to the conditional ELBO Lx|y , up to a per-datapoint scaling factor q ( yi | xi ) . For continuous latent variables, as well as for many discrete random variables, the expectations over z can be reparameterized into a form where the gradients can be approximated with a single sampled value. Evaluating Equation (4) at this point yields estimators for the ELBO L̂ and conditional ELBO L̂x|y , as well as corresponding single-sample gradient estimates ∇̂L and ∇̂Lx|y for each set of parameters. Gradient estimates for θ and φz are proportional to the gradients of the conditional ELBO as
∇̂θ L(θ, φ;xi) ∣∣ y=yi = qφy ( yi | xi ) ∇̂θ Lx|y ,
∇̂φz L(θ, φ;xi) ∣∣ y=yi = qφy ( yi | xi ) ∇̂φz Lx|y ,
while the gradient with respect to the “classifier” parameters φy takes a different form. Applying the product rule to Equation (4) we have
∇̂φy L(θ, φ;xi) ∣∣ y=yi
= [ L̂x|y + log p ( yi ) − log qφy ( yi | xi )] ∇φy qφy ( yi | xi ) − qφy ( yi | xi ) ∇φy log qφy ( yi | xi ) = [ L̂x|y + log p ( yi ) − log qφy ( yi | xi ) − 1 ] ∇φy qφy ( yi | xi
) = qφy ( yi | xi )[ L̂ − 1 ] ∇φy log qφy ( yi | xi ) .
MODEL AND NETWORK PARAMETERS
We note for that all the experiments, save the one involving Street-View House Numbers (SVHN), were run using a 2-3 layer MLP with 512 nodes and using a Bernoulli loss function. For SVHN, we additionally employed a two stage convolutional and a 2 stage deconvolutional network to effectively extract features for the standard MLP model for the recognition network and the generative model respectively; training the entire network end-to-end. For learning, we used AdaM (Kingma & Ba, 2014) with a learning rate of 0.001 (0.0003 for SVHN) and momentum-correction terms set to their default values. As for the minibatch sizes, they varied from 80-500 depending on the dataset being used and its size.
MODELS
The syntax of our computation graph construction is such that the first call instantiates the computation, and the second instantiates the node and its connections. For specified random variables, the first set of parameters defines the prior and second set the parameters for the proposal distributions. In all our models, we extract the common, feature-extraction portions of the recognition model qφ into a simple pre-encoder. Parameters and structure for this are specified above.
The class-conditional model for MNIST and SVHN.
local ndim = 50
local program = {}
function program:getNetwork() local input = nn.Identity()() -- required to make nngraph play nice -- the actual program local d = pp.DiscreteR({torch.Tensor(1,10):fill(1/10)})({input}) local mu = nn.Sequential() :add(nn.JoinTable(2)) :add(nn.FluidLinear(ndim)) :add(nn.SoftPlus())
local sig = nn.Sequential() :add(nn.JoinTable(2)) :add(nn.FluidLinear(ndim)) :add(nn.SoftPlus()) local n = pp.Gaussian({ torch.zeros(1,ndim), torch.zeros(1,ndim) })({mu({d, input}), sig({d, input})}) -- end program nngraph.annotateNodes() -- necessary to annotate nodes with local varnames return pp.gModule({input}, {d, n})
end
return program
The model used for the faces dataset.
local program = {}
function program:getNetwork() local input = nn.Identity()() -- required to make nngraph play nice -- the actual program local id = pp.DiscreteR({torch.Tensor(1,38):fill(1/38)})({input}) local light = pp.Gaussian({
torch.zeros(1,3), torch.zeros(1,3)
})({pp.r(input), pp.r(input)}) local factorQ = nn.Sequential()
:add(nn.JoinTable(2)) :add(nn.FluidLinear(20)) :add(nn.SoftPlus())
local shading = pp.Gaussian({ torch.zeros(1,20), torch.zeros(1,20) })({pp.r(factorQ({id,light})), pp.r(factorQ({id,light}))}) local reflectance = pp.Gaussian({
torch.zeros(1,20), torch.zeros(1,20)
})({pp.r(input), pp.r(input)}) -- end program nngraph.annotateNodes() -- necessary to annotate nodes with local varnames return pp.gModule({input}, {shading, reflectance})
end
return program
The model used for the multi-mnist dataset.
local program = {}
local function mnist() local input = nn.Identity()() -- required to make nngraph play nice local d = pp.DiscreteR({torch.Tensor(1,10):fill(0.1)})({input}) local n = pp.Gaussian({
torch.zeros(1,50), torch.zeros(1,50)
})({pp.r(input), pp.r(input)}) -- end program nngraph.annotateNodes() -- necessary to annotate nodes with local varnames return pp.gModule({input}, {d, n})
end
function program:getNetwork() local input = nn.Identity()() -- required to make nngraph play nice -- the actual program local c = pp.Discrete(({torch.Tensor(1,5):fill(0.2)})({input})) -- needswork: have to handle number of inputs and inter-repeat-state local ds = pp.Repeat(mnist())({input, c}) -- end program nngraph.annotateNodes() -- necessary to annotate nodes with local varnames return pp.gModule({input}, {ds}) end
return program

This paper introduces a variant of the semi-supervised variational auto-encoder (VAE) framework. The authors present a way of introducing structure (observed variables) inside the recognition network.
I find that the presentation of the inference with auxiliary variables could be avoided, as it actually makes the presentation unnecessarily complicated. Specifically, the expressions with auxiliary variables are helpful for devising a unified implementation, but modeling-wise one can get the same model without these auxiliary variables and recover a minimal extension of VAE where part of the generating space is actually observed. The observed variables mean that the posterior needs to also condition on those, so as to incorporate the information they convey. The way this is done in this paper is actually not very different from Kingma et al. 2014, and I am surprised that the experiments show a large deviation in these two methods' results. Given the similarity of the models, it'd be useful if the authors could give a possible explanation on the superiority of their method compared to Kingma et al. 2014. By the way, I was wondering if the experimental setup is the same as in Kingma et al. 2014 for the results of Fig. 5 (bottom) - the authors mention that they use CNNs for feature extraction but from the paper it's not clear if Kingma et al. do the same.
On a related note, I was wondering the same for the comparison with Jampani et al. 2015. In particular, is that model also using the same rate of supervision for a fair comparison?
The experiment in section 4.3 is interesting and demonstrates a useful property of the approach.
The discussion of the supervision rate (and the pre-review answer) is helpful in giving some insight about what is a successful training protocol to use in semi-supervised learning.
Overall, the paper is interesting but the title and introduction made me expect something more from it. From the title I expected a method for interpreting general deep generative models, instead the described approach was about a semi-supervised variant of VAE - naturally including labelled examples disentangles the latent space, but this is a general property of any semi-supervised probabilistic model and not unique to the approach described here. Moreover, from the intro I expected to see a more general approximation scheme for the variational posterior (similar to Ranganath et al. 2015  which trully allows very flexible distributions), however this is not the case here.
Given the above, the contributions of this paper are in defining a slight variant of the semi-supervised VAE, and (perhaps more importantly) formulating it in a way that is amendable to easier automation in terms of software. But methodologically there is not much contribution to the current literature. The authors mention that they plan to extend the framework in the probabilistic programming setting. It seems indeed that this would be a very promising and useful extension.
Minor note: three of Kingma's papers are all cited in the main text as Kingma et al. 2014, causing confusion. I suggest using Kingma et al. 2014a etc.


EPITOMIC VARIATIONAL AUTOENCODER
In this paper, we propose epitomic variational autoencoder (eVAE), a probabilistic generative model of high dimensional data. eVAE is composed of a number of sparse variational autoencoders called ‘epitome’ such that each epitome partially shares its encoder-decoder architecture with other epitomes in the composition. We show that the proposed model greatly overcomes the common problem in variational autoencoders (VAE) of model over-pruning. We substantiate that eVAE is efficient in using its model capacity and generalizes better than VAE, by presenting qualitative and quantitative results on MNIST and TFD datasets.
1 INTRODUCTION
Unsupervised learning holds the promise of learning the inherent structure in data so as to enable many future tasks including generation, prediction and visualization. Generative modeling is an approach to unsupervised learning wherein an explicit stochastic generative model of data is defined, such that independent draws from this model are likely to produce the original data distribution, while the learned latent structure itself is useful in prediction, classification and visualization tasks.
The recently proposed variational autoencoder (VAE) (Kingma & Welling, 2014) is an example of one such generative model. VAE pairs a top down generative model with a bottom up recognition network for amortized probabilistic inference. Both networks are jointly trained to maximize a variational lower bound on the data likelihood. A number of recent works use VAE as a modeling framework, including iterative conditional generation of images (Gregor et al., 2015) and conditional future frame prediction (Xue et al., 2016).
A commonly known problem with the VAE lower bound is that it is known to self-prune or under utilize the model’s capacity (Mackay, 2001). This can lead to poor generalization. A common approach to alleviate this problem is to resort to optimization schedules and regularization techniques (Bowman et al., 2015; Kaae Sonderby et al., 2016) that trade-off two competing terms, latent cost and data reconstruction, in the bound. Fig. 1 provides a quick insight into this problem of over-pruning and how commonly used regularization techniques may not be sufficient. Detailed discussion is provided in § 2.1. In this paper, we take a model-based approach to directly address this problem. We present an extension of variational autoencoders called epitomic variational autoencoder (Epitomic VAE, or eVAE, for short) that automatically learns to utilize its model capacity more effectively, leading to better generalization. Consider the task of learning a D-dimensional representation for the examples in a given dataset. The motivation for our model stems from the hypothesis that a single example in the dataset can be sufficiently embedded in a smaller K-dimensional (K D) subspace of D. However, different data points may need different subspaces, hence the need for D. Sparse coding methods also exploit a similar hypothesis. Epitomic VAE exploits sparsity using an additional categorical latent variable in the encoder-decoder architecture of the VAE. Each value of the variable activates only a contiguous subset of latent stochastic variables to generate an observation. This ∗Work done during an internship at Facebook AI Research.
enables learning multiple shared subspaces such that each subspace specializes, and also increases the use of model capacity (Fig. 4), enabling better representation. The choice of the name Epitomic VAE comes from the fact that multiple miniature models with shared parameters are trained simultaneously.
The rest of the paper is organized as follows. We first describe variational autoencoders and mathematically show the model pruning effect in § 2. We then present our epitomic VAE model in § 3 that overcomes these shortcomings. Experiments showing qualitative and quantitative results are presented in § 4. We finally provide more general context of our work in the related work in § 5, and conclude with discussions.
2 VARIATIONAL AUTOENCODERS
The generative model (decoder) of a VAE consists of first generating a D-dimensional stochastic variable z drawn from a standard multivariate Gaussian
p(z) = N (z; 0; I) (1) and then generating the N-dimensional observation x from a parametric family of distributions such as a Gaussian pθ(x|z) = N (x; f1(z); exp(f2(z))) (2) where f1 and f2 define non-linear deterministic transformations of z modeled using a neural network. The parameters θ of the model are the weights and biases of the neural network that encodes the functions f1 and f2.
Given a dataset X of T i.i.d samples, the model is learned such that it maximizes the likelihood of the parameters to have generated the data, p(X|θ). This maximization requires marginalizing the unobserved z. However, computing p(z|x) is intractable due to dependencies induced between the zi when conditioned on x.
Variational autoencoders, as the name suggests, use variational inference to approximate the exact posterior with a surrogate parameterized distribution. However, instead of having separate parameters for the posterior distribution of each observation, VAE amortizes the cost by learning a neural network with parameters φ that outputs the posterior distribution of the form qφ(z|x) = ∏ d q(zi|x). This results in the lower bound given by
log pθ(X) = T∑ t=1 log ∫ z pθ(x (t), z) (3)
≥ T∑ t=1 Eqφ(z|x(t))[log p(x (t)|z)]−KL ( qφ(z|x(t)) ‖ p(z) ) (4)
VAE is trained with standard backpropagation using minibatch gradient descent to minimize the negative of the lowerbound
Cvae = − T∑ t=1 Eqφ(z|x(t))[log p(x (t)|z)] + T∑ t=1 D∑ i=1 KL ( qφ(zi|x(t)) ‖ p(zi) ) (5)
2.1 AUTOMATIC MODEL OVER-PRUNING IN VAE
Cvae introduces a trade-off between data reconstruction (first term) and satisfying the independence assumption of p(z) (second term, KL).
Of particular interest is the KL term. Since the KL term is the sum of independent contributions from each dimension d of D, it provides unduly freedom for the model in how it minimizes this term. In particular, the model needs to only ensure that the overall KL term is minimized, on average, and not per component wise. The easiest way for the model to do this is to have a large number of components that satisfies the KL term effectively, by turning off the units so that the posterior for those units becomes the same as the prior1. This effect is quite pronounced in the early iterations of
1Since log variance is modeled using the neural network, turning it off will lead to a variance of 1.
training: the model for log p(x|z) is quite impoverished and hence the easiest way to improve the bound is by turning off the KL terms. However, once the units have become inactive, it is almost impossible for them to resurrect, and hence the full capacity of the model is not utilized.
A quantity that is useful in understanding this effect, is the activity level of a unit. Following Burda et al. (2015), we define a unit to be used, or “active”, if Au = Covx(Eu∼q(u|x)[u]) > 0.02.
A commonly used approach to overcome this problem is to use a trade-off between the two terms using parameter λ so that the cost is
C = −Eqφ(z|x)[log p(x|z)] + λ D∑ i=1 KL ( qφ(zi|x) ‖ p(zi) ) (6)
Fig. 1 shows the effect of λ on unit activity and generation, with λ = 1 being the correct objective to optimize. While tuning down λ increases the number of active units, samples generated from the model are still poor. Fig. 2 shows generation using all units, active units only, and dead units only, for λ = 1. The model spends its capacity in ensuring that reconstruction of the training set is optimized (reconstruction visualizations are shown in § 8.1), at the cost of generalization. This has led to more sophisticated schemes such as using an annealed optimization schedule for λ (Bowman et al., 2015; Kaae Sonderby et al., 2016) or enforcing minimum KL contribution from subsets of the latent units (Kingma et al., 2016).
In this paper, we present a model based approach called “epitomic variational autoencoder” to address the problem of over pruning.
3 MODEL
We propose epitomic variational autoencoder (eVAE) to overcome the shortcomings of VAE by enabling more efficient use of model capacity to gain better generalization. We base this on the observation that while we may need a D-dimensional representation to accurately represent every example in a dataset, each individual example can be represented with a smaller K-dimensional subspace. As an example, consider MNIST with its variability in terms of digits, strokes and thick-
ness of ink, to name a few. While the overall D is large, it is likely that only a few K dimensions of D are needed to capture the variability in strokes of some digits (see Fig. 3).
Epitomic VAE can be viewed as a variational autoencoder with latent stochastic dimension D that is composed of a number of sparse variational autoencoders called epitomes, such that each epitome partially shares its encoder-decoder architecture with other epitomes in the composition. In this paper, we assume simple structured sparsity for each epitome: in particular, only K contiguous dimensions of D are active2.
The generative process can be described as follows: A D-dimensional stochastic variable z is drawn from a standard multivariate Gaussian p(z) = N (z; 0; I). In tandem, an epitome is implicitly chosen through an epitome selector variable y, which has a uniform prior over possible epitomes. The N -dimensional observation x is then drawn from a Gaussian distribution:
pθ(x|y, z) = N (x; f1(my z), exp(f2(my z))) (7) my enforces the epitome constraint: it is also aD-dimensional vector that is zero everywhere except in the active dimensions of the epitome. is element-wise multiplication between the two operands. Thus, my masks the dimensions of z other than those dictated by the choice of y. Fig. 3 illustrates this for an 8-d z with epitome size K = 2, so that there are four possible epitomes (the model also allows for overlapping epitomes, but this is not shown for illustration purposes). Epitome structure is defined using size K and stride s, where s = 1 corresponds to full overlap in D dimensions3. Our model generalizes the VAE and collapses to a VAE when D = K = s.
f1( ) and f2( ) define non-linear deterministic transformations of modeled using neural networks. Note that the model does not snip off the K dimensions corresponding to an epitome, but instead deactivates the D-K dimensions that are not part of the chosen epitome. While the same deterministic functions f1 and f2 are used for any choice of epitome, the functions can still specialize due to the
2The model also allows for incorporating other forms of structured sparsity. 3The strided epitome structure allows for learning O(D) specialized subspaces, that when sampled during generation can each produce good samples. In contrast, if only a simple sparsity prior is introduced over arbitrary subsets (e.g. with Bernoulli latent units to specify if a unit is active for a particular example), it can lead to poor generation results, which we confirmed empirically but do not report. The reason for this is as follows: due to an exponential number of potential combinations of latent units, sampling a subset from the prior during generation cannot be straightforwardly guaranteed to be a good configuration for a subconcept in the data, and often leads to uninterpretable samples.
sparsity of their inputs. Neighboring epitomes will have more overlap than non-overlapping ones, which manifests itself in the representation space; an intrinsic ordering in the variability is learned.
3.1 OVERCOMING OVER-PRUNING
Following Kingma & Welling (2014), we use a recognition network q(z, y|x) for approximate posterior inference, with the functional form
q(z, y|x) = q(y|x)q(z|y,x) (8) = q(y|x)N (z;my µ, exp (my φ)) (9)
where µ = h1(x) and φ = h2(x) are neural networks that map x to D dimensional space.
We use a similar masking operation to deactivate units, as decided by the epitome y. Unlike the generative model (eq. 7), the masking operation defined by y operates directly on outputs of the recognition network that characterizes the parameters of q(z|y,x). As in VAE, we can derive the lower bound on the log probability of a dataset, and hence the cost function (negative bound) is
Cevae = − T∑ t=1 Eq(z,y|x(t))[log p(x (t)|y, z)]
− T∑ t=1 KL [ qφ(y|x(t)) ‖ pθ(y) ] − T∑ t=1 ∑ y qφ(y|x(t))KL [ qφ(z|y,x(t)) ‖ pθ(z) ] (10)
The epitomic VAE departs from the VAE in how the contribution from the KL term is constrained. Let us consider the third term in eq. 10, and substituting in eq. 9:
T∑ t=1 ∑ y qφ(y|x(t))KL [ qφ(z|y,x(t)) ‖ pθ(z) ] (11)
= T∑ t=1 ∑ y qφ(y|x(t))KL [ N (z;my µ(t), exp (my φ(t))) ‖ N (z;0, I) ] (12)
= T∑ t=1 ∑ y qφ(y|x(t)) D∑ d=1 1[md,y = 1]KL [ N (zd;µ(t)d , exp(φ (t) d )) ‖ N (0, 1) ] (13)
where 1[?] is an indicator variable that evaluates to 1 if only if its operand ? is true.
For a training example x(t) and for a fixed y (and hence the corresponding epitome), the number of KL terms that will contribute to the bound is exactly K. The dimensions of z that are not part of the corresponding epitome will have zero KL because their posterior parameters are masked to have unit Gaussian, the same as the prior. By design, this ensures that only the K dimensions that explain x(t) contribute to Cevae.
This is quite in contrast to how VAE optimizes Cvae (§. 2.1). For Cvae to have a small contribution from the KL term of a particular zd, it has to infer that unit to have zero mean and unit variance for many examples in the training set. In practice, this results in VAE completely deactivating units, and leading to many dead units. EpitomicVAE chooses the epitome based on x(t) and ensures that the dimensions that are not useful in explaining x(t) are ignored in Cevae. This means that the unit is still active, but by design, only a fraction of examples in the training set contributes a possible non-zero value to zd’s KL term in Cevae. This added flexibility gives the model the freedom to use more total units without deactivating them, while optimizing the bound. With these characteristics, during training, the data points will naturally group themselves to different epitomes, leading to a more balanced use of z.
In Fig. 4 we compare the activity levels of VAE, dropout VAE and our model. We see that compared with VAE, our model is able to better use the model capacity. In the same figure, we also compare with adding dropout to the latent variable z of the VAE (Dropout VAE). While this increases the number of active units, it generalizes poorly as it uses the dropout layers to merely replicate representation, in contrast to eVAE. See Fig. 5 along with the explanation in § 4.1 where we compare generation results for all three models.
3.2 TRAINING
The generative model and the recognition network are trained simultaneously, by minimizing Cevae in eq. 10.
For the stochastic continuous variable z, we use the reparameterization trick as in VAE. The trick involves reparametrizing the recognition distribution in terms of auxiliary variables with fixed distributions. This allows efficient sampling from the posterior distribution as they are deterministic functions of the inputs and auxiliary variables.
For the discrete variable y, we cannot use the reparameterization trick. We therefore approximate q(y|x) by a point estimate y∗ so that q(y|x) = δ(y = y∗), where δ evaluates to 1 only if y = y∗ and the best y∗ = argmin Cevae. We also explored modeling q(y|x) = Cat(h(x)) as a discrete distribution with h being a neural network. In this case, the backward pass requires either using REINFORCE or passing through gradients for the categorical sampler. In our experiments, we found that these approaches did not work well, especially when the number of possible values of y becomes large. We leave this as future work to explore.
The recognition network first computes µ and φ. It is then combined with the optimal y∗ for each example, to arrive at the final posterior. The model is trained using a simple algorithm outlined in Algo. 1. Backpropagation with minibatch updates is used, with each minibatch constructed to be balanced with respect to epitome assignment.
Algorithm 1 Learning Epitomic VAE 1: θ, φ←Initialize parameters 2: for until convergence of parameters (θ, φ) do 3: Assign each x to its best y∗ = argmin Cevae 4: Randomize and then partition data into minibatches with each minibatch having proportion-
ate number of examples ∀ y 5: for k ∈ numbatches do 6: Update model parameters using kth minibatch consisting of x, y pairs 7: end for 8: end for
4 EXPERIMENTS
We present experimental results on two datasets, MNIST (LeCun et al., 1998) and Toronto Faces Database (TFD) (Susskind et al., 2010). We show generation results that illustrate eVAE’s ability to better utilize model capacity for modeling data variability, and then evaluate the effect of epitome choice and model complexity. Finally we present quantitative comparison with other models and qualitative samples from eVAE. We emphasize that in all experiments, we keep the weight of the KL term λ = 1 to evaluate performance under optimizing the true derived lower bound, without introducing an additional hyperparameter to tune.
We use standard splits for both MNIST and TFD. In our experiments, the encoder and decoder are fullyconnected networks, and we show results for different depths and number of units of per layer. ReLU nonlinearities are used, and models are trained using the Adam update rule (Kingma & Ba, 2014) for 200 epochs (MNIST) and 250 epochs (TFD), with base learning rate 0.001.
4.1 OVERCOMING OVER-PRUNING.
We first qualitatively illustrate the ability of eVAE to overcome over-pruning and utilize latent capacity to model greater variability in data. Fig. 5 compares generation results for VAE, Dropout VAE, and eVAE for different dimensionsD of latent variable z. WithD = 2, VAE generates realistic digits but suffers from lack of diversity. When D is increased to 5, the generation exhibits some greater variability but also begins to degrade in quality. As D is further increased to 10 and 20, the degradation continues. As explained in Sec. 2.1, this is due to VAE’s propensity to use only a portion of its latent units for modeling the training data and the rest to minimize the KL term. The under-utilization of model capacity means that VAE learns to model well only regions of the posterior manifold near training samples, instead of generalizing to model the space of possible generations. The effect of this is good reconstruction (examples are shown in Fig. 9) but poor generation samples.
Adding dropout to the latent variable z of the VAE (row 2 of Fig. 5) encourages increased usage of model capacity, as shown in Fig. 4 and the discussion in Sec. 2. However, due to the stochastic nature of dropout, the model is forced to use the additional capacity to encode redundancy in the representation. It therefore does not achieve the desired effect of encoding additional data variability, and furthermore leads to blurred samples due to the redundant encoding. Epitomic VAE addresses the crux of the problem by learning multiple specialized subspaces. Since the effective dimension of any example is still small, eVAE is able to model each subspace well, while encoding variability through multiple possibly shared subspaces. This enables the model to overcome over-pruning from which VAE suffered. Fig. 5 shows that as the dimension D of z is increased
while maintaining epitomes of size K = 2, eVAE is able to model greater variability in the data. Highlighted digits in the 20-d eVAE show multiple styles such as crossed versus un-crossed 7, and pointed, round, thick, and thin 4s. Additional visualization of the variability in the learned 2-d manifolds are shown in Fig. 3. In contrast, the 2-d VAE generates similar-looking digits, and is unable to increase variability and maintain sample quality as the latent dimension is increased.
4.2 CHOICE OF EPITOME SIZE
We next investigate how the choice of epitome size, K, affects generation performance. We evaluate the generative models quantitatively through their samples by measuring the log-density with a Parzen window estimator Rifai et al. (2012). Fig. 6 shows the Parzen log-density for different choices of epitome size on MNIST, with encoder and decoder consisting of a single deterministic layer of 500 units. Epitomes are nonoverlapping, and the results are grouped by total dimension D of the latent variable z. For comparison, we also show the log-density for VAE models with the same dimension D, and for mixture VAE (mVAE), an ablative version of eVAE where parameters are not shared. mVAE can also be seen as a mixture of independent VAEs trained in the same manner as eVAE. The number of deterministic units in each mVAE component is computed so that the total number of parameters is comparable to eVAE.
As we increase D, the performance of VAE drops significantly, due to over-pruning. In fact, the number of active units for VAE are 8, 22 and 24 respectively, forD values of 8, 24 and 48. In contrast, eVAE performance increases as we increase D, with an epitome size K that is significantly smaller than D. Table 1 provides more comparisons. This confirms the advantage of using eVAE to avoid overpruning and effectively capture data distribution.
eVAE also performs comparably or better than mVAE at all epitome sizes. Intuitively, the advantage of parameter sharing in eVAE is that each epitome can also benefit from general features learned across the training set.
4.3 INCREASING COMPLEXITY OF ENCODER AND DECODER
Here, we would like to understand the role of encoder and decoder architectures on over pruning, and the generative performance. We control model complexity through number of layers L of deterministic hidden units, and number of hidden units H in each deterministic layer.
Table 1 shows the Parzen log-densities of VAE, mVAE and eVAE models trained on MNIST and TFD with different latent dimension D. For mVAE and eVAE models on MNIST, the maximum over epitomes of size K = 3 and K = 4 is used, and on TFD epitomes of size K = 5 are used. All epitomes are non-overlapping.
We observe that for VAE, increasing the number of hidden units H (e.g. from 500 to 1000) for a fixed network depth L has a negligible effect on the number of active units and performance. On the other hand, as the depth of the encoder and decoder L is increased, the number of active units in VAE decreases though performance is still able to improve. This illustrates that increase in the complexity of the interactions through use of multiple
layers counteract the perils of the over-pruning. However, this comes with the cost of substantial increase in the number of model parameters to be learned.
In contrast, for any given model configuration, eVAE is able to avoid the over-pruning effect in the number of active units and outperform VAE. While both VAE and eVAE approach what appears to be a ceiling in generative performance with large models for MNIST, the difference between VAE and eVAE is significant for all TFD models.
Table 1 also shows results for mVAE, the ablative version of eVAE where parameters are not shared. The number of deterministic units per layer in each mVAE component is computed so that the total number of parameters is comparable to eVAE. While mVAE and eVAE perform comparably on MNIST especially with larger models (reaching a limit in performance that VAE also nears), eVAE demonstrates an advantage on smaller models and when the data is more complex (TFD). These settings are in line with the intuition that parameter sharing is helpful in more challenging settings when each epitome can also benefit from general features learned across the training set.
H = 500 H = 1000 L = 1 L = 2 L = 3 L = 1 L = 2 L = 3
MNIST
D = 8 VAE 283(8) 292(8) 325(8) 283(8) 290(8) 322(6) mVAE 300(8) 328(8) 337(8) 309(8) 333(8) 335(8) eVAE 300(8) 330(8) 337(8) 312(8) 331(8) 334(8)
D = 24 VAE 213(22) 273(11) 305(8) 219(24) 270(12) 311(7) mVAE 309(24) 330(24) 336(24) 313(24) 333(24) 338(24) eVAE 311(24) 331(24) 336(24) 317(24) 332(24) 336(24)
D = 48 VAE 213(24) 267(13) 308(8) 224(24) 273(12) 309(8) mVAE 314(48) 334(48) 336(48) 315(48) 333(48) 337(48) eVAE 319(48) 334(48) 337(48) 321(48) 334(48) 332(48)
TFD
4.4 COMPARISON WITH OTHER MODELS
In Table 2 we compare the generative performance of eVAE with other models, using Parzen log-density. VAE−, mVAE−, and eVAE− refer to models trained using the same architecture as Adversarial Autoencoders, for comparison. Encoders and decoders have L = 2 layers of H = 1000 deterministic units. D = 8 for MNIST, and D = 15 for TFD. VAE, mVAE, and eVAE refer to the best performing models over all architectures from Table 1. For MNIST, the VAE model is (L,H,D) = (3, 500, 8), mVAE is (3, 1000, 24), and eVAE is (3, 500, 48). For TFD, the VAE model is (3, 500, 15), mVAE is (3, 1000, 50), and eVAE is (3, 500, 25).
We observe that eVAE significantly improves over VAE and is competitive with several state-of-the-art models, notably Adversarial Autoencoders. Samples from eVAE on MNIST and TFD are shown in Fig. 7.
5 RELATED WORK
A number of applications use variational autoencoders as a building block. In Gregor et al. (2015), a generative model for images is proposed in which the generator of the VAE is an attention-based recurrent model that is conditioned on the canvas drawn so far. Eslami et al. (2016) proposes a VAE-based recurrent generative model that describes images as formed by sequentially choosing an object to draw and adding it to a canvas that is updated over time. In Kulkarni et al. (2015), VAEs are used for rendering 3D objects. Conditional variants of VAE are also used for attribute specific image generation (Yan et al., 2015) and future frame synthesis (Xue et al., 2016). All these applications suffer from the problem of model over-pruning and hence have adopted strategies that takes away the clean mathematical formulation of VAE. We have discussed these in § 2.1.
A complementary approach to the problem of model pruning in VAE was proposed in Burda et al. (2015); the idea is to improve the variational bound by using multiple weighted posterior samples. Epitomic VAE provides improved latent capacity even when only single sample is drawn from the posterior.
Methods to increase the flexibility of posterior inference are proposed in (Salimans et al., 2015; Rezende & Mohamed, 2016; Kingma et al., 2016). In Rezende & Mohamed (2016), posterior approximation is constructed by transforming a simple initial density into a complex one with a sequence of invertible transformations. In a similar vein, Kingma et al. (2016) augments the flexibility of the posterior through autoregression over projections of stochastic latent variables. However, the problem of over pruning still persists: for instance, Kingma et al. (2016) enforces a minimum information constraint to ensure that all units are used.
Related is the research in unsupervised sparse overcomplete representations, especially with group sparsity constraints c.f. (Gregor et al., 2011; Jenatton et al., 2011). In the epitomic VAE, we have similar motivations that enable learning better generative models of data.
6 CONCLUSION
This paper introduces Epitomic VAE, an extension of variational autoencoders, to address the problem of model over-pruning, which has limited the generation capability of VAEs in high-dimensional spaces. Based on the intuition that subconcepts can be modeled with fewer dimensions than the full latent space, epitomic VAE models the latent space as multiple shared subspaces that have learned specializations. We show how this model addresses the model over-pruning problem in a principled manner, and present qualitative and quantitative analysis of how eVAE enables increased utilization of the model capacity to model greater data variability. We believe that modeling the latent space as multiple structured subspaces is a promising direction of work, and allows for increased effective capacity that has potential to be combined with methods for increasing the flexibility of posterior inference.
7 ACKNOWLEDGMENTS
We thank the reviewers for constructive comments. Thanks to helpful discussions with Marc’Aurelio Ranzato, Joost van Amersfoort and Ross Girshick. We also borrowed the term ‘epitome’ from an earlier work of Jojic et al. (2003).
8 APPENDIX
8.1 EFFECT OF KL WEIGHT λ ON RECONSTRUCTION
We visualize VAE reconstructions as the KL term weight λ is tuned down to keep latent units active. The top half of each figure are the original digits, and the bottom half are the corresponding reconstructions. While reconstruction performance is good, generation is poor (Fig. 1). This illustrates that VAE learns to model well only regions of the posterior manifold near training samples, instead of generalizing to model well the full posterior manifold.
8.2 EFFECT OF INCREASING LATENT DIMENSION ON RECONSTRUCTION
In § 4.1, Fig. 5 shows the effect of increasing latent dimension on generation for VAE, Dropout VAE, and eVAE models. Here we show the effect of the same factor on reconstruction quality for the models. The top half of each figure are the original digits, and the bottom half are the corresponding reconstructions. As the dimension of the latent variable z increases from 2-d to 20-d, reconstruction becomes very sharp (the best model), but generation degrades (Fig. 5). Dropout VAE has poorer reconstruction but still blurred generation, while eVAE is able to achieve both good reconstruction and generation.
8.3 EVALUATION METRIC FOR GENERATION
There have been multiple approaches for evaluation of variational autoencoders, in particular log-likelihood lower bound and log-density (using the Parzen window estimator, Rifai et al. (2012)). Here we show that for the generation task, log-density is a more appropriate measure than log-likelihood lower bound. Models are trained on binarized MNIST, to be consistent with literature reporting likelihood bounds. The encoder and decoder for all models consist of a single deterministic layer with 500 units.
Table 3 shows the log-likelihood bound and log-density for VAE and eVAE models as the dimensionD of latent variable z is increased. For VAE, as D increases, the likelihood bound improves, but the log-density decreases. Referring to the corresponding generation samples in Fig. 11, we see that sample quality in fact decreases, counter to the likelihood bound but consistent with log-density. The reported VAE bounds and sample quality also matches Figs. 2 and 5 in Kingma & Welling (2014). On the other hand, eVAE log-density first decreases and then improves with larger D. We see that this is also consistent with Fig. 11, where eVAE samples for D = 8 are the most interpretable overall, and D = 48 improves over D = 24 but still has some degenerate or washed out digits. (Note that these models are consistent with Kingma & Welling (2014) but are not the best-performing models reported in our experiments.) Since our work is motivated by the generation task, we therefore use log-density as the evaluation metric in our experiments.
Intuitively, the reason why VAE improves the likelihood bound but generation quality still decreases can be seen in the breakdown of the bound into the reconstruction and KL terms (Table 3 and Fig. 10). The improvement of the bound is due to large improvement in reconstruction, but the KL becomes significantly worse. This has a negative effect on generation, since the KL term is closely related to generation. On the other hand, eVAE reconstruction improves to a lesser extent, but the KL is also not as strongly affected, so generation ability remains stronger overall. As a result of this, simply tuning the KL weight λ in the training objective is insufficient to improve VAE generation, as shown in Fig. 1 in the main paper.

This paper replaces the Gaussian prior often used in a VAE with a group sparse prior. They modify the approximate posterior function so that it also generates group sparse samples. The development of novel forms for the generative model and inference process in VAEs is an active and important area of research. I don't believe the specific choice of prior proposed in this paper is very well motivated however. I believe several of the conceptual claims are incorrect. The experimental results are unconvincing, and I suspect compare log likelihoods in bits against competing algorithms in nats.
Some more detailed comments:
In Table 1, the log likelihoods reported for competing techniques are all in nats. The reported log likelihood of cVAE using 10K samples is not only higher than the likelihood of true data samples, but is also higher than the log likelihood that can be achieved by fitting a 10K k-means mixture model to the data (eg as done in "A note on the evaluation of generative models"). It should nearly impossible to outperform a 10K k-means mixture on Parzen estimation, which makes me extremely skeptical of these eVAE results. However, if you assume that the eVAE log likelihood is actually in bits, and multiply it by log 2 to convert to nats, then it corresponds to a totally believable log likelihood. Note that some Parzen window implementations report log likelihood in bits. Is this experiment comparing log likelihood in bits to competing log likelihoods in nats? (also, label units -- eg bits or nats -- in table)
It would be really, really, good to report and compare the variational lower bound on the log likelihood!! Alternatively, if you are concerned your bound is loose, you can use AIS to get a more exact measure of the log likelihood. Even if the Parzen window results are correct, Parzen estimates of log likelihood are extremely poor. They possess any drawback of log likelihood evaluation (which they approximate), and then have many additional drawbacks as well.
The MNIST sample quality does not appear to be visually competitive. Also -- it appears that the images are of the probability of activation for each pixel, rather than actual samples from the model. Samples would be more accurate, but either way make sure to describe what is shown in the figure.
There are no experiments on non-toy datasets.
I am still concerned about most of the issues I raised in my questions below. Briefly, some comments on the authors' response:
1. "minibatches are constructed to not only have a random subset of training examples but also be balanced w.r.t. to epitome assignment (Alg. 1, ln. 4)."
Nice! This makes me feel better about why all the epitomes will be used.
2. I don't think your response addresses why C_vae would trade off between data reconstruction and being factorial. The approximate posterior is factorial by construction -- there's nothing in C_vae that can make it more or less factorial.
3. "For C_vae to have zero contribution from the KL term of a particular z_d (in other words, that unit is deactivated), it has to have all the examples in the training set be deactivated (KL term of zero) for that unit"
This isn't true. A standard VAE can set the variance to 1 and the mean to 0 (KL term of 0) for some examples in the training set, and have non-zero KL for other training examples.
4. The VAE loss is trained on a lower bound on the log likelihood, though it does have a term that looks like reconstruction error. Naively, I would imagine that if it overfits, this would correspond to data samples becoming more likely under the generative model.
5/6. See Parzen concerns above. It's strange to train a binary model, and then treat it's probability of activation as a sample in a continuous space.
6. "we can only evaluate the model from its samples"
I don't believe this is true. You are training on a lower bound on the log likelihood, which immediately provides another method of quantitative evaluation. Additionally, you could use techniques such as AIS to compute the exact log likelihood.
7. I don't believe Parzen window evaluation is a better measure of model quality, even in terms of sample generation, than log likelihood.


CHARGED POINT NORMALIZATION AN EFFICIENT SOLUTION TO THE SADDLE POINT PROBLEM
Recently, the problem of local minima in very high dimensional non-convex optimization has been challenged and the problem of saddle points has been introduced. This paper introduces a dynamic type of normalization that forces the system to escape saddle points. Unlike other saddle point escaping algorithms, second order information is not utilized, and the system can be trained with an arbitrary gradient descent learner. The system drastically improves learning in a range of deep neural networks on various data-sets in comparison to non-CPN neural networks. 1 SADDLE POINT PROBLEM 1.
1 SADDLE POINT PROBLEM
1.1 INTRODUCTION
Recently more and more attention has focused on the problem of saddle points in very high dimensional non-convex optimization. Saddle points represent points in the optimization problem where the first order gradients are all zero, but the stationary point is neither a maxima or a minima. The saddle point of a function can be confirmed by using the eigenvalues of Hessian matrix. If the set of eigenvalues contains at least one negative eigenvalue and at least one positive eigenvalue the point is said to be a saddle point. One way to analyze the prevalence of saddle point is to assign a joint probability density to the eigenvalues of the Hessian matrix at a critical point.
• If the eigenvalues are all negative, then the critical point is a local maximum • If the eigenvalues are all positive, then the critical point is a local minimum. • If the eigenvalues contain at least one positive and at least one negative eigenvalue then the
point is a saddle point.
If p(λ1, λ2, ..., λn) is the joint probability density function then the probability that the Hessian matrix resembles a saddle point is, given that the Hessian is not singular is:
1− ∫ ∞ 0 ∫ ∞ 0 ... ∫ ∞ 0 p(λ1, λ2, ..., λn)dλ1dλ2...dλn
− ∫ 0 −∞ ∫ 0 −∞ ... ∫ 0 −∞ p(λ1, λ2, ..., λn)dλ1dλ2...dλn
(1)
Another way to interpret the expression above is to realize that each of the two n-integrals represents the joint density summation of the two hyper-cubes, one in the direction of all the positive axis, and the other in all the negative axis. Each respectively representing minimas and maximas.
Theorem 1 The space of eigenvalues of a non-singular Hessian matrix that represent minimas and maximas in comparison to the total space, decreases by 2n asymptotically.
The amount of unique hypercubes starting from the origin and spanning along the axis is 2n. The amount of hypercubes representing minimas and maximas is two. Therefore the fraction of the space that contains the eigenvalues that would indicate either a minima or a maxima is 21−n, where n represents the dimensionality of the Hessian matrix.
What this shows is that as we increase the dimensionality of our optimization problem, the fraction of the total space that represents either a local minima or maxima decreases exponentially by a factor of two.
Although this interpretation gives some intuition behind the saddle point problem, we cannot conclusively say that the probability of a critical point being a saddle point increases exponentially because we do not know the behavior of the joint probability function.
1.2 GRADIENT DESCENT BEHAVIOR AROUND SADDLE POINTS
To understand the shortcomings of first order gradient descent algorithms around saddle points we will analyze the neighborhood a saddle point. Given a function f , the Taylor expansion around the saddle point x is given by:
f(x+ δ) = f(x) + 1
2 δTHδ (2)
The first order term disappears because we are at a critical point. Denoting e1, e2, ..., en as the eigenvectors of the non-degenerate Hessian H , and λ1, λ2, ..., λn as the respective eigenvalues, we can use the change of coordinates methods to rewrite the Taylor expansion in terms of the eigenvectors:
v = 1
2 eT1... eTn  δ f(x+ δ) = f(x) + 1
2 n∑ i=1 λi(e T i δ) 2 = f(x) + n∑ i=1 λiv 2 i
(3)
From the last equation we can analyze the behavior of first order gradient descent algorithms. Specifically by looking at the behavior with respect to the signs of the eigenvalues. If eigenvalue λi is positive then the optimization point will move toward the critical point x. If eigenvalue λi is negative the optimization point will move away from the critical point.
This shows that the direction of the gradient descent algorithm is not the problem with gradient descent algorithms around saddle points, but rather the step of the algorithm. This problem is sometimes amplified because of the plateaus surrounding the critical point, as shown in (Saad & Solla, 1996). Another complication visible from equation 2, is that if the step size is greater than maxλ−1, the gradient descent algorithm will begin to diverge. Therefore one large eigenvalues of the surface of the error function, will force the gradient descent algorithms to take very small steps in all the other directions.
A very similiar derivation and explanation was shows in (Dauphin et al., 2014)
2 CHARGED POINT NORMALIZATION
2.1 METAPHOR
The metaphor for our method goes as follows. The current point in our optimization is a small positively charged point that is moving on the neutral surface of error. Our normalization works by dynamically placing other positively charged points around the surface of error to ‘push’ our optimization point away from undesirable positions. Optimally we would run the gradient descent algorithm until convergence, check if the converged point is a saddle point, place a positively charged point near the saddle point and continue the optimization. This metaphor was what gave inspiration to the derivation of our normalization.
2.2 INTRODUCTION
The general optimization problem is defined as:
L(f ;X,Y ) = n∑ i=1 V (f(Xi),Yi) + λR(f) (4)
The formulation is static, given the same function and the same X and Y the loss will always be equal. Our formulation introduces a dynamic normalization function R. Therefore the loss function becomes defined as:
Lt(f ;X,Y ) = n∑ i=1 V (f(Xi),Yi) + λRt(f) (5)
The function f contains dynamic parameters W t1 ,W t 2 , ...,W t n, while the function R contains parameters: β, p, φ and Ŵ t1 , Ŵ t 2 , ..., Ŵ t n, symbolizing the decay factor, norm, merge function and merge values respectfully. The t term inW tn represents the value ofWn at time t of the optimization algorithm. Charged Point Normalization is now defined as:
Rt(f) = e−βt∑n
i=1 ||W ti − Ŵ ti ||p (6)
The update for the merge values is defined as:
Ŵ t+1i = φ(Ŵ t i ,W t i )
Ŵ 1i =W 1 i +
(7)
where is a source of random error to ensure we do not have a division by zero. In our experiments was a matrix of the same size as W 1i with random entries sampled from a normal distribution with a zero mean and a very small standard deviation.
What this type of normalization attempts to do is to reward the optimization algorithm for taking steps that maximize the distance between the new point and the trailing point. This can be seen as a more dynamic and adaptive version of momentum that kicks in when the optimization problem settles down into a saddle point or long plateau. That being said, CPN can still be used with traditional momentum methods, as shown by the experiments below.
2.3 CHOICE OF HYPERPARAMETER
The φ function can be any function that merges the two parameters into one parameter of the same dimension. Throughout this whole paper we used the exponential moving average for our φ function.
φ(Ŵ ti ,W t i ) = αŴ t + (1− α)W ti α ∈ (0, 1)
(8)
Although to keep up with the metaphor Coulomb’s inverse squared law did not work as well as projected, through trial and error, the p value that worked the best was 1. The 1-norm simply is the sum of absolute values.
3 EXPERIMENTS
3.1 INTRODUCTION
Charged Point Normalization was implemented in Theano (Bastien et al., 2012) and integrated with the Keras (Chollet, 2015) library. We utilized the convolutional neural networks and recurrent networks in the keras library as well. All training and testing was run on a Nvidia GTX 980 GPU. We
do not show results on a validation set, because we care about the efficiency and performance of the optimization algorithm, not whether or not it overfits. The over-fitting of a model is not the fault of the optimization routine but rather the fault of the field it is optimizing over. All comparisons between the standard and charged model, started with identical set’s of weights. Throughout all of our experiments, we utilize a softmax layer as the final layer, and consequently all the losses measured throughout this paper will be in terms of categorical cross-entropy. We used the train split of each data-set.
3.2 SIMPLE DEEP NEURAL NETWORKS
3.2.1 MNIST: MULTILAYER PERCEPTRON
The first test conducted was using a multilayer perceptron on the MNIST dataset. The architecture of the neural net contained layers with sizes 784→ 512→ 512→10. All intermediate layers contained rectified linear activations (He et al., 2015), while the final layer is a softmax layer. Between layers, dropout (Hinton et al., 2012) with a probability of 0.2 was added. We compare the standard batch gradient descent algorithm with a step size of 0.001 and batchsize of 128, on the net described above and the same net with Charged Point Normalization (CPN). The CPN hyper-parameters were: β = 0.001, λ = 0.1 with the moving average parameter α = 0.95. The loss we were optimizing over was categorical cross entropy.
3.2.2 MNIST:DEEP AUTOENCODER
The second test conducted on simple neural networks, was in the form of an autoencoder. The architecture of the autoencoder contained layers with sizes 784→ 512→ 512→10→ 512→ 512→10. All layers contained rectified linear activations. Between layers, dropout with a probability of 0.2 was added. The set up of the experiment is almost identical to the previous experiment. The only difference being that in this case, we optimized for binary cross-entropy.
0 20 40 60 80 100 0
0.2
0.4
0.6
0.8
1
Epoch
Loss on MNIST Dataset (MLP)
Charged Standard
0 20 40 60 80 100 0.2
0.4
0.6
Epoch
Loss on MNIST Dataset (AutoEncoder)
Charged Standard
3.2.3 NOTES
It is interesting to note that when the optimization problem is relatively simple, more specifically if the optimization algorithm takes smooth steps, CPN allows the optimization algorithm to take bigger steps in the correct direction. CPN does not display any periodic or chaotic behavior in this scenario. This is not the case for more complicated optimization problems that will be presented below.
3.3 CONVOLUTIONAL NEURAL NETWORKS
3.3.1 CIFAR10
The next experiment conducted was using a convolutional neural network on the CIFAR10 (Krizhevsky et al., a). The architecture was as such:
Convolution2D (32,3,3) → ReLU → Convolution2D (32,3,3) → ReLU → MaxPooling (2,2) → Dropout (0.25)→ Convolution2D (64,3,3)→ ReLU→ Convolution2D (64,3,3)→ ReLU→MaxPooling (2,2)→Dropout (0.25)→Dense (512)→ReLU→Dropout (0.5)→Dense (10)→ Softmax Convolution2D takes the parameters, number of filters, width and height respectfully. Dense take one parameter describing the size of the layer. MaxPooling takes two parameters that signify the
pool size. ReLU is the rectified linear function, while Softmax is the softmax activation function. The optimization algorithm used was stochastic gradient descent with a learning rate of 0.01, decay of 1e − 6, momentum of 0.9, with nesterov acceleration. The batch size used was 32. The hyperparameters for CPN were: β = 0.01, λ = 0.1 with the moving average parameter α = 0.95. 10,000 random images were used from the CIFAR10 data-set instead of the full dataset to speed up learning.
0 20 40 60 80 100 0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Epoch
Loss on CIFAR10 Dataset
Charged Standard
0 20 40 60 80 100 0
0.2
0.4
0.6
0.8
1
Epoch
Accuracy on CIFAR10 Dataset
Charged Standard
It is interesting to note that CPN performs worse until the optimization algorithm reaches the ‘elbow’ of the curve, where then CPN continues along its path, while the standard algorithm begins to converge. CPN also takes steps that are much less ‘optimal’ in the greedy sense, which is why both the loss and accuracy curve behave partially chaotic.
3.3.2 CIFAR100
The CIFAR100 (Krizhevsky et al., b) setup was nearly identical to the CIFAR10 setup. The same architecture of the neural network was used. The only difference was in the λ parameter in the normalization term, which in this case was equal to 0.01. 20, 000 random images were used.
0 20 40 60 80 100 0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Epoch
Loss on CIFAR100 Dataset
Charged Standard
0 20 40 60 80 100 0
0.2
0.4
0.6
0.8
1
Epoch
Accuracy on CIFAR100 Dataset
Charged Standard
The same behavior as in the CIFAR10 experiment was exhibited. The elbow of the loss curve was the point where CPN began to outperform standard optimization.
3.4 RECURRENT NEURAL NETWORKS
3.4.1 INTRODUCTION
Recurrent neural networks are notorious for being hard to train, and having a tendency to generally underfit (Pascanu et al., 2012). In this section we show that CPN successfully escapes saddle points presented in recurrent neural networks.
Sentence Embedding Question Embedding
RNN RNN
Concat
Dense
We selected the path-finding problem of the BABI dataset due to it being the most difficult task. The architecture consisted of two recurrent networks and one standard neural network. Each of the recurrent neural networks had a structure: Embedding→ RNN. The embedding, sentence and query hidden layer size was set to 3. The final network concatenated the two recurrent network outputs and fed the result into a dense layer with an output size of vocabsize. Refer to figure 1 for a diagram. We ran our experiment with two different recurrent neural network structures: Gated Recurrent Units (GRU) (Chung et al., 2014) and Long Short Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) . The ADAM (Kingma & Ba, 2014) optimization algorithm was used for both recurrent structures with the parameters: α = 0.001, β1 = 0.9, β2 = 0.999, = 1e − 08. For the LSTM architecture, CPN hyper-parameters were: β = 0.0025, λ = .03, α = 0.95. For the GRU architecture, CPN hyper-parameters were: β = 0.1, λ = .1, α = 0.95.
-0 .0 05 0. 0 0. 00 5 0. 01 0. 01 5 0. 02 0. 02 5 0. 03 0. 03 5 0. 04
0
10
20
30
40
50
60 I->H
-0 .0 2 0. 0 0. 02 0. 04 0. 06 0. 08 0. 1 0. 12 0. 14 0. 16
0
10
20
30
40
50
60 H->O
Charged: Eigenvalue Distribution
-0 .0 05 0. 0 0. 00 5 0. 01 0. 01 5 0. 02 0. 02 5 0. 03 0. 03 5
0
10
20
30
40
50
60
70
80 I->H
-0 .0 5 0. 0 0. 05 0. 1 0. 15 0. 2 0. 25
0
1
2
3
4
5
6
7
8 H->O
Vanilla: Eigenvalue Distribution
From the accuracy graphs we can see the CPN causes the recurrent network to escape the saddle point earlier than a recurrent network with no CPN.
4 NORMALIZATION BEHAVIOR
4.1 EXPLORATION VS EXPLOITATION
In a standard gradient descent with no normalization, the updates taken by the algorithm are always greedy, in terms of always minimizing the loss of the model. Their is no exploration done; gradient descent is by nature a greedy algorithm, optimizing only locally. What CPN allows the gradient descent to do, is to move in non-optimal directions early on in the optimization routine, while still allowing for precise finetuning at the end of the model. This trade-off is controlled by the β parameter.
4.2 BEHAVIOR AROUND SADDLE POINTS
A vanilla neural network with one single hidden layer was trained on a down sampled 8× 8 version of the MNIST dataset (Lecun et al., 1998). Full gradient descent was ran on the 10, 000 random images until convergence. We compare the differences between the eigenvalue distributions between the neural network with CPN and the neural network without it. Recall the tighter the range of the eigenvalues is, the larger steps the gradient descent algorithm can take without worrying about divergence as explained in section 1.2.
The graph above shows a kernel density estimation done on the input to hidden and hidden to output Hessian’s at the near critical point. There are both negative and positive eigenvalues, especially in the hidden to output weights, therefore it is safe enough to say that we are at a saddle point (Turlach, 1993). The first graph represents the CPN neural network while next graph represents a non-normalized neural network. The CPN network shows a tighter distribution as well as more of the eigenvalues being focused near 0.
4.3 TOY EXAMPLE
To ensure that the normalization is actually repelling the optimization point from saddle points, and that the results achieved in the experimental section are not due to some confounding factors we utilize a low-dimensional experiment to show the repelling effects of CPN.
We utilize the monkey saddle as the optimization surface. The monkey saddle has a saddle point surrounded by plateaus in all directions. It is defined as x3 − 3xy2. Referring to section 1.2, we discussed that the direction of gradient descent algorithms was not the shortcoming of gradient descent algorithms around saddle points,but rather the problem was with the step size. What CPN should in theory do is allow the optimization routine to take larger steps.
Below are two figures. The first one shows the behavior of a five common gradient descent algorithms starting at a point near the saddle point (point: (x = 0.0001, y = −0.0001)) (Zeiler, 2012), (Duchi et al., 2010). The next figure shows the same algorithms starting at the same point but utilizing CPN. All visualization were done using the matplotlib library (Hunter, 2007).
The hyper-parameters used, were all the default hyper-parameters in the keras library apart from Adam (to make it visible on the graphs). All hyper-parameters are available in Table 1. SGD Accelerated refers to the standard SGD algorithm using momentum and nesterov acceleration. The CPN parameters were chosen using a very small discrete grid-search. In reality just about any reasonable arbitrary parameters can be chosen in order for CPN to work in this experiment, a gridsearch was not neccessary to find a solution. This is why we reuse two sets of hyper-parameters for this toy problem. (Nesterov, 1983).
Each algorithm performed better when coupled with CPN than without, the loss was computed using the monkey saddle equation above. All the losses for both CPN and Non-CPN are available in Table 2. CPN allowed the optimization algorithms to escape the saddle point quickly even though the gradient near the starting point of the optimization was near zero.
• Without CPN only the Adam algorithm escaped the plateau in less than a 1000 iterations. • With CPN every algorithm apart from AdaGrad successfully escaped the plateau in less
than 120 iterations, most notable being SGD Accelerated, which escaped in just 8 iterations.
From this toy example we can conclude that CPN does in fact repel the optimization algorithm away from saddle points, and therefore the results from the experiments are due to this phenomena and most likely no other confounding factors.
4.4 PERIODICITY AND TERMINAL BEHAVIOR
As shown in the experiments done on the CIFAR datasets, CPN has a tendency to force the optimization algorithm to act more chaotically. The exponential term in the normalization term is there to ensure that the optimization algorithm does not get stuck in an periodic path. It is trivial to see that as the time of the optimization goes toward infinity the impact of the normalization will tend toward 0. Therefore if the optimization algorithm does not reach a local minimum, but is rather in an elliptical path, assuming that the λ term is not great enough to push the point out of the local minimum, the optimization algorithm will eventually reach the local minimum.
5 NOTES ON HYPER-PARAMETERS
Due to restrictions on our hardware resources, we did not have enough time to run a comprehensive study on the behavior of CPN with respect to its hyper-parameters. Throughout this paper the selection of hyper-parameters was kept rather simple. We selecting the hyper-parameters in a feasible range, and then adjusted them by either by hand around 4-8 times, or similarly by running a basic discrete grid-search that ran over that same amount of hyper-parameters. So in no way are the hyper-parameters for CPN chosen in this paper optimal for the various setups explained, but yet the results we found were somewhat substantial, which we find quite optimistic.
6 WEAKNESSES
• CPN with a exponential moving average for the φ function, introduces 2 extra hyperparameters, not including the normalization scalar λ.
• In terms of implementation; CPN doubles the amount of memory needed for the optimization problem, as a trailing copy of the parameters must be kept.
• The fraction term in CPN will generally contain small floating points in both numerator and denominator and this can sometimes lead to numerical instability.
• If saddle points are reached at a really late time in the optimization algorithm, the exponential decay will nullify the effects of CPN. A possible solution would be to substitute the exponential decay term with some type of periodic decay.
7 CONCLUSION
In this paper we introduced a new type of dynamic normalization that allows gradient based optimization algorithms to escape saddle points. We showed empirical results on standard data-sets, that show CPN successful escapes saddle points on various neural network architectures. We discussed the theoretical properties of first order gradient descent algorithms around saddle points as well as discussed the influence of the largest eigenvalue on the step taken. Empirical results were shown that confirmed the hunch that the hessian of charged point normalized neural networks contains eigenvalues which are less in magnitude than their non-normalized counterpart.

Summary:
This paper proposes a regularizer that is claimed to help escaping from the saddle points. The method is inspired from physics, such that thinking of the optimization process is moving a positively charged particle would over the error surface which would be pushed away from saddle points due to the saddle point being positively changed as well. Authors of the paper show results over several different datasets.
Overview of the Review:
    Pros:
        - The idea is very interesting.
        - The diverse set of results on different datasets.
    Cons:
        - The justification is not strong enough.
        - The paper is not well-written.
        - Experiments are not convincing enough.
Criticisms:
I liked the idea and the intuitions coming from the paper. However, I think this paper is not written well. There are some variables introduced in the paper and not explained good-enough, for example in 2.3, the authors start to talk about p without introducing and defining it properly. The only other place it appears before is Equation 6. The Equations need some work as well, some work is needed in terms of improving the flow of the paper, e.g., introducing all the variables properly before using them.
Equation 6 appears without a proper explanation and justification. It is necessary to explain it what it means properly since I think this is one of the most important equation in this paper. More analysis on what it means in terms of optimization point of view would also be appreciated.
$\phi$ is not a parameter, it is a function which has its own hyper-parameter $\alpha$.
It would be interesting to report validation or test results on a few tasks as well. Since this method introduced as an additional cost function, its effect on the validation/test results would be interesting as well.
The authors should discuss more on how they choose the hyper-parameters of their models.
The Figure 2 and 3 does not add too much to the paper and they are very difficult to understand or draw any conclusions from.
There are lots of Figures under 3.4.2 without any labels of captions. Some of them are really small and difficult to understand since the labels on the figures appear very small and somewhat unreadable.
A small question:
* Do you also backpropagate through $\tilde{\mW}_i^t$?