AUTOENCODING VARIATIONAL INFERENCE FOR TOPIC MODELS
Topic models are one of the most popular methods for learning representations of text, but a major challenge is that any change to the topic model requires mathematically deriving a new inference algorithm. A promising approach to address this problem is autoencoding variational Bayes (AEVB), but it has proven difficult to apply to topic models in practice. We present what is to our knowledge the first effective AEVB based inference method for latent Dirichlet allocation (LDA), which we call Autoencoded Variational Inference For Topic Model (AVITM). This model tackles the problems caused for AEVB by the Dirichlet prior and by component collapsing. We find that AVITM matches traditional methods in accuracy with much better inference time. Indeed, because of the inference network, we find that it is unnecessary to pay the computational cost of running variational optimization on test data. Because AVITM is black box, it is readily applied to new topic models. As a dramatic illustration of this, we present a new topic model called ProdLDA, that replaces the mixture model in LDA with a product of experts. By changing only one line of code from LDA, we find that ProdLDA yields much more interpretable topics, even if LDA is trained via collapsed Gibbs sampling.
1 INTRODUCTION
Topic models (Blei, 2012) are among the most widely used models for learning unsupervised representations of text, with hundreds of different model variants in the literature, and have have found applications ranging from the exploration of the scientific literature (Blei & Lafferty, 2007) to computer vision (Fei-Fei & Perona, 2005), bioinformatics (Rogers et al., 2005), and archaeology (Mimno, 2009). A major challenge in applying topic models and developing new models is the computational cost of computing the posterior distribution. Therefore a large body of work has considered approximate inference methods, the most popular methods being variational methods, especially mean field methods, and Markov chain Monte Carlo, particularly methods based on collapsed Gibbs sampling.
Both mean-field and collapsed Gibbs have the drawback that applying them to new topic models, even if there is only a small change to the modeling assumptions, requires re-deriving the inference methods, which can be mathematically arduous and time consuming, and limits the ability of practitioners to freely explore the space of different modeling assumptions. This has motivated the development of black-box inference methods (Ranganath et al., 2014; Mnih & Gregor, 2014; Kucukelbir et al., 2016; Kingma & Welling, 2014) which require only very limited and easy to compute information from the model, and hence can be applied automatically to new models given a simple declarative specification of the generative process.
Autoencoding variational Bayes (AEVB) (Kingma & Welling, 2014; Rezende et al., 2014) is a particularly natural choice for topic models, because it trains an inference network (Dayan et al., 1995), a neural network that directly maps a document to an approximate posterior distribution,
∗Additional affiliation: Alan Turing Institute, British Library, 96 Euston Road, London NW1 2DB
without the need to run further variational updates. This is intuitively appealing because in topic models, we expect the mapping from documents to posterior distributions to be well behaved, that is, that a small change in the document will produce only a small change in topics. This is exactly the type of mapping that a universal function approximator like a neural network should be good at representing. Essentially, the inference network learns to mimic the effect of probabilistic inference, so that on test data, we can enjoy the benefits of probabilistic modeling without paying a further cost for inference.
However, despite some notable successes for latent Gaussian models, black box inference methods are significantly more challenging to apply to topic models. For example, in initial experiments, we tried to apply ADVI (Kucukelbir et al., 2016), a recent black-box variational method, but it was difficult to obtain any meaningful topics. Two main challenges are: first, the Dirichlet prior is not a location scale family, which hinders reparameterisation, and second, the well known problem of component collapsing (Dinh & Dumoulin, 2016), in which the inference network becomes stuck in a bad local optimum in which all topics are identical.
In this paper, we present what is, to our knowledge, the first effective AEVB inference method for topic models, which we call Autoencoded Variational Inference for Topic Models or AVITM1. On several data sets, we find that AVITM yields topics of equivalent quality to standard mean-field inference, with a large decrease in training time. We also find that the inference network learns to mimic the process of approximate inference highly accurately, so that it is not necessary to run variational optimization at all on test data.
But perhaps more important is that AVITM is a black-box method that is easy to apply to new models. To illustrate this, we present a new topic model, called ProdLDA, in which the distribution over individual words is a product of experts rather than the mixture model used in LDA. We find that ProdLDA consistently produces better topics than standard LDA, whether measured by automatically determined topic coherence or qualitative examination. Furthermore, because we perform probabilistic inference using a neural network, we can fit a topic model on roughly a one million documents in under 80 minutes on a single GPU, and because we are using a black box inference method, implementing ProdLDA requires a change of only one line of code from our implementation of standard LDA.
To summarize, the main advantages of our methods are:
1. Topic coherence: ProdLDA returns consistently better topics than LDA, even when LDA is trained using Gibbs sampling.
2. Computational efficiency: Training AVITM is fast and efficient like standard mean-field. On new data, AVITM is much faster than standard mean field, because it requires only one forward pass through a neural network.
3. Black box: AVITM does not require rigorous mathematical derivations to handle changes in the model, and can be easily applied to a wide range of topic models.
Overall, our results suggest that AVITM is ready to take its place alongside mean field and collapsed Gibbs as one of the workhorse inference methods for topic models.
2 BACKGROUND
To fix notation, we begin by describing topic modelling and AVITM.
2.1 LATENT DIRICHLET ALLOCATION
We describe the most popular topic model, latent Dirichlet allocation (LDA). In LDA, each document of the collection is represented as a mixture of topics, where each topic βk is a probability distribution over the vocabulary. We also use β to denote the matrix β = (β1 . . . βK). The generative process is then as described in Algorithm 1. Under this generative model, the marginal likelihood of
1Code available at https://github.com/akashgit/autoencoding_vi_for_topic_models
for each document w do Draw topic distribution θ ∼ Dirichlet(α); for each word at position n do
Sample topic zn ∼ Multinomial(1, θ); Sample word wn ∼ Multinomial(1, βzn);
end end
Algorithm 1: LDA as a generative model.
a document w is
(1)p(w|α, β) = ∫ θ ( N∏ n=1 k∑ zn=1 p(wn|zn, β)p(zn|θ) ) p(θ|α)dθ.
Posterior inference over the hidden variables θ and z is intractable due to the coupling between the θ and β under the multinomial assumption (Dickey, 1983).
2.2 MEAN FIELD AND AEVB
A popular approximation for efficient inference in topic models is mean field variational inference, which breaks the coupling between θ and z by introducing free variational parameters γ over θ and φ over z and dropping the edges between them. This results in an approximate variational posterior q(θ, z|γ, φ) = qγ(θ) ∏ n qφ(zn), which is optimized to best approximate the true posterior p(θ, z|w, α, β). The optimization problem is to minimize
(2)L(γ, φ | α, β) = DKL [q(θ, z|γ, φ)||p(θ, z|w, α, β)]− log p(w|α, β).
In fact the above equation is a lower bound to the marginal log likelihood, sometimes called an evidence lower bound (ELBO), a fact which can be easily verified by multiplying and dividing (1) by the variational posterior and then applying Jensen’s inequality on its logarithm. Note that the mean field method optimizes over an independent set of variational parameters for each document. To emphasize this, we will refer to this standard method by the non-standard name of Decoupled Mean-Field Variational Inference (DMFVI).
For LDA, this optimization has closed form coordinate descent equations due to the conjugacy between the Dirichlet and multinomial distributions. Although this is a computationally convenient aspect of DMFVI, it also limits its flexibility. Applying DMFVI to new models relies on the practitioner’s ability to derive the closed form updates, which can be impractical and sometimes impossible.
AEVB (Kingma & Welling, 2014; Rezende et al., 2014) is one of several recent methods that aims at “black box” inference methods to sidestep this issue. First, rewrite the ELBO as
(3)L(γ, φ | α, β) = −DKL [q(θ, z|γ, φ)||p(θ, z|α)] + Eq(θ,z|γ,φ)[log p(w|z, θ, α, β)]
This form is intuitive. The first term attempts to match the variational posterior over latent variables to the prior on the latent variables, while the second term ensures that the variational posterior favors values of the latent variables that are good at explaining the data. By analogy to autoencoders, this second term is referred to as a reconstruction term.
What makes this method “Autoencoding,” and in fact the main difference from DMFVI, is the parameterization of the variational distribution. In AEVB, the variational parameters are computed by using a neural network called an inference network that takes the observed data as input. For example, if the model prior p(θ) were Gaussian, we might define the inference network as a feedforward neural network (µ(w),v(w)) = f(w, γ), where µ(w) and v(w) are both vectors of length k, and γ are the network’s parameters. Then we might choose a Gaussian variational distribution qγ(θ) = N(θ;µ(w), diag(v(w))), where diag(· · ·) produces a diagonal matrix from a column vector. The variational parameters γ can then be chosen by optimizing the ELBO (3). Note that we have
now, unlike DMFVI, coupled the variational parameters for different documents because they are all computed from the same neural network. To compute the expectations with respect to q in (3), Kingma & Welling (2014); Rezende et al. (2014) use a Monte Carlo estimator which they call the “reparameterization trick” (RT; appears also in Williams (1992)). In the RT, we define a variate U with a simple distribution that is independent of all variational parameters, like a uniform or standard normal, and a reparameterization function F such that F (U, γ) has distribution qγ . This is always possible, as we could choose F to be the inverse cumulative distribution function of qγ , although we will additionally want F to be easy to compute and differentiable. If we can determine a suitable F , then we can approximate (3) by taking Monte Carlo samples of U , and optimize γ using stochastic gradient descent.
3 AUTOENCODING VARIATIONAL BAYES IN LATENT DIRICHLET ALLOCATION
Although simple conceptually, applying AEVB to topic models raises several practical challenges. The first is the need to determine a reparameterization function for q(θ) and q(zn) to use the RT. The zn are easily dealt with, but θ is more difficult; if we choose q(θ) to be Dirichlet, it is difficult to apply the RT, whereas if we choose q to be Gaussian or logistic normal, then the KL divergence in (3) becomes more problematic. The second issue is the well known problem of component collapsing (Dinh & Dumoulin, 2016), which a type of bad local optimum that is particularly endemic to AEVB and similar methods. We describe our solutions to each of those problems in the next few subsections.
3.1 COLLAPSING z’S
Dealing with discrete variables like z using reparameterization can be problematic, but fortunately in LDA the variable z can be conveniently summed out. By collapsing z we are left with having to sample from θ only, reducing (1) to
(4)p(w|α, β) = ∫ θ ( N∏ n=1 p(wn|β, θ) ) p(θ|α)dθ.
where the distribution of wn|β, θ is Multinomial(1, βθ), recalling that β denotes the matrix of all topic-word probability vectors.
3.2 WORKING WITH DIRICHLET BELIEFS: LAPLACE APPROXIMATION
LDA gets its name from the Dirichlet prior on the topic proportions θ, and the choice of Dirichlet prior is important to obtaining interpretable topics (Wallach et al., 2009). But it is difficult to handle the Dirichlet within AEVB because it is difficult to develop an effective reparameterization function for the RT. Fortunately, a RT does exist for the Gaussian distribution and has been shown to perform quite well in the context of variational autoencoder (VAE) (Kingma & Welling, 2014).
We resolve this issue by constructing a Laplace approximation to the Dirichlet prior. Following MacKay (1998), we do so in the softmax basis instead of the simplex. There are two benefits of this choice. First, Dirichlet distributions are unimodal in the softmax basis with their modes coinciding with the means of the transformed densities. Second, the softmax basis also allows for carrying out unconstrained optimization of the cost function without the simplex constraints. The Dirichlet probability density function in this basis over the softmax variable h is given by
(5)P (θ(h)|α) = Γ( ∑ k αk)∏
k Γ(αk) ∏ k θαkk g(1 Th).
Here θ = σ(h), where σ(.) represents the softmax function. Recall that the Jacobian of σ is proportional to ∏ k θk and g(·) is an arbitrary density that ensures integrability by constraining the redundant degree of freedom. We use the Laplace approximation of Hennig et al. (2012), which
has the property that the covariance matrix becomes diagonal for large k (number of topics). This approximation to the Dirichlet prior p(θ|α) is results in the distribution over the softmax variables h as a multivariate normal with mean µ1 and covariance matrix Σ1 where
µ1k = logαk − 1
K ∑ i logαi
Σ1kk = 1
αk
( 1− 2
K
) + 1
K2 ∑ i 1 αk . (6)
Finally, we approximate p(θ|α) in the simplex basis with p̂(θ|µ1,Σ1) = LN (θ|µ1,Σ1) where LN is a logistic normal distribution with parameters µ1,Σ1. Although we approximate the Dirichlet prior in LDA with a logistic normal, this is not the same idea as a correlated topic model (Blei & Lafferty, 2006), because we use a diagonal covariance matrix. Rather, it is an approximation to standard LDA.
3.3 VARIATIONAL OBJECTIVE
Now we can write the modified variational objective function. We use a logistic normal variational distribution over θ with diagonal covariance. More precisely, we define two inference networks as feed forward neural networks fµ and fΣ with parameters δ; the output of each network is a vector in RK . Then for a document w, we define q(θ) to be logistic normal with mean µ0 = fµ(w, δ) and diagonal covariance Σ0 = diag(fΣ(w, δ)), where diag converts a column vector to a diagonal matrix. Note that we can generate samples from q(θ) by sampling ∼ N (0, I) and computing θ = σ(µ0 + Σ 1/2 0 ).
We can now write the ELBO as
L(Θ) = D∑ d=1
[ − ( 1
2
{ tr(Σ−11 Σ0) + (µ1 −µ0)TΣ −1 1 (µ1 −µ0)−K + log
|Σ1| |Σ0|
}) (7)
+E ∼N (0,I) [ w>d log ( σ(β)σ(µ0 + Σ 1/2 0 ) )]] ,
where Θ represents the set of all the model and variational parameters and w1 . . .wD are the documents in the corpus. The first line in this equation arises from the KL divergence between the two logistic normal distributions q and p̂, while the second line is the reconstruction error.
In order to impose the simplex constraint on the β matrix during the optimization, we apply the softmax transformation. That is, each topic βk ∈ RV is unconstrained, and the notation σ(β) means to apply the softmax function separately to each column of the matrix β. Note that the mixture of multinomials for each word wn can then be written as p(wn|β, θ) = [σ(β)θ]wn , which explains the dot product in (7). To optimize (7), we use stochastic gradient descent using Monte Carlo samples from , following the Law of the Unconscious Statistician.
3.4 TRAINING AND PRACTICAL CONSIDERATIONS: DEALING WITH COMPONENT COLLAPSING
AEVB is prone to component collapsing (Dinh & Dumoulin, 2016), which is a particular type of local optimum very close to the prior belief, early on in the training. As the latent dimensionality of the model is increased, the KL regularization in the variational objective dominates, so that the outgoing decoder weights collapse for the components of the latent variable that reach close to the prior and do not show any posterior divergence. In our case, the collapsing specifically occurs because of the inclusion of the softmax transformation to produce θ. The result is that the k inferred topics are identical as shown in table 7.
We were able to resolve this issue by tweaking the optimization. Specifically, we train the network with the ADAM optimizer (Kingma & Ba, 2015) using high moment weight (β1) and learning rate (η). Through training at higher rates, early peaks in the functional space can be easily avoided. The
problem is that momentum based training coupled with higher learning rate causes the optimizer to diverge. While explicit gradient clipping helps to a certain extent, we found that batch normalization (Ioffe & Szegedy, 2015) does even better by smoothing out the functional space and hence curbing sudden divergence.
Finally, we also found an increase in performance with dropout units when applied to θ to force the network to use more of its capacity.
While more prominent in the AEVB framework, the collapsing can also occurs in DMFVI if the learning offset (referred to as the τ parameter (Hofmann, 1999)) is not set properly. Interestingly, a similar learning offset or annealing based approach can also be used to down-weight the KL term in early iterations of the training to avoid local optima.
4 PRODLDA: LATENT DIRICHLET ALLOCATION WITH PRODUCTS OF EXPERTS
In LDA, the distribution p(w|θ, β) is a mixture of multinomials. A problem with this assumption is that it can never make any predictions that are sharper than the components that are being mixed (Hinton & Salakhutdinov, 2009). This can result in some topics appearing that are poor quality and do not correspond well with human judgment. One way to resolve this issue is to replace this word-level mixture with a weighted product of experts which by definition is capable of making sharper predictions than any of the constituent experts (Hinton, 2002). In this section we present a novel topic model PRODLDA that replaces the mixture assumption at the word-level in LDA with a weighted product of experts, resulting in a drastic improvement in topic coherence. This is a good illustration of the benefits of a black box inference method, like AVITM, to allow exploration of new models.
4.1 MODEL
The PRODLDA model can be simply described as latent Dirichlet allocation where the word-level mixture over topics is carried out in natural parameter space, i.e. the topic matrix is not constrained to exist in a multinomial simplex prior to mixing. In other words, the only changes from LDA are that β is unnormalized, and that the conditional distribution of wn is defined as wn|β, θ ∼ Multinomial(1, σ(βθ)).
The connection to a product of experts is straightforward, as for the multinomial, a mixture of natural parameters corresponds to a weighted geometric average of the mean parameters. That is, consider two N dimensional multinomials parametrized by mean vectors p and q. Define the corresponding natural parameters as p = σ(r) and q = σ(s), and let δ ∈ [0, 1]. It is then easy to show that
P ( x|δr + (1− δ)s ) ∝ N∏ i=1 σ(δri + (1− δ)si)xi ∝ N∏ i=1 [rδi · s (1−δ) i ] xi .
So the PRODLDA model can be simply described as a product of experts, that is, p(wn|θ, β) ∝∏ k p(wn|zn = k, β)θk . PRODLDA is an instance of the exponential-family PCA (Collins et al., 2001) class, and relates to the exponential-family harmoniums (Welling et al., 2004) but with nonGaussian priors.
5 RELATED WORK
For an overview of topic modeling, see Blei (2012). There are several examples of topic models based on neural networks and neural variational inference (Hinton & Salakhutdinov, 2009; Larochelle & Lauly, 2012; Mnih & Gregor, 2014; Miao et al., 2016) but we are unaware of methods that apply AEVB generically to a topic model specified by an analyst, or even of a successful application of AEVB to the most widely used topic model, latent Dirichlet allocation.
Recently, Miao et al. (2016) introduced a closely related model called the Neural Variational Document Model (NVDM). This method uses a latent Gaussian distribution over topics, like probabilistic latent semantic indexing, and averages over topic-word distributions in the logit space. However,
they do not use either of the two key aspects of our work: explicitly approximating the Dirichlet prior using a Gaussian, or high-momentum training. In the experiments we show that these aspects lead to much improved training and much better topics.
6 EXPERIMENTS AND RESULTS
Qualitative evaluation of topic models is a challenging task and consequently a large body of work has developed automatic evaluation metrics that attempt to match human judgment of topic quality. Traditionally, perplexity has been used to measure the goodness-of-fit of the model but it has been repeatedly shown that perplexity is not a good metric for qualitative evaluation of topics (Newman et al., 2010). Several new metrics of topic coherence evaluation have thus been proposed; see Lau et al. (2014) for a comparative review. Lau et al. (2014) showed that among all the competing metrics, normalized pointwise mutual information (NPMI) between all the pairs of words in a set of topics matches human judgment most closely, so we adopt it in this work. We also report perplexity, primarily as a way of evaluating the capability of different optimizers. Following standard practice (Blei et al., 2003), for variational methods we use the ELBO to calculate perplexity. For AEVB methods, we calculate the ELBO using the same Monte Carlo approximation as for training.
We run experiments on both the 20 Newsgroups (11,000 training instances with 2000 word vocabulary) and RCV1 Volume 2 ( 800K training instances with 10000 word vocabulary) datasets. Our preprocessing involves tokenization, removal of some non UTF-8 characters for 20 Newsgroups and English stop word removal. We first compare our AVITM inference method with the standard online mean-field variational inference (Hoffman et al., 2010) and collapsed Gibbs sampling (Griffiths & Steyvers, 2004) on the LDA model. We use standard implementations of both methods, scikit-learn for DMFVI and mallet (McCallum, 2002) for collapsed Gibbs. Then we compare two autoencoding inference methods on three different topic models: standard LDA, PRODLDA using our inference method and the Neural Variational Document Model (NVDM) (Miao et al., 2016), using the inference described in the paper.2
Tables 1 and 2 show the average topic coherence values for all the models for two different settings of k, the number of topics. Comparing the different inference methods for LDA, we find that, consistent with previous work, collapsed Gibbs sampling yields better topics than mean-field methods. Among the variational methods, we find that VAE-LDA model (AVITM) 3 yields similar topic coherence and perplexity to the standard DMFVI (although in some cases, VAE-LDA yields significantly better topics). However, AVITM is significantly faster to train than DMFVI. It takes 46 seconds on 20 Newsgroup compared to 18 minutes for DMFVI. Whereas for a million document corpus of RCV1 it only under 1.5 hours while scikit-learn’s implementation of DMFVI failed to return any results even after running for 24 hours.4
Comparing the new topic models than LDA, it is clear that PRODLDA finds significantly better topics than LDA, even when trained by collapsed Gibbs sampling. To verify this qualitatively, we display examples of topics from all the models in Table 6. The topics from ProdLDA appear visually more coherent than NVDM or LDA. Unfortunately, NVDM does not perform comparatively to LDA
2We have used both https://github.com/carpedm20/variational-text-tensorflow and the NVDM author’s (Miao et al., 2016) implementation.
3We recently found that ’whitening’ the topic matrix significantly improves the topic coherence for VAELDA. Manuscript in preparation.
4Therefore, we were not able to report topic coherence for DMFVI on RCV1
for any value of k. To avoid any training dissimilarities we train all the competing models until we reach the perplexities that were reported in previous work. These are reported in Table 35.
A major benefit of AVITM inference is that it does not require running variational optimization, which can be costly, for new data. Rather, the inference network can be used to obtain topic proportions for new documents for new data points without running any optimization. We evaluate whether this approximation is accurate, i.e. whether the neural network effectively learns to mimic probabilistic inference. We verify this by training the model on the training set, then on the test set, holding the topics (β matrix) fixed, and comparing the test perplexity if we obtain topic proportions by running the inference neural network directly, or by the standard method of variational optimization of the inference network on the test set. As shown in Table 4, the perplexity remains practically un-changed. The computational benefits of this are remarkable. On both the datasets, computing perplexity using the neural network takes well under a minute, while running the standard variational approximation takes ∼ 3 minutes even on the smaller 20 Newsgroups data. Finally, we investigate the reasons behind the improved topic coherence in PRODLDA. First, Table 5 explores the effects of each of our two main ideas separately. In this table, “Dirichlet” means that the prior is the Laplace approximation to Dirichlet(α = 0.02), while “Gaussian” indicates that we use a standard Gaussian as prior. ‘High Learning Rate” training means we use β1 > 0.8 and 0.1 > η > 0.0016 with batch normalization, whereas “Low Learning Rate” means β1 > 0.8 and 0.0009 > η > 0.00009 without batch normalization. (For both parameters, the precise value was chosen by Bayesian optimization. We found that these values in the ”with BN” cases were close to the default settings in the Adam optimizer.) We find that the high topic coherence that we achieve in this work is only possible if we use both tricks together. In fact the high learning rates with momentum is required to avoid local minima in the beginning of the training and batch-normalization is required to be able to train the network at these values without diverging. If trained at a lower momentum value or at a lower learning rate PRODLDA shows component collapsing. Interestingly, if we choose a Gaussian prior, rather than the logistic normal approximation used in ProdLDA or NVLDA, the model is easier to train even with low learning rate without any momentum or batch normalization.
The main advantage of AVITM topic models as opposed to NVDM is that the Laplace approximation allows us to match a specific Dirichlet prior of interest. As pointed out by Wallach et al. (2009), the choice of Dirichlet hyperparameter is important to the topic quality of LDA. Following this reasoning, we hypothesize that AVITM topics are higher quality than those of NVDM because they are much more focused, i.e., apply to a more specific subset of documents of interest. We provide support for this hypothesis in Figure 1, by evaluating the sparsity of the posterior proportions over topics, that is, how many of the model’s topics are typically used to explain each document. In order to estimate the sparsity in topic proportions, we project samples from the Gaussian latent spaces of PRODLDA and NVDM in the simplex and average them across documents. We compare the topic
5We note that much recent work follows Hinton & Salakhutdinov (2009) in reporting perplexity for the LDA Gibbs sampler on only a small subset of the test data. Our results are different because we use the entire test dataset.
6β1 is the weight on the average of the gradients from the previous time step and η refers to the learning rate.
sparsity for the standard Gaussian prior used by NVDM to the Laplace approximation of Dirichlet priors with different hyperparameters. Clearly the Laplace approximation to the Dirichlet prior significantly promotes sparsity, providing support for our hypothesis that preserving the Dirichlet prior explains the the increased topic coherence in our method.
The inference network architecture can be found in figure 2 in the appendix.
7 DISCUSSION AND FUTURE WORK
We present what is to our knowledge the first effective AEVB inference algorithm for latent Dirichlet allocation. Although this combination may seem simple in principle, in practice this method is difficult to train because of the Dirichlet prior and because of the component collapsing problem. By addressing both of these problems, we presented a black-box inference method for topic models with the notable advantage that the neural network allows computing topic proportions for new documents without the need to run any variational optimization. As an illustration of the advantages of
1. write article get thanks like anyone please know look one
black box inference techniques, we presented a new topic model, ProdLDA, which achieves significantly better topics than LDA, while requiring a change of only one line of code from AVITM for LDA. Our results suggest that AVITM inference is ready to take its place alongside mean field and collapsed Gibbs as one of the workhorse inference methods for topic models. Future work could include extending our inference methods to handle dynamic and correlated topic models.
ACKNOWLEDGMENTS
We thank Andriy Mnih, Chris Dyer, Chris Russell, David Blei, Hannah Wallach, Max Welling, Mirella Lapata and Yishu Miao for helpful comments, discussions and feedback.
A NETWORK ARCHITECTURE

This paper proposes the use of neural variational inference method for topic models. The paper shows a nice trick to approximate Dirichlet prior using softmax basis with a Gaussian and then the model is trained to maximize the variational lower bound. Also, the authors study a better way to alleviate the component collapsing issue, which has been problematic for continuous latent variables that follow Gaussian distribution. The results look promising and the experimental protocol sounds fine.
Minor comments:
Please add citation to [1] or [2] for neural variational inference, and [2] for VAE.
A typo in “This approximation to the Dirichlet prior p(θ|α) is results in the distribution”, it should be “This approximation to the Dirichlet prior p(θ|α) results in the distribution”
In table 2, it is written that DMFVI was trained more than 24hrs but failed to deliver any result, but why not wait until the end and report the numbers?
In table 3, why are the perplexities of LDA-Collapsed Gibbs and NVDM are lower while the proposed models (ProdLDA) generates more coherent topics? What is your intuition on this?
How does the training speed (until the convergence) differs by using different learning-rate and momentum scheduling approaches shown as in figure 1?
It may be also interesting to add some more analysis on the latent variables z (component collapsing and etc., although your results indirectly show that the learning-rate and momentum scheduling trick removes this issue).
Overall, the paper clearly proposes its main idea, explain why it is good to use NVI, and its experimental results support the original claim. It explains well what are the challenges and demonstrate their solutions.
[1] Minh et al., Neural Variational Inference and Learning in Belief Networks, ICML’14
[2] Rezende et al., Stochastic Backpropagation and Approximate Inference in Deep Generative Models, ICML’14


OPTIMAL BINARY AUTOENCODING WITH PAIRWISE CORRELATIONS
We formulate learning of a binary autoencoder as a biconvex optimization problem which learns from the pairwise correlations between encoded and decoded bits. Among all possible algorithms that use this information, ours finds the autoencoder that reconstructs its inputs with worst-case optimal loss. The optimal decoder is a single layer of artificial neurons, emerging entirely from the minimax loss minimization, and with weights learned by convex optimization. All this is reflected in competitive experimental results, demonstrating that binary autoencoding can be done efficiently by conveying information in pairwise correlations in an optimal fashion.
1 INTRODUCTION
Consider a general autoencoding scenario, in which an algorithm learns a compression scheme for independently, identically distributed (i.i.d.) V -dimensional bit vector data { x̂(1), . . . , x̂(n) } . For
some encoding dimension H , the algorithm encodes each data example x̂(i) = (x̂(i)1 , . . . , x̂ (i) V ) > into an H-dimensional representation e(i), with H < V . It then decodes each e(i) back into a reconstructed example x̃(i) using some small amount of additional memory, and is evaluated on the quality of the reconstruction by the cross-entropy loss commonly used to compare bit vectors. A good autoencoder learns to compress the data into H bits so as to reconstruct it with low loss.
When the loss is squared reconstruction error and the goal is to compress data in RV to RH , this is often accomplished with principal component analysis (PCA), which projects the input data on the top H eigenvectors of their covariance matrix (Bourlard & Kamp (1988); Baldi & Hornik (1989)). These eigenvectors in RV constitute V H real values of additional memory needed to decode the compressed data in RH back to the reconstructions in RV , which are linear combinations of the eigenvectors. Crucially, this total additional memory does not depend on the amount of data n, making it applicable when data are abundant.
This paper considers a similar problem, except using bit-vector data and the cross-entropy reconstruction loss. Since we are compressing samples of i.i.d. V -bit data into H-bit encodings, a natural approach is to remember the pairwise statistics: the V H average correlations between pairs of bits in the encoding and decoding, constituting as much additional memory as the eigenvectors used in PCA. The decoder uses these along with the H-bit encoded data, to produce V -bit reconstructions.
We show how to efficiently learn the autoencoder with the worst-case optimal loss in this scenario, without any further assumptions, parametric or otherwise. It has some striking properties.
The decoding function is identical in form to the one used in a standard binary autoencoder with one hidden layer (Bengio et al. (2013a)) and cross-entropy reconstruction loss. Specifically, each bit v of the decoding is the output of a logistic sigmoid artificial neuron of the encoded bits, with some learned weights wv ∈ RH . This form emerges as the uniquely optimal decoding function, and is not assumed as part of any explicit model.
We show that the worst-case optimal reconstruction loss suffered by the autoencoder is convex in these decoding weights W = {wv}v∈[V ], and in the encoded representations E. Though it is not ∗Most of the work was done as a PhD student at UC San Diego.
jointly convex in both, the situation still admits a natural and efficient optimization algorithm in which the loss is alternately minimized in E and W while the other is held fixed. The algorithm is practical and performs well empirically, learning incrementally from minibatches of data in a stochastic optimization setting.
1.1 NOTATION
The observed data and encodings can be written in matrix form, representing bits as ±1:
X̂ = x̂ (1) 1 · · · x̂ (n) 1 ... . . .
... x̂ (1) V · · · x̂ (n) V
 ∈ [−1, 1]V×n , E = e (1) 1 · · · e (n) 1 ... . . .
... e (1) H · · · e (n) H  ∈ [−1, 1]H×n (1) Here the encodings are allowed to be randomized, represented by values in [−1, 1] instead of just the two values {−1, 1}; e.g. e(1)i = 12 is +1 w.p. 3 4 and −1 w.p. 1 4 . The data in X are also allowed to be randomized, which we will see essentially loses no generality (Appendix B). We write the columns of X̂,E as x̂(i), e(i) for i ∈ [n] (where [s] := {1, . . . , s}), representing the data. The rows are written as x̂v = (x (1) v , . . . , x (n) v )> for v ∈ [V ] and eh = (e(1)h , . . . , e (n) h ) > for h ∈ [H].
We also consider the correlation of each bit h of the encoding with each decoded bit v over the data, i.e. bv,h := 1n ∑n i=1 x (i) v e (i) h . This too can be written in matrix form as B := 1 nX̂E
> ∈ RV×H , whose rows and columns we respectively write as bv = (bv,1, . . . , bv,H)> over v ∈ [V ] and bh = (b1,h, . . . , bV,h) > over h ∈ [H]; the indexing will be clear from context.
As alluded to earlier, the loss incurred on any example x(i) is the cross-entropy between the example and its reconstruction x̃(i), in expectation over the randomness in x(i). Defining `±(x̃ (i) v ) =
ln ( 2
1±x̃(i)v
) (the partial losses to true labels ±1), the loss is written as:
`(x(i), x̃(i)) := V∑ v=1
[( 1 + x (i) v
2
) `+(x̃ (i) v ) + ( 1− x(i)v
2
) `−(x̃ (i) v ) ] (2)
In addition, define a potential well Ψ(m) := ln (1 + em) + ln (1 + e−m) with derivative Ψ′(m) := 1−e−m 1+e−m . Univariate functions like this are applied componentwise to matrices in this paper.
1.2 PROBLEM SETUP
With these definitions, the autoencoding problem we address can be precisely stated as two tasks, encoding and decoding. These share only the side information B. Our goal is to perform these steps so as to achieve the best possible guarantee on reconstruction loss, with no further assumptions. This can be written as a zero-sum game of an autoencoding algorithm seeking to minimize loss against an adversary, by playing encodings and reconstructions:
• Using X̂, algorithm plays (randomized) encodings E, resulting in pairwise correlations B. • Using E and B, algorithm plays reconstructions X̃ = ( x̃(1); . . . ; x̃(n) ) ∈ [−1, 1]V×n.
• Given X̃,E,B, adversary plays X ∈ [−1, 1]V×n to maximize reconstruction loss 1 n ∑n i=1 `(x (i), x̃(i)).
To incur low loss, the algorithm must use an E and B such that no adversary playing X can inflict higher loss. The algorithm never sees X, which represents the worst the data could be given the algorithm’s incomplete memory of it (E,B) and reconstructions (X̃).
We find the autoencoding algorithm’s best strategy in two parts. First, we find the optimal decoding function of any encodings E given B, in Section 2. Then, we use the resulting optimal reconstruction function to outline the best encoding procedure, i.e. one that finds the E,B that lead to the best reconstruction, in Section 3.1. Combining these ideas yields an autoencoding algorithm in Section
3.2 (Algorithm 1), where its implementation and interpretation are specified. Further discussion and related work in Section 4 are followed by more extensions of the framework in Section 5. Experiments in Section 6 show extremely competitive results with equivalent fully-connected autoencoders trained with backpropagation.
2 OPTIMALLY DECODING AN ENCODED REPRESENTATION
To address the game of Section 1.2, we first assume E and B are fixed, and derive the optimal decoding rule given this information. We show in this section that the form of this optimal decoder is precisely the same as in a classical autoencoder: having learned a weight vector wv ∈ RH for each v ∈ [V ], the vth bit of each reconstruction x̃i is expressed as a logistic function of a wv-weighted combination of the H encoded bits ei – a logistic artificial neuron with weights wv. The weight vectors are learned by convex optimization, despite the nonconvexity of the transfer functions.
To develop this, we minimize the worst-case reconstruction error, where X is constrained by our prior knowledge that B = 1nXE >, i.e. 1nExv = bv ∀v ∈ [V ]. This can be written as a function of E:
L∗B(E) := min x̃(1),...,x̃(n)∈[−1,1]V
max x(1),...,x(n)∈[−1,1]V , ∀v∈[V ]: 1nExv=bv
1
n n∑ i=1 `(x(i), x̃(i)) (3)
We solve this minimax problem for the optimal reconstructions played by the minimizing player in (3), written as x̃(1)∗, . . . , x̃(n)∗.
Theorem 1. Define the bitwise slack function γE(w,b) := −b>w + 1n ∑n i=1 Ψ(w
>e(i)), which is convex in w. W.r.t. any bv, this has minimizing weights w∗v := w ∗ v(E,B) := arg min
w∈RH γE(w,bv).
Then the minimax value of the game (3) is L∗B(E) = 1
2 V∑ v=1 γE(w∗v,bv). For any example i ∈ [n],
the minimax optimal reconstruction can be written for any bit v as x̃(i)∗v := 1−e −w∗>v e (i)
1+e−w ∗> v e
(i) .
This tells us that the optimization problem of finding the minimax optimal reconstructions x̃(i) is extremely convenient in several respects. The learning problem decomposes over the V bits in the decoding, reducing to solving for a weight vector w∗v ∈ RH for each bit v, by optimizing each bitwise slack function. Given the weights, the optimal reconstruction of any example i can be specified by a layer of logistic sigmoid artificial neurons of its encoded bits, with w∗>v e (i) as the bitwise logits.
Hereafter, we write W ∈ RV×H as the matrix of decoding weights, with rows {wv}Vv=1. In particular, the optimal decoding weights W∗(E,B) are the matrix with rows {w∗v(E,B)} V v=1.
3 LEARNING AN AUTOENCODER
3.1 FINDING AN ENCODED REPRESENTATION
Having computed the optimal decoding function in the previous section given any E and B, we now switch perspectives to the encoder, which seeks to compress the input data X̂ into encoded representations E (from which B is easily calculated to pass to the decoder). We seek to find (E,B) to ensure the lowest worst-case reconstruction loss after decoding; recall that this is L∗B(E) from (3).
Observe that 1nX̂E > = B by definition, and that the encoder is given X̂. Therefore, by using Thm. 1 and substituting bv = 1nEx̂v ∀v ∈ [V ],
L∗B(E) = 1
2n n∑ i=1 V∑ v=1 [ −x̂(i)v (w∗>v e(i)) + Ψ(w∗>v e(i)) ] := L(W∗,E) (4)
So it is convenient to define the feature distortion 1 for any v ∈ [V ] with respect to W, between any example x and its encoding e:
βWv (e,x) := −xvw>v e + Ψ(w>v e) (5)
From the above discussion, the best E given any decoding W, written as E∗(W), solves the minimization
min E∈[−1,1]H×n L(W,E) = 1 2n n∑ i=1 min e(i)∈[−1,1]H V∑ v=1 βWv (e (i), x̂(i))
which immediately yields the following result.
Proposition 2. Define the optimal encodings for decoding weights W as E∗(W) := arg min
E∈[−1,1]H×n L(W,E). Then e(i)∗(W) can be computed separately for each example x̂(i) ∈ [−1, 1]V , minimizing its total feature distortion over the decoded bits w.r.t. W:
ENC(x̂(i);W) := e(i)∗(W) := arg min e∈[−1,1]H V∑ v=1 βWv (e, x̂ (i)) (6)
Observe that the encoding function ENC(x̂(i);W) can be efficiently computed to any desired precision since the feature distortion βWv (e, x̂
(i)) of each bit v is convex and Lipschitz in e; an L1 error of can be reached in O( −2) linear-time first-order optimization iterations. Note that the encodings need not be bits, and can be e.g. unconstrained ∈ RH instead; the proof of Thm. 1 assumes no structure on them, and the optimization will proceed as above but without projecting into the hypercube.
3.2 AN AUTOENCODER LEARNING ALGORITHM
Our ultimate goal is to minimize the worst-case reconstruction loss. As we have seen in (3) and (6), it is convex in the encoding E and in the decoding parameters W, each of which can be fixed while minimizing with respect to the other. This suggests a learning algorithm that alternately performs two steps: finding encodings E that minimize L(W,E) as in (6) with a fixed W, and finding decoding parameters W∗(E,B), as given in Algorithm 1.
Algorithm 1 Pairwise Correlation Autoencoder (PC-AE)
Input: Size-n dataset X̂, number of epochs T Initialize W0 (e.g. with each element being i.i.d. ∼ N (0, 1)) for t = 1 to T do
Encode each example to ensure accurate reconstruction using weights Wt−1, and compute the associated pairwise bit correlations Bt:
∀i ∈ [n] : [e(i)]t = ENC(x̂(i);Wt−1) , Bt = 1
n X̂E>t
Update weight vectors [wv]t for each v ∈ [V ] to minimize slack function, using encodings Et:
∀v ∈ [V ] : [wv]t = arg min w∈RH
[ −[bv]>t w + 1
n n∑ i=1 Ψ(w>e (i) t ) ] end for Output: Weights WT
1Noting that Ψ(w>v e) ≈ ∣∣w>v e∣∣, we see that βWv (e, x̂) ≈ w>v e (sgn(w>v e) − x̂v). So the optimizer tends
to change e so that w>v e matches signs with x̂v , motivating the name.
3.3 EFFICIENT IMPLEMENTATION
Our derivation of the encoding and decoding functions involves no model assumptions at all, only using the minimax structure and pairwise statistics that the algorithm is allowed to remember. Nevertheless, the (en/de)coders can be learned and implemented efficiently.
Decoding is a convex optimization in H dimensions, which can be done in parallel for each bit v ∈ [V ]. This is relatively easy to solve in the parameter regime of primary interest when data are abundant, in which H < V n. Similarly, encoding is also a convex optimization problem in only H dimensions. If the data examples are instead sampled in minibatches of size n, they can be encoded in parallel, with a new minibatch being sampled to start each epoch t. The number of examples n (per batch) is essentially only limited by nH , the number of compressed representations that fit in memory.
So far in this paper, we have stated our results in the transductive setting, in which all data are given together a priori, with no assumptions whatsoever made about the interdependences between the V features. However, PC-AE operates much more efficiently than this might suggest. Crucially, the encoding and decoding tasks both depend on n only to average a function of x(i) or e(i) over i ∈ [n], so they can both be solved by stochastic optimization methods that use first-order gradient information, like variants of stochastic gradient descent (SGD). We find it remarkable that the minimax optimal encoding and decoding can be efficiently learned by such methods, which do not scale computationally in n. Note that the result of each of these steps involves Ω(n) outputs (E and X̃), which are all coupled together in complex ways.
Furthermore, efficient first-order convex optimization methods for both encoding and decoding steps manipulate more intermediate gradient-related quantities, with facile interpretations. For details, see Appendix A.2.
3.4 CONVERGENCE AND WEIGHT REGULARIZATION
As we noted previously, the objective function of the optimization is biconvex. This means that the alternating minimization algorithm we specify is an instance of alternating convex search, shown in that literature to converge under broad conditions (Gorski et al. (2007)). It is not guaranteed to converge to the global optimum, but each iteration will monotonically decrease the objective function. In light of our introductory discussion, the properties and rate of such convergence would be interesting to compare to stochastic optimization algorithms for PCA, which converge efficiently under broad conditions (Balsubramani et al. (2013); Shamir (2016)).
The basic game used so far has assumed perfect knowledge of the pairwise correlations, leading to equality constraints ∀v ∈ [V ] : 1nExv = bv. This makes sense in PC-AE, where the encoding phase of each epoch gives the exact Bt for the decoding phase. However, in other stochastic settings as for denoising autoencoders (see Sec. 5.2), it may be necessary to relax this constraint. A relaxed constraint of ∥∥ 1 nExv − bv ∥∥ ∞ ≤ exactly corresponds to an extra additive regularization term of
‖wv‖1 on the corresponding weights in the convex optimization used to find W (Appendix D.1). Such regularization leads to provably better generalization (Bartlett (1998)) and is often practical to use, e.g. to encourage sparsity. But we do not use it for our PC-AE experiments in this paper.
4 DISCUSSION AND RELATED WORK
Our approach PC-AE is quite different from existing autoencoding work in several ways.
First and foremost, we posit no explicit decision rule, and avoid optimizing the highly non-convex decision surface traversed by traditional autoencoding algorithms that learn with backpropagation (Rumelhart et al. (1986)). The decoding function, given the encodings, is a single layer of artificial neurons only because of the minimax structure of the problem when minimizing worst-case loss. This differs from reasoning typically used in neural net work (see Jordan (1995)), in which the loss is the negative log-likelihood (NLL) of the joint probability, which is assumed to follow a form specified by logistic artificial neurons and their weights. We instead interpret the loss in the usual direct way as the NLL of the predicted probability of the data given the visible bits, and avoid any assumptions on the decision rule (e.g. not monotonicity in the score w>v e (i), or even dependence on such a score).
This justification of artificial neurons – as the minimax optimal decision rules given information on pairwise correlations – is one of our more distinctive contributions (see Sec. 5.1).
Crucially, we make no assumptions whatsoever on the form of the encoding or decoding, except on the memory used by the decoding. Some such “regularizing" restriction is necessary to rule out the autoencoder just memorizing the data, and is typically expressed by assuming a model class of compositions of artificial neuron layers. We instead impose it axiomiatically by limiting the amount of information transmitted through B, which does not scale in n; but we do not restrict how this information is used. This confers a clear theoretical advantage, allowing us to attain the strongest robust loss guarantee among all possible autoencoders that use the correlations B.
More importantly in practice, avoiding an explicit model class means that we do not have to optimize the typically non-convex model, which has long been a central issue for backpropagation-based learning methods (e.g. Dauphin et al. (2014)). Prior work related in spirit has attempted to avoid this through convex relaxations, including for multi-layer optimization under various structural assumptions (Aslan et al. (2014); Zhang et al. (2016)), and when the number of hidden units is varied by the algorithm (Bengio et al. (2005); Bach (2014)).
Our approach also isolates the benefit of higher n in dealing with overfitting, as the pairwise correlations B can be measured progressively more accurately as n increases. In this respect, we follow a line of research using such pairwise correlations to model arbitary higher-order structure among visible units, rooted in early work on (restricted) Boltzmann Machines (Ackley et al. (1985); Smolensky (1986); Rumelhart & McClelland (1987); Freund & Haussler (1992)). More recently, theoretical algorithms have been developed with the perspective of learning from the correlations between units in a network, under various assumptions on the activation function, architecture, and weights, for both deep (Arora et al. (2014)) and shallow networks (using tensor decompositions, e.g. Livni et al. (2014); Janzamin et al. (2015)). Our use of ensemble aggregation techniques (from Balsubramani & Freund (2015a; 2016)) to study these problems is anticipated in spirit by prior work as well, as discussed at length by Bengio (2009) in the context of distributed representations.
4.1 OPTIMALITY, OTHER ARCHITECTURES, AND DEPTH
We have established that a single layer of logistic artificial neurons is an optimal decoder, given only indirect information about the data through pairwise correlations. This is not a claim that autoencoders need only a single-layer architecture in the worst case. Sec. 3.1 establishes that the best representations E are the solution to a convex optimization, with no artificial neurons involved in computing them from the data. Unlike the decoding function, the optimal encoding function ENC cannot be written explicitly in terms of artificial neurons, and is incomparable to existing architectures (though it is analogous to PCA in prescribing an efficient operation that yields the encodings from unlabeled data). Also, the encodings are only optimal given the pairwise correlations; training algorithms like backpropagation, which communicate other knowledge of the data through derivative composition, can learn final decoding layers that outperform ours, as we see in experiments.
In our framework so far, we explore using all the pairwise correlations between hidden and visible bits to inform learning by constraining the adversary, resulting in a Lagrange parameter – a weight – for each constraint. These V H weights W constitute the parameters of the optimal decoding layer, describing a fully connected architecture. If just a select few of these correlations were used, only they would constrain the adversary in the minimax problem of Sec. 2, so weights would only be introduced for them, giving rise to sparser architectures.
Our central choices – to store only pairwise correlations and minimize worst-case reconstruction loss – play a similar regularizing role to explicit model assumptions, and other autoencoding methods may achieve better performance on data for which these choices are too conservative, by e.g. making distributional assumptions on the data. From our perspective, other architectures with more layers – particularly highly successful ones like convolutional, recurrent, residual, and ladder networks (LeCun et al. (2015); He et al. (2015); Rasmus et al. (2015)) – lend the autoencoding algorithm more power by allowing it to measure more nuanced correlations using more parameters, which decreases the worst-case loss. Applying our approach with these would be interesting future work.
Extending this paper’s convenient minimax characterization to deep representations with empirical success is a very interesting open problem. Prior work on stacking autoencoders/RBMs (Vincent et al.
(2010)) and our learning algorithm PC-AE suggest that we could train a deep network in alternating forward and backward passes. Using this paper’s ideas, the forward pass would learn the weights to each layer given the previous layer’s activations (and inter-layer pairwise correlations) by minimizing the slack function, with the backward pass learning the activations for each layer given the weights to / activations of the next layer by convex optimization (as we learn E). Both passes would consist of successive convex optimizations dictated by our approach, quite distinct from backpropagation, though loosely resembling the wake-sleep algorithm (Hinton et al. (1995)).
4.2 GENERATIVE APPLICATIONS
Particularly recently, autoencoders have been of interest largely for their many applications beyond compression, especially for their generative uses. The most directly relevant to us involve repurposing denoising autoencoders (Bengio et al. (2013b); see Sec. 5.2); moment matching among hidden and visible units (Li et al. (2015)); and generative adversarial network ideas (Goodfellow et al. (2014); Makhzani et al. (2015)), the latter particularly since the techniques of this paper have been applied to binary classification (Balsubramani & Freund (2015a;b)). These are outside this paper’s scope, but suggest themselves as future extensions of our approach.
5 EXTENSIONS
5.1 OTHER RECONSTRUCTION LOSSES
It may make sense to use another reconstruction loss other than cross-entropy, for instance the expected Hamming distance between x(i) and x̃(i). It turns out that the minimax manipulations we use work under very broad conditions, for nearly any loss that additively decomposes over the V bits as cross-entropy does. In such cases, all that is required is that the partial losses `+(x̃ (i) v ), `−(x̃ (i) v ) are monotonically decreasing and increasing respectively (recall that for cross-entropy loss, this is true as `±(x̃ (i) v ) = ln ( 2
1±x̃(i)v
) ); they need not even be convex. This monotonicity is a natural condition,
because the loss measures the discrepancy to the true label, and holds for all losses in common use.
Changing the partial losses only changes the structure of the minimax solution in two respects: by altering the form of the transfer function on the decoding neurons, and the univariate potential well Ψ optimized to learn the decoding weights. Otherwise, the problem remains convex and the algorithm is identical. Formal statements of these general results are in Appendix E.
5.2 DENOISING AUTOENCODING
Our framework can be easily applied to learn a denoising autoencoder (DAE; Vincent et al. (2008; 2010)), which uses noise-corrupted data (call it Ẋ) for training, and uncorrupted data for evaluation. From our perspective, this corresponds to leaving the learning of W unchanged, but using corrupted data when learning E. Consequently, the minimization problem over encodings must be changed to account for the bias on B introduced by the noise; so the algorithm plays given the noisy data, but to minimize loss against X. This is easiest to see for zero-mean noise, for which our algorithms are completely unchanged because B does not change (in expectation) after the noise is added.
Another common scenario illustrating this technique is to mask a ρ fraction of the input bits uniformly at random (in our notation, changing 1s to−1s). This masking noise changes each pairwise correlation bv,h by an amount δv,h := 1n ∑n i=1(ẋ (i) v −x(i)v )e(i)h . Therefore, the optimand Eq. (4) must be modified by subtracting this factor δv,h. This δv,h can be estimated (w.h.p.) given ẋv, eh, ρ,xv . But even with just the noisy data and not xv , we can estimate δv,h w.h.p. by extrapolating the correlation of the bits of ẋv that are left as +1 (a 1− ρ fraction) with the corresponding values in eh (see Appendix C).
6 EXPERIMENTS
In this section we compare our approach 2 empirically to a standard autoencoder with one hidden layer (termed AE here) trained with backpropagation, and a thresholded PCA baseline. Our goal is simply to verify that our approach, though very different, is competitive in reconstruction performance.
The datasets we use are first normalized to [0, 1], and then binarized by sampling each pixel stochastically in proportion to its intensity, following prior work (Salakhutdinov & Murray (2008)). Changing between binary and real-valued encodings in PC-AE requires just a line of code, to project the encodings into [−1, 1]H after convex optimization updates to compute ENC(·). We use Adagrad (Duchi et al. (2011)) for the convex minimizations of our algorithms; we observed that their performance is not very sensitive to the choice of optimization method, explained by our approach’s convexity.
We compare to a basic AE with a single hidden layer, trained using the Adam method with default parameters (Kingma & Ba (2014)). Other models like variational autoencoders (Kingma & Welling (2013)) are not shown here because they do not aim to optimize reconstruction loss or are not comparably general autoencoding architectures. We also use a sign-thresholded PCA baseline (essentially a completely linear autoencoder, but with the output layer thresholded to be in [−1, 1]); see Appendix A for more details. We vary the number of hidden units H for all algorithms, and try both binary and unconstrained real-valued encodings where appropriate; the respective AE uses logistic sigmoid and ReLU transfer functions for the encoding neurons. The results are in Table 1.
The reconstruction performance of PC-AE indicates that it can encode information very well using pairwise correlations, compared to the directly learned AE and PCA approaches. Loss can become extremely low when H is raised, giving B the capacity to robustly encode almost all the information in the input bits X̂. The performance is roughly equal between binary hidden units and unconstrained ones, which is expected by our derivations.
We also try learning just the decoding layer of Sec. 2, on the encoded representation of the AE. This is motivated by the fact that Sec. 2 establishes our decoding method to be worst-case optimal given any E and B. We find the results to be significantly worse than the AE alone in all datasets used (e.g. reconstruction loss of ∼ 171/133 on MNIST, and ∼ 211/134 on Omniglot, with 32/100 hidden units respectively). This reflects the AE’s training backpropagating information about the data beyond pairwise correlations, through non-convex function compositions – however, this comes at the cost of being more difficult to optimize. The representations learned by the ENC function of PC-AE are quite different and capture much more of the pairwise correlation information, which is used by the decoding layer in a worst-case optimal fashion. We attempt to visually depict the differences between the representations in Fig. 3.
As discussed in Sec. 4, we do not claim that this paper’s method will always achieve the best empirical reconstruction loss, even among single-layer autoencoders. We would like to make the encoding
2TensorFlow code available at https://github.com/aikanor/pc-autoencoder .
function quicker to compute, as well. But we believe this paper’s results, especially when H is high, illustrate the potential of using pairwise correlations for autoencoding as in our approach, learning to encode with alternating convex minimization and extremely strong worst-case robustness guarantees.
ACKNOWLEDGMENTS
I am grateful to Jack Berkowitz, Sanjoy Dasgupta, and Yoav Freund for helpful discussions; Daniel Hsu and Akshay Krishnamurthy for instructive examples; and Gary Cottrell for an enjoyable chat. I acknowledge funding from the NIH (grant R01ES02500902).
A EXPERIMENTAL DETAILS
In addition to MNIST, we use the preprocessed version of the Omniglot dataset found in Burda et al. (2016), split 1 of the Caltech-101 Silhouettes dataset, the small notMNIST dataset, and the UCI Adult (a1a) dataset. The results reported are the mean of 10 Monte Carlo runs, and the PC-AE significance results use 95% Monte Carlo confidence intervals. Only notMNIST comes without a predefined split, so the displayed results use 10-fold cross-validation. Non-binarized versions of all datasets (grayscale pixels) resulted in nearly identical PC-AE performance (not shown); this is as expected from its derivation using expected pairwise correlations, which with high probability are nearly invariant under binarization (by e.g. Hoeffding bounds).
We used minibatches of size 250. All standard autoencoders use the ’Xavier’ initialization and trained for 500 epochs or using early stopping on the test set. The “PCA" baseline was run on exactly the same input data as the others; it finds decodings by mean-centering this input, finding the top H principal components with standard PCA, reconstructing the mean-centered input with these components, adding back the means, and finally thresholding the result to [−1, 1]V . We did not evaluate against other types of autoencoders which regularize (Kingma & Welling (2013)) or are otherwise not trained for direct reconstruction loss minimization. Also, not shown is the performance of a standard convolutional autoencoder (32-bit representation, depth-3 64-64-32 (en/de)coder) which performs better than the standard autoencoder, but is still outperformed by PC-AE on our image-based datasets. A deeper architecture could quite possibly achieve superior performance, but the greater number of channels through which information is propagated makes fair comparison with our flat fully-connected approach difficult. We consider extension of our PC-AE approach to such architectures to be fascinating future work.
A.1 FURTHER RESULTS
Our bound on worst-case loss is invariably quite tight, as shown in Fig. 4. Similar results are found on all datasets. This is consistent with our conclusions about the nature of the PC-AE representations – conveying almost exactly the information available in pairwise correlations.
A 2D visualization of MNIST is in Fig. 6, showing that even with just two hidden units there is enough information in pairwise correlations for PC-AE to learn a sensible embedding. We also include more pictures of our autoencoders’ reconstructions, and visualizations of the hidden units when H = 100 in Fig. 5.
A.2 PC-AE INTERPRETATION AND IMPLEMENTATION DETAILS
Here we give some details that are useful for interpretation and implementation of the proposed method.
A.2.1 ENCODING
Proposition 2 defines the encoding function for any data example x as the vector that minimizes the total feature distortion, summed over the bits in the decoding, rewritten here for convenience:
ENC(x(i);W) := arg min e∈[−1,1]H V∑ v=1 [ −x(i)v w>v e(i) + Ψ(w>v e(i)) ] (7)
Doing this on multiple examples at once (in memory as a minibatch) can be much faster than on each example separately. We can now compute the gradient of the objective function w.r.t. each example i ∈ [n], writing the gradient w.r.t. example i as column i of a matrix G ∈ RH×n. G can be calculated efficiently in a number of ways, for example as follows:
• Compute matrix of hallucinated data X̆ := Ψ′(WE) ∈ RV×n. • Subtract X to compute residuals R := X̆−X ∈ RV×n. • Compute G = 1nW >R ∈ RH×n.
Optimization then proceeds with gradient descent using G, with the step size found using line search. Note that since the objective function is convex, the optimum E∗ leads to optimal residuals R∗ ∈ RV×n such that G = 1nW
>R∗ = 0H×n, so each column of R∗ is in the null space of W>, which maps the residual vectors to the encoded space. We conclude that although the compression is not perfect (so the optimal residuals R∗ 6= 0V×n in general), each column of R∗ is orthogonal to the decoding weights at an equilibrium towards which the convex minimization problem of (7) is guaranteed to stably converge.
A.2.2 DECODING
The decoding step finds W to ensure accurate decoding of the given encodings E with correlations B, solving the convex minimization problem:
W∗ = arg min W∈RV×H V∑ v=1 [ −b>v wv + 1 n n∑ i=1 Ψ(w>v e (i)) ] (8)
This can be minimized by first-order convex optimization. The gradient of (8) at W is:
−B + 1 n [Ψ′(WE)]E> (9)
The second term can be understood as “hallucinated" pairwise correlations B̆, between bits of the encoded examples E and bits of their decodings under the current weights, X̆ := Ψ′(WE). The hallucinated correlations can be written as B̆ := 1nX̆E
>. Therefore, (9) can be interpreted as the residual correlations B̆ −B. Since the slack function of (8) is convex, the optimum W∗ leads to hallucinated correlations B̆∗ = B, which is the limit reached by the optimization algorithm after many iterations.
B ALLOWING RANDOMIZED DATA AND ENCODINGS
In this paper, we represent the bit-vector data in a randomized way in [−1, 1]V . Randomizing the data only relaxes the constraints on the adversary in the game we play; so at worst we are working with an upper bound on worst-case loss, instead of the exact minimax loss itself, erring on the conservative side. Here we briefly justify the bound as being essentially tight, which we also see empirically in this paper’s experiments.
In the formulation of Section 2, the only information we have about the data is its pairwise correlations with the encoding units. When the data are abundant (n large), then w.h.p. these correlations are close to their expected values over the data’s internal randomization, so representing them as continuous values w.h.p. results in the same B and therefore the same solutions for E,W. We are effectively allowing the adversary to play each bit’s conditional probability of firing, rather than the binary realization of that probability.
This allows us to apply minimax theory and duality to considerably simplify the problem to a convex optimization, when it would otherwise be nonconvex, and computationally hard (Baldi (2012)). The fact that we are only using information about the data through its expected pairwise correlations with the hidden units makes this possible.
The above also applies to the encodings and their internal randomization, allowing us to learn binary randomized encodings by projecting to the convex set [−1, 1]H .
C DENOISING AUTOENCODER WITH MASKING NOISE: DETAILS
This section elaborates on the discussion of Sec. 5.2.
Recall the correlation correction term δv,h from Sec. 5.2:
δv,h = 1
n n∑ i=1 (ẋ(i)v − x(i)v )e (i) h
Here, we express this in terms of the known quantities ẋv, eh, ρ, and not the unknown denoised data xv .
Consider that
(ẋ(i)v − x(i)v )e (i) h = 1 ( x(i)v = −1 ) (ẋ(i)v − x(i)v )e (i) h + 1 ( x(i)v = +1 ) (ẋ(i)v − x(i)v )e (i) h
Now if x(i)v = −1, then ẋ(i)v = −1, so (ẋ(i)v − x(i)v )e(i)h = 0. Therefore the first term above is zero, and the expression can be simplified:
(ẋ(i)v − x(i)v )e (i) h = 1 ( x(i)v = +1 ) (ẋ(i)v − x(i)v )e (i) h = 1 ( x(i)v = +1 ∧ ẋ(i)v = −1 ) (−2)e(i)h (10)
Now on any example i, independent of the value of e(i)h , a ρ fraction of the bits where x (i) v = +1 are flipped to get ẋ(i)v . Therefore,
1
ρ n∑ i=1 1 ( x(i)v = +1 ∧ ẋ(i)v = −1 ) e (i) h ≈ 1 1− ρ n∑ i=1 1 ( x(i)v = +1 ∧ ẋ(i)v = +1 ) e (i) h
Putting it all together,
δv,h = 1
n n∑ i=1 (ẋ(i)v − x(i)v )e (i) h = − 2 n n∑ i=1 1 ( x(i)v = +1 ∧ ẋ(i)v = −1 ) e (i) h
≈ − 2 n ρ 1− ρ n∑ i=1 1 ( x(i)v = +1 ∧ ẋ(i)v = +1 ) e (i) h = − 2 n ρ 1− ρ n∑ i=1 1 ( ẋ(i)v = +1 ) e (i) h
D PROOFS
Proof of Theorem 1. Writing Γ(x̃(i)v ) := `−(x̃ (i) v ) − `+(x̃(i)v ) = ln ( 1+x̃(i)v 1−x̃(i)v ) for convenience, we can simplify L∗, using the definition of the loss (2), and Lagrange duality for all V H constraints involving B.
This leads to the following chain of equalities, where for brevity the constraint sets are sometimes omitted when clear, and we write X as shorthand for the data x(1), . . . ,x(n) and X̃ analogously for the reconstructions.
L∗ = 1 2 min x̃(1),...,x̃(n)
∈[−1,1]V max x(1),...,x(n)∈[−1,1]V , ∀v∈[V ]: 1nExv=bv
1
n n∑ i=1 V∑ v=1 [( 1 + x(i)v ) `+(x̃ (i) v ) + ( 1− x(i)v ) `−(x̃ (i) v ) ]
= 1
2 min X̃ max X min W∈RV×H
[ 1
n n∑ i=1 V∑ v=1 ( `+(x̃ (i) v ) + `−(x̃ (i) v )− x(i)v Γ(x̃(i)v ) ) + V∑ v=1 w>v ( 1 n Exv − bv )] (a) = 1
2 min w1,...,wV
[ −
V∑ v=1 b>v wv + 1 n min X̃ max X V∑ v=1 [ n∑ i=1 ( `+(x̃ (i) v ) + `−(x̃ (i) v )− x(i)v Γ(x̃(i)v ) ) + w>v Exv ]]
= 1
2 min w1,...,wV
[ −
V∑ v=1 b>v wv + 1 n min X̃ n∑ i=1 V∑ v=1 [ `+(x̃ (i) v ) + `−(x̃ (i) v ) + max x(i)∈[−1,1]V x(i)v ( w>v e (i) − Γ(x̃(i)v ) )]]
(11)
where (a) uses the minimax theorem (Cesa-Bianchi & Lugosi (2006)), which can be applied as in linear programming, because the objective function is linear in x(i) and wv. Note that the weights are introduced merely as Lagrange parameters for the pairwise correlation constraints, not as model assumptions.
The strategy x(i) which solves the inner maximization of (11) is to simply match signs with w>v e (i)− Γ(x̃ (i) v ) coordinate-wise for each v ∈ [V ]. Substituting this into the above,
L∗ = 1 2 min w1,...,wV
[ −
V∑ v=1 b>v wv + 1 n n∑ i=1 min x̃(i)∈[−1,1]V V∑ v=1 ( `+(x̃ (i) v ) + `−(x̃ (i) v ) + ∣∣∣w>v e(i) − Γ(x̃(i)v )∣∣∣) ]
= 1
2 V∑ v=1 min wv∈RH [ −b>v wv + 1 n n∑ i=1 min x̃ (i) v ∈[−1,1] ( `+(x̃ (i) v ) + `−(x̃ (i) v ) + ∣∣∣w>v e(i) − Γ(x̃(i)v )∣∣∣) ]
The absolute value breaks down into two cases, so the inner minimization’s objective can be simplified:
`+(x̃ (i) v ) + `−(x̃ (i) v ) + ∣∣∣w>v e(i) − Γ(x̃(i)v )∣∣∣ = { 2`+(x̃ (i) v ) + w>v e (i) if w>v e (i) ≥ Γ(x̃(i)v )
2`−(x̃ (i) v )−w>v e(i) if w>v e(i) < Γ(x̃ (i) v )
(12)
Suppose x̃(i)v falls in the first case of (12), so that w>v e (i) ≥ Γ(x̃(i)v ). By definition of `+(·), 2`+(x̃ (i) v ) +w>v e (i) is decreasing in x̃(i)v , so it is minimized for the greatest x̃ (i)∗ v ≤ 1 s.t. Γ(x̃(i)∗v ) ≤ w>v e (i). This means Γ(x̃(i)∗v ) = w>v e (i), so the minimand (12) is `+(x̃ (i)∗ v ) + `−(x̃ (i)∗ v ), where x̃i∗v = 1−e−w > v e (i)
1+e−w > v e
(i) .
A precisely analogous argument holds if x̃(i)v falls in the second case of (12), where w>v e (i) < Γ(x̃ (i) v ).
Putting the cases together, we have shown the form of the summand Ψ. We have also shown the dependence of x̃(i)∗v on w∗>v e
(i), where w∗v is the minimizer of the outer minimization of (11). This completes the proof.
D.1 L∞ CORRELATION CONSTRAINTS AND L1 WEIGHT REGULARIZATION
Here we formalize the discussion of Sec. 3.4 with the following result. Theorem 3.
min x̃(1),...,x̃(n)∈[−1,1]V
max x(1),...,x(n)∈[−1,1]V ,
∀v∈[V ]: ‖ 1nExv−bv‖∞≤ v
1
n n∑ i=1 `(x(i), x̃(i))
= 1
2 V∑ v=1 min wv∈RH [ −b>v wv + 1 n n∑ i=1 Ψ(w>v e (i)) + v ‖wv‖1 ]
For each v, i, the minimizing x̃(i)v is a logistic function of the encoding e(i) with weights equal to the minimizing w∗v above, exactly as in Theorem 1.
Proof. The proof adapts the proof of Theorem 1, following the result on L1 regularization in Balsubramani & Freund (2016) in a very straightforward way; we describe this here.
We break each L∞ constraint into two one-sided constraints for each v, i.e. 1nExv − bv ≤ v1 n and 1 nExv − bv ≥ − v1
n. These respectively give rise to two sets of Lagrange parameters λv, ξv ≥ 0H for each v, replacing the unconstrained Lagrange parameters wv ∈ RH . The conditions for the minimax theorem apply here just as in the proof of Theorem 1, so that (11) is replaced by
1 2 min
λ1,...,λV ξ1,...,ξV
[ −
V∑ v=1 ( b>v (ξv − λv)− v1>(ξv + λv) ) (13)
+ 1
n min X̃ n∑ i=1 V∑ v=1 [ `+(x̃ (i) v ) + `−(x̃ (i) v ) + max x(i) x(i)v ( (ξv − λv)>e(i) − Γ(x̃(i)v ) )]] (14)
Suppose for some h ∈ [H] that ξv,h > 0 and λv,h > 0. Then subtracting min(ξv,h, λv,h) from both does not affect the value [ξv − λv]h, but always decreases [ξv + λv]h, and therefore always decreases the objective function. Therefore, we can w.l.o.g. assume that ∀h ∈ [H] : min(ξv,h, λv,h) = 0. Defining wv = ξv − λv (so that ξv,h = [wv,h]+ and λv,h = [wv,h]− for all h), we see that the term v1 >(ξv + λv) in (13) can be replaced by v ‖wv‖1.
Proceeding as in the proof of Theorem 1 gives the result.
E GENERAL RECONSTRUCTION LOSSES
In this section we extend Theorem 1 to a larger class of reconstruction losses for binary autoencoding, of which cross-entropy loss is a special case. This uses techniques recently employed by Balsubramani & Freund (2016) for binary classification.
Since the data X are still randomized binary, we first broaden the definition of (2), rewritten here:
`(x(i), x̃(i)) := V∑ v=1
[( 1 + x (i) v
2
) `+(x̃ (i) v ) + ( 1− x(i)v
2
) `−(x̃ (i) v ) ] (15)
We do this by redefining the partial losses `±(x̃ (i) v ), to any functions satisfying the following monotonicity conditions. Assumption 1. Over the interval (−1, 1), `+(·) is decreasing and `−(·) is increasing, and both are twice differentiable.
Assumption 1 is a very natural one and includes many non-convex losses (see Balsubramani & Freund (2016) for a more detailed discussion, much of which applies bitwise here). This and the additive decomposability of (15) over the V bits are the only assumptions we make on the reconstruction loss `(x(i), x̃(i)). The latter decomposability assumption is often natural when the loss is a log-likelihood, where it is tantamount to conditional independence of the visible bits given the hidden ones.
Given such a reconstruction loss, define the increasing function Γ(y) := `−(y)−`+(y) : [−1, 1] 7→ R, for which there exists an increasing (pseudo)inverse Γ−1. Using this we broaden the definition of the potential function Ψ in terms of `±:
Ψ(m) :=  −m+ 2`−(−1) if m ≤ Γ(−1) `+(Γ −1(m)) + `−(Γ −1(m)) if m ∈ (Γ(−1),Γ(1))
m+ 2`+(1) if m ≥ Γ(1)
Then we may state the following result, describing the optimal decoding function for a general reconstruction loss. Theorem 4. Define the potential function
min x̃(1),...,x̃(n)∈[−1,1]V max x(1),...,x(n)∈[−1,1]V , ∀v∈[V ]: 1nExv=bv
1
n n∑ i=1 `(x(i), x̃(i))
= 1
2 V∑ v=1 min wv∈RH [ −b>v wv + 1 n n∑ i=1 Ψ(w>v e (i)) ]
For each v ∈ [V ], i ∈ [n], the minimizing x̃(i)v is a sigmoid function of the encoding e(i) with weights equal to the minimizing w∗v above, as in Theorem 1. The sigmoid is defined as
x̃(i)∗v :=  −1 if w∗v>e(i) ≤ Γ(−1) Γ−1(w∗v >e(i)) if w∗v >e(i) ∈ (Γ(−1),Γ(1))
1 if w∗v >e(i) ≥ Γ(1)
(16)
The proof is nearly identical to that of the main theorem of Balsubramani & Freund (2016). That proof is essentially recapitulated here for each bit v ∈ [V ] due to the additive decomposability of the loss, through algebraic manipulations (and one application of the minimax theorem) identical to the proof of Theorem 1, but using the more general specifications of Ψ and Γ in this section. So we do not rewrite it in full here.
A notable special case of interest is the Hamming loss, for which `±(x̃ (i) v ) = 1 2 ( 1∓ x̃(i)v ) , where the reconstructions are allowed to be randomized binary values. In this case, we have Ψ(m) = max(|m| , 1), and the sigmoid used for each decoding neuron is the clipped linearity max(−1,min(w∗v>e(i), 1)).
F ALTERNATE APPROACHES
We made some technical choices in the derivation of PC-AE, which prompt possible alternatives not explored here for a variety of reasons. Recounting these choices gives more insight into our framework.
The output reconstructions could have restricted pairwise correlations, i.e. 1nX̃E > = B. One option is to impose such restrictions instead of the existing constraints on X, leaving X unrestricted. However, this is not in the spirit of this paper, because B is our means of indirectly conveying information to the decoder about how X is decoded.
Another option is to restrict both X̃ and X. This is possible and may be useful in propagating correlation information between layers of deeper architectures while learning, but its minimax solution does not have the conveniently clean structure of the PC-AE derivation.
In a similar vein, we could restrict E during the encoding phase, using B and X. As B is changed only during this phase to better conform to the true data X, this tactic fixes B during the optimization, which is not in the spirit of this paper’s approach. It also performed significantly worse in our experiments.

The paper propose to find an optimal decoder for binary data using a min-max decoder on the binary hypercube given a linear constraint on the correlation between the encoder and the  data.
The paper gives finally that the optimal decoder as logistic of the lagragian W multiplying the encoding e.

Given the weights of the ‘min-max’decoder W the paper finds the best encoding for the data distribution considered, by minimizing that error as a function of the encoding.
The paper then alternates that optimization between the encoding and the min-max decoding, starting from random weights W.
clarity:
-The paper would be easier to follow if the real data (x in section 3 ) is differentiated from the worst case data played by the model (x in section 2).
significance
Overall I like the paper, however I have some doubts on what the alternating optimization optimum ends up being.  The paper ends up implementing a single layer network. The correlation constraints while convenient in the derivation, is  a bit intriguing. Since linear relation between the encoding and the data  seems to be weak modeling constraint and might be not different from what PCA would implement.
- what is the performance of PCA on those tasks? one could you use a simple sign function to decode. This is related to one bit compressive sensing.
- what happens if you initialize W in algorithm one with PCA weights? or weighted pca weights?
- Have you tried on more complex datasets such as cifar?